A routine version upgrade to a backend cache changed how the API handled its request queues. Under unusually heavy request load, those queues filled up and the API started dropping traffic, returning timeouts and errors.
Over the course of the incident, affected users saw elevated API error rates that intermittently disrupted dashboard logins, API access, and device connectivity.
We mitigated it by rate-limiting the abnormal traffic at our edge and rolling the cache back to the previous version, which restored normal queue behaviour and recovered the platform.
To avoid a repeat, we're improving rate limiting so a single heavy consumer can't degrade service for others, and exploring autoscaling and load testing so infrastructure changes are validated under realistic load before they reach production.