Elevated error rate/Degraded performance on the API
Incident Report for Balena.io
Postmortem

The deployment of a new version of the API, addressing recent degraded performance incidents, resulted in the introduction of a bug on our rate limiting, which affected device online status updates on our VPN service. During the incident, an increase of the database load resulted in degraded performance and an elevated error rate on our API service. After verification, fixes were deployed addressing the issues and gradually restoring the performance on both services.

Posted Oct 15, 2019 - 14:32 UTC

Resolved
This incident has been resolved.
Posted Oct 10, 2019 - 18:13 UTC
Update
We are continuing to monitor for any further issues.
Posted Oct 10, 2019 - 16:46 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Oct 10, 2019 - 16:27 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Oct 10, 2019 - 16:19 UTC
Investigating
We are currently investigating this issue.
Posted Oct 10, 2019 - 16:17 UTC
This incident affected: API and Cloudlink (VPN).