Elevated API Errors

Incident Report for balena.io

Postmortem

A vulnerability mitigation update required replacing compute cluster nodes, which, when applied, rolled back due to a timeout. This caused some workloads (API, delta, builder) to become temporarily unavailable and triggering some undesired secondary effects, including ungracefully terminating a few long lived instances servicing VPN connections. While the rest of the services came back reasonably quickly within a minute or so, it too around one and a quarter hours to re-establish VPN tunnels.

A scheduled maintenance will be posted later to perform this update during a planned outage window.

Posted May 05, 2026 - 21:23 UTC

Resolved

This incident has been resolved.
Posted May 05, 2026 - 21:03 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted May 05, 2026 - 18:43 UTC

Identified

The issue has been identified and a fix is being implemented.
Posted May 05, 2026 - 18:19 UTC

Investigating

We're experiencing an elevated level of API errors and are currently looking into the issue.
Posted May 05, 2026 - 18:12 UTC
This incident affected: API, Application Builder, and Delta Image Downloads.