Elevated Application Registry Errors
Incident Report for balena.io
Postmortem

We experienced timeouts for deltas and builds when pushing images to our registry hosted in the US East (N. Virginia) region. This issue impacted our cloud builders in Finland and Germany, among other regions.

The root cause was identified as a public routing issue between certain regions, affecting the ability of some of our systems to access the registry efficiently.

We resolved the issue by enabling proxied routing protocols for our registry endpoint. This allowed us to bypass the impacted network paths and restore normal operations.

Impact

  • Cloud builders in Finland and Germany experienced delays in image pushing
  • Potential delays in deployment pipelines for affected regions
  • No data loss or security breaches occurred
Posted Jul 31, 2024 - 13:26 UTC

Resolved
This incident has been resolved.
Posted Jul 31, 2024 - 13:25 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 26, 2024 - 15:13 UTC
Update
We are continuing to investigate this issue.
Posted Jul 26, 2024 - 12:13 UTC
Update
We are continuing to investigate this issue.
Posted Jul 25, 2024 - 23:59 UTC
Update
We are continuing to investigate this issue.
Posted Jul 25, 2024 - 20:31 UTC
Investigating
We're experiencing degraded performance around creating new releases on our builders and are currently looking into the issue. Users experiencing issues creating releases using the `balena push` command, are advised to temporarily switch to local builds via `balena deploy`, if possible, until the incident is resolved.
Posted Jul 25, 2024 - 16:57 UTC
This incident affected: Application Builder, Application Registry, and Delta Image Downloads.