Elevated GIT/Application Builder Errors

Incident Report for balena.io

Postmortem

Delta generation service experienced degraded performance from ~0900 UTC Feb 4 to ~1700 UTC Feb 5, 2026, due to 32-bit address space limitations affecting large image delta processing.

Impact:

  • Delta generation timeout rate increased significantly for large container images
  • Users experienced slower image deployments and potential timeouts
  • No data loss or security impact

Root Cause: Worker auto-discovery began routing some large delta generation tasks to 32-bit Docker sockets, causing memory constraint failures.

Resolution: Corrected worker selection routing to only use 64-bit sockets for delta generation.

Follow-up Actions:

  • Improve monitoring of delta generation performance and failure rates
  • Add alerting for threshold violations to detect similar issues faster

We apologize for the service interruption and any inconvenience this caused to your deployments. We are committed to improving our monitoring and alerting to prevent similar issues in the future.

Posted Feb 09, 2026 - 15:12 UTC

Resolved

This incident has been resolved.
Posted Feb 06, 2026 - 13:53 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Feb 05, 2026 - 17:56 UTC

Investigating

We're experiencing an elevated level of errors in our application builder infrastructure and are currently looking into the issue.
Posted Feb 04, 2026 - 20:43 UTC
This incident affected: Application Builder.