Builder partially degraded service
Incident Report for Balena.io
Postmortem

Discovered because builders were crash-looping, indicated by error codes related to long strings from large buffer sizes. Quickly linked to an accumulation of uncleaned images on the builder instances.

A manual command balena image prune was executed to remove images and temporarily recover builder performance. Recognized the need for more strategic, long-term solutions.

Posted Mar 29, 2024 - 17:15 UTC

Resolved
This incident has been resolved.
Posted Mar 29, 2024 - 17:14 UTC
Monitoring
Discovered because builders were crash-looping, indicated by error codes related to long strings from large buffer sizes.
Quickly linked to an accumulation of uncleaned images on the builder instances.

A manual command balena image prune was executed to remove images and temporarily recover builder performance.
Recognized the need for more strategic, long-term solutions.
Posted Mar 29, 2024 - 14:53 UTC
Investigating
We are currently investigating this issue.
Posted Mar 29, 2024 - 13:44 UTC
This incident affected: Application Builder.