Builder partially degraded service

Incident Report for balena.io

Postmortem

Discovered because builders were crash-looping, indicated by error codes related to long strings from large buffer sizes. Quickly linked to an accumulation of uncleaned images on the builder instances.

A manual command balena image prune was executed to remove images and temporarily recover builder performance. Recognized the need for more strategic, long-term solutions.

Posted Mar 29, 2024 - 17:15 UTC

Resolved

This incident has been resolved.
Posted Mar 29, 2024 - 17:14 UTC

Monitoring

Discovered because builders were crash-looping, indicated by error codes related to long strings from large buffer sizes.
Quickly linked to an accumulation of uncleaned images on the builder instances.

A manual command balena image prune was executed to remove images and temporarily recover builder performance.
Recognized the need for more strategic, long-term solutions.
Posted Mar 29, 2024 - 14:53 UTC

Investigating

We are currently investigating this issue.
Posted Mar 29, 2024 - 13:44 UTC
This incident affected: Application Builder.
← Current Status Powered by Atlassian Statuspage