Elevated balenaOS Download Errors
Incident Report for balena.io
Postmortem

Incident Summary: A modification in our Continuous Integration (CI) system inadvertently led to the creation of non-annotated GitHub release tags. This triggered a disruption in the balenaOS finalized release pipeline with duplicate releases containing no associated artifacts.

Impact: While balenaOS releases were marked as final and publicly available for download, the corresponding image artifacts were missing from our backend storage. This issue did not affect device hostApp updates; however, attempts to download OS release images via the latest tags resulted in failures.

Resolution: We invalidated the affected releases, resolved the tagging issue in our CI pipeline to ensure the generation of correct tags in the future, and implemented a fallback in our OS deployment pipeline to accommodate both types of git tags.

Posted Apr 30, 2024 - 17:27 UTC

Resolved
This incident has been resolved.
Posted Apr 30, 2024 - 14:01 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Apr 26, 2024 - 19:44 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Apr 26, 2024 - 19:22 UTC
Investigating
We're experiencing an elevated level of balenaOS download errors and are currently looking into the issue.
Posted Apr 26, 2024 - 19:07 UTC
This incident affected: BalenaOS Download.