Elevated API Errors

Incident Report for balena.io

Postmortem

An internal observability feature led to unreasonable base memory footprint for API instances under production load, leading to frequent evictions.

For now, we’ve rolled back to a previous API version to restore stability, while we investigate the root cause.

Posted Mar 04, 2025 - 00:38 UTC

This incident has been resolved.

Posted Mar 04, 2025 - 00:36 UTC

A fix has been implemented and we are monitoring the results.

Posted Mar 03, 2025 - 21:03 UTC

The issue has been identified and a fix is being implemented.

Posted Mar 03, 2025 - 20:46 UTC

The issue has been identified and a fix is being implemented.

Posted Mar 03, 2025 - 19:48 UTC

We're experiencing an elevated level of API errors and are currently looking into the issue.

Posted Mar 03, 2025 - 18:59 UTC

This incident affected: API.