Elevated API Errors

Incident Report for balena.io

Resolved

This incident has been resolved.
Posted Sep 10, 2025 - 22:08 UTC

Update

All tests are passing and devices are stable. We'll keep monitoring.
Posted Sep 10, 2025 - 21:26 UTC

Update

We are continuing to monitor for any further issues.
Posted Sep 10, 2025 - 21:25 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Sep 10, 2025 - 21:19 UTC

Identified

Most devices have reconnected, but we're still experiencing an elevated level of device SSH errors and are currently looking into the issue. Connection that goes through are slow. This is also affecting the public URL feature.
Posted Sep 10, 2025 - 19:29 UTC

Update

Most devices have reconnected, but we're still experiencing an elevated level of device SSH errors and are currently looking into the issue. Connection that goes through are slow. This is also affecting the public URL feature.
Posted Sep 10, 2025 - 18:55 UTC

Update

It appears that the cloudlink status of many devices didn't update when disconnecting.
If your device appear connected to cloudlink for more than 1h, its status is wrong and it has not yet reconnected (it will happen soon)
We're still monitoring the recovery.
Posted Sep 10, 2025 - 15:46 UTC

Update

During the incident a lot of devices disconnected from cloudlink.
They're slowly but steadily reconnecting.
We are monitoring the recovery.
Posted Sep 10, 2025 - 15:37 UTC

Update

We are continuing to monitor for any further issues.
Posted Sep 10, 2025 - 14:34 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Sep 10, 2025 - 14:25 UTC

Investigating

We're experiencing an elevated level of API errors and are currently looking into the issue.
Posted Sep 10, 2025 - 13:47 UTC
This incident affected: API, BalenaOS Download, Dashboard, Device URLs, Cloudlink (VPN), and balenahub.