Elevated Device URLs/VPN Errors
Incident Report for Balena.io
Postmortem

balena devices that were connecting to the VPN were unable to report their VPN connectivity status to balenaCloud. This was because the auto-incrementing ID of a database table related to device connectivity had reached its limit, which prevented the balenaCloud backend from recording new connectivity events. Devices that were already connected to balenaCloud were not affected.

To address this issue, we performed emergency database maintenance to change the data type of the column and allow for a larger limit on the auto-incrementing ID. We also checked other database tables to determine which columns need modification and will schedule a separate maintenance activity to preemptively modify these tables.

Posted Mar 20, 2023 - 14:05 UTC

Resolved
This incident has been resolved.
Posted Mar 17, 2023 - 04:49 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Mar 17, 2023 - 04:37 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Mar 17, 2023 - 00:56 UTC
Investigating
We're experiencing an elevated level of errors in our Device URLs and VPN infrastructure and are currently looking into the issue.
Posted Mar 17, 2023 - 00:13 UTC
This incident affected: Device URLs and Cloudlink (VPN).