Elevated error rate on the API, Dashboard and Builders
Incident Report for Balena.io
Postmortem

Today we deployed foundational changes for the introduction of our organisations feature. These included database schema changes which introduced additional levels of indirection for permission queries. While we were expecting an increase in database load for these queries, the actual load was higher than anticipated leading to a percentage of API requests to be dropped.

After analysing the cause of the higher load we identified some key database indexes that we added to our database. Furthermore we optimised some of the problematic permission queries.

Posted Apr 09, 2019 - 17:39 UTC

Resolved
This incident has been resolved.
Posted Apr 09, 2019 - 14:33 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Apr 09, 2019 - 14:00 UTC
Investigating
Unfortunately our previous fix has not fully resolved the issue. We are continuing to investigate.
Posted Apr 09, 2019 - 12:48 UTC
This incident affected: API, Application Builder, and Dashboard.