We found that delta images were not being generated by our delta servers since 2022-04-18 19:08 UTC. Further investigation revealed that some delta image records in the database are causing requests to fail because of duplicate delta image records for successfully generated deltas. We modified the duplicate records to indicate that only the valid ones were successful. This allowed us to restore the service and allow new deltas to be generated.
We did another review of the delta image records and found that only a single record was causing the issue. We reverted the status of the other modified records to their original state. This does not affect the service in any way. We are still investigating how a single record had a duplicate status and how we can avoid this from blocking the delta generating service in the future.
Posted Apr 19, 2022 - 16:01 UTC
Resolved
The errors have been resolved but we are still investigating the cause of the problem. The image delta records we found earlier were identified to be valid so we are still investigating why modifying those records resolved the issue.
Posted Apr 19, 2022 - 07:11 UTC
Monitoring
We found some duplicate delta image records in our database that were causing the errors earlier. We corrected these records and have seen the error rate drop since. We still need to investigate how these duplicate records were created and are currently monitoring if the problem will occur again.
Posted Apr 19, 2022 - 06:41 UTC
Investigating
We are currently investigating an issue that prevents new delta images from being generated. Delta images generated before `2022-04-18 19:08` are still downloadable by devices.