We rescheduled our Proxy service onto another server causing it to restart. The traffic generated by reconnecting devices overloaded the service. Users were unable to SSH into their devices, and connections via device URLs were also unavailable.
We released an optimization for the service which fixes the issue. The optimization also prevents this situation from occurring again in the future. We are also checking if we can apply this optimization to our other services.
Posted Jul 28, 2020 - 13:54 UTC
Resolved
This incident has been resolved.
Posted Jul 28, 2020 - 11:43 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 28, 2020 - 11:13 UTC
Identified
The issue has been identified and a fix is being implemented. The Device URLs are also currently inaccessible.
Posted Jul 28, 2020 - 11:05 UTC
Investigating
We're experiencing an elevated level of device SSH errors and are currently looking into the issue.