When a device is connected to our VPN pool it gets assigned to an instance which we then store in our database so that we can resolve future device URL requests. When a device URL comes in it hits a VPN instance at random which checks if the device is connected to itself. In the case the instance does not have the device connected on to itself it would get the correct instance from the database and forward the request there.

Under the right circumstances the server to forward the request to could be the instance itself, causing a never ending infinite loop which lead to a large number of API requests and network traffic.

On the day of the incident we implemented a short term fix immediatelly and started developing a longer term fix. The long term fix was deployed on April 7th.