We've recently experienced some instabilities with our Cloudlink service which impacts: webterminal, ssh, tunnels, actions, and public-url.

We host our Cloudlink servers along with some other parts of the balena-cloud infrastructure on a Kubernetes cluster.

Kubernetes scale and reorganize those servers depending on the load.

When a Cloudlink server needs to be shut down, connections are drained, this is expected and in most cases, transparent as devices reconnect to another Cloudlink server automatically.

What happened lately, and is problematic, is our Kubernetes cluster moving Cloudlink servers on hosts that are not able to sustain them at that moment. Causing another move shortly after, while devices didn't have the time to reconnect.

Hopefully, only a small amount of devices are concerned as only a few servers are moved at any given time, but it's still a very big problem for the customers who own those devices.

We already deployed some remediation to reduce the time it takes to move one of those servers and we're working on multiple different solutions to continue to both reduce the recovery time and reduce the occurrence of such issues.