On February 26, 2026 at approximately 19:00 UTC, users began experiencing failures when connecting to their devices via balena device ssh. The command returned a generic "host error" with no further detail. The issue affected all CLI-based SSH access through the balena proxy.
Web terminal access and balena device tunnel remained functional throughout the incident and were available as workarounds. Devices themselves were online and healthy; the failure occurred at the authentication stage between the proxy and the device.
Root cause
The balena proxy runs several internal processes. As a resource-saving measure, the service that handles web terminal connections had been disabled in the proxy pods dedicated to CLI SSH, since those pods do not serve web terminal sessions. What was not recognized at the time was that this service also carried the responsibility for loading SSH identity keys into the shared SSH agent used by the proxy. With it disabled, the proxy pods had no keys loaded and could not authenticate with devices, causing every SSH connection attempt to fail at the authentication stage.
Resolution
The service that handles web terminal connections was temporarily re-enabled in the SSH proxy pods while a permanent fix was being developed, restoring SSH key availability and resolving the authentication failures. A permanent fix has since been deployed.
Follow-up
We have decoupled the SSH key loading from the service that handles web terminal connections so that it is an independent initialization process in the proxy. We also added additional monitoring to ensure that SSH access through the proxy is adequately checked.
We sincerely apologize for the impact this caused. We are committed to improving our processes to prevent issues like these.