Impact
Users were unable to access their devices via SSH through the web-terminal or the CLI. The CLI would return an error:
user does not have permission to access device
An error in our Renovate configuration allowed our automation system to merge and deploy an unintended backend component to production.
Resolution
Our team quickly identified the issue and:
- Immediately rolled back the component to the previous verified version
- Restored remote SSH access for all affected devices
- Corrected the Renovate bot configuration to prevent similar automatic deployments
Response
- Enhanced deployment controls: We've restored our automation configuration to ensure all components must pass manual review before production deployment
- Improved monitoring: We're considering implementing additional alerts to catch similar issues faster
- Process review: We're reviewing our automated deployment processes to identify other potential gaps
We apologize for the disruption and appreciate your patience as we resolved this issue. If you continue to experience any problems, please contact our support team.