Sections:Fails

 

Dear SureDesk™ Users,

At 11:15 am ET on June 13, 2017 we identified an overload on our File Server.  We immediately messaged users requesting an emergency log off from the server at 11:30 am ET.  All users were fully logged off by 11:45 am ET.  We then performed an emergency reboot of the File Server to resolve the issue.  When rebooting the server a small but critical service on the system that handles user profile data, such as personal documents and settings like shortcuts, mapped drives and internet settings, etc., did not come back up upon server reboot.  We immediately escalated to our Amazon Engineer team to resolve the issue.  

Our engineers restored the server and all services were restored at 1:45 pm ET with no data loss.  Additionally we learned how to recover 4x faster from such an outage in the future.  

We strive for better than 99.9% uptime and this falls within that SLA.  This was the first all user SureDesk outage we have experienced.  

In a stranger than fiction irony in the middle of this exerpience we were doing emgency resuce of the large majority of users from another cloud service provider that suffered a catastrophic server loss in their data center.  All of this reinforced our greatest strengths in our focus on building the most resilient cloud IT network possible - but also overstretched our HelpDesk and our team was not able to message users as quickly as we would like during this outage.  We have already put updates in place to improve this messaging and updates to customers as well.  

 

If you have any questions, please email This email address is hidden from email harvesters via JavaScript  or call 1-800-882-8701 x1.

 

Thank You,

- Your SureTech™ Solutions Team

blog comments powered by Disqus