On Tuesday, February 28th, 2017, an Amazon employee made a typo during a routine billing system repair. The repair should have taken only a few servers offline. Instead, it took down a significant number of the nearly 150,000 websites and apps on Amazon’s S3 cloud service. The exact number is still unspecified.
Affected sites and apps across the web
Around noon EST, users noticed that many websites were unavailable or inconsistently responsive. Those sites included Netflix, Spotify, Pinterest, Buzzfeed, Slack, Github, Trello, Venmo, Quora, Sailthru, Business Insider, and Giphy. But the downtime didn’t end there. Amazon also supports many “Internet of Things” home devices, so Nest-controlled thermostats and other connected appliances went offline too. Amazon resolved the issue after a four-hour window. However, its own monitoring website ran on an impacted server, so accurate information spread unevenly.
A harbinger of issues to come?
Amazon Web Services (AWS) is the leading cloud service provider. It currently holds 40% of the worldwide market share of cloud services, serving clients from individuals to multinational corporations. AWS accounts for 8% of Amazon’s total yearly revenue. When that much data runs through one hub, the risk of disruption is always present. Amazon did not immediately release the cause, but it turned out to be simple human error.
More redundant servers and services can protect against bugs in code, hardware failures, or even weather-related issues. They won’t alleviate the risks that come when people get involved. Barring the impending perfection of artificial intelligence, no one can eliminate humans from technology. We can only minimize the potential for disasters.
Amazon addressed the incident in their statement. The company said it is “making several changes as a result of this operational event.” It explained that its tool removed too much capacity too quickly. Then it added, “We have modified this tool to remove capacity more slowly and added safeguards.”
Cautions and positivity
There are encouraging results of this unfortunate and costly outage. Amazon used other, unaffected channels, namely Twitter, to share the extent of the problem and its progress toward resolution. This incident also highlights the real risks that come when a company employs only one cloud-based file system. Even when distributed physically, systemic infrastructure can still be a hazard. The solution might be to disperse files among various cloud sharing systems entirely, from AWS to Google and others. Making this type of move can be prohibitively expensive for most companies. However, the costs of lost revenue during downtime can be just as prohibitive, as this outage showed.
Want to See How Volico Data Center Can Help You?
An incident like this could hit your business too, so now is the time to plan for such a disruption. Contact Volico at (305) 735-8098. A member of our team is standing by to help you decide which solutions would benefit your IT operations.
Ready to See How Volico Data Center Can Help You?
Got questions? Want to talk specifics? That’s what we’re here for.
Have one of our friendly experts contact you to begin the conversation. Discover how Volico can help you with your Cloud Hosting needs.
- Call: (305) 735-8098
- Chat with a member of our team to discuss which solution best fits your needs.
