Someone Broke The Internet: Major Network Issue

September 2, 2015

On Thursday, August 27th, 2015, Volico started receiving calls from clients regarding a network issue around 2:55PM EST. Our engineers immediately began investigating the problem and escalated it to top priority.

Multiple eBGP sessions were flapping across several providers. Volico immediately began opening upstream support tickets to determine whether the issue affected only South Florida. An upstream provider alerted us that a major Tier 1 provider was currently causing routing issues across North America. Multiple Tier 1 providers were also showing eBGP Prefix List fluctuations, confirming the eBGP flapping that Volico was seeing.

Further investigation using www.downdetector.com showed that this issue was not limited to South Florida. It was, in fact, a nationwide event. Volico began taking measures to limit routing to carriers that did not appear to be affected. During mitigation, Volico noticed that eBGP sessions began showing stability across the board around 3:25PM EST, and networks across the United States began to stabilize.

What Caused the Outage

In conclusion, the event appears to have three major ROOT causes.

1) Level3 conducted a network configuration change in the Chicago Area causing a nationwide instability in the Level3 Network.

2) Time Warner Cable was also performing a maintenance window that also caused a network outage and fluctuation of traffic across their network further saturating other Tier 1 providers.

3) Century Link conducted an emergency shutdown of one of their Core facilities in the New York area due to a Catastrophic Cooling system shutdown causing a major Century Link Network hub to go offline and further reroute traffic to other Tier 1 providers.

By this time, the network instability had spread across the entire United States.

We have provided images below from www.downdetector.com showing the network outage heat maps affecting every major population zone across America. Please note that these are only a few of the many carriers this event affected.

Everyone here at Volico would like to thank its customers for their help and dedication during the event.