Facebook Provides More Details on How Monday’s Outage Happened

Facebook provided more details on the outage that crippled its platform and internal systems for more than six hours Monday, leaving media buyers reeling, as well. Vice president of infrastructure Santosh Janardhan reinforced the company’s claims late Monday that foul play was not involved, saying that the issue was triggered by the system that manages Facebook’s global backbone network capacity, or the network that connects all of its computing facilities, made up of tens of thousands of miles of fiber-optic cables worldwide and linking all of its data centers. Janardhan explained that routers determine where to send incoming and outgoing data, and engineers often need to take part of the backbone offline for routine maintenance tasks such as repairing fiber lines, adding capacity or updating router software. He wrote, “This was the source of (Monday’s) outage. During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all of the connections in our backbone network, effectively disconnecting Facebook data centers globally. Our systems are designed to audit commands like these to prevent mistakes like this, but a bug in that audit tool prevented it from properly stopping the command.” The complete disconnection of Facebook’s server connections between its data centers and the internet caused its Domain Name System servers to become unreachable, even though they were still operational, meaning that the rest of the internet could not find its servers. Janardhan wrote, “All of this happened very fast. And as our engineers worked to figure out what was happening and why, they faced two large obstacles: First, it was not possible to access our data centers through our normal means because their networks were down, and second, the total loss of DNS broke many of the internal tools we’d normally use to investigate and resolve outages like this.” It took Facebook engineers some time to access the physical servers due to security measures at the company’s data centers, making it difficult for them to both access the facility and modify the servers. Janardhan concluded, “Helpfully, this is an event we’re well prepared for thanks to the ‘storm’ drills we’ve been running for a long time now. In a storm exercise, we simulate a major system failure by taking a service, data center or entire region offline, stress testing all of the infrastructure and software involved. Experience from these drills gave us the confidence and experience to bring things back online and carefully manage the increasing loads. In the end, our services came back up relatively quickly without any further systemwide failures. And while we’ve never previously run a storm that simulated our global backbone being taken offline, we’ll certainly be looking for ways to simulate events like this moving forward.”

Platforms

Facebook Provides More Details on How Monday’s Outage Happened

A command to assess the availability of global backbone capacity unintentionally took down all connections in its network

David Cohen

About

Subscriptions

Events

Publications

WORK SMARTER - LEARN, GROW AND BE INSPIRED.

Subscribe today!

David Cohen