Case of the Mondays: Facebook, Instagram, WhatsApp, Other Apps Down for 6+ Hours

Facebook, Instagram, WhatsApp and other properties both public and internal were nonexistent for the bulk of Monday, with Downdetector initially reporting the outages at 11:40 a.m. ET. Parts of the platforms appeared to be returning around 6 p.m. ET, and Facebook confirmed in a tweet at 6:33 p.m., “To the huge community of people and businesses around the world who depend on us: We're sorry. We’ve been working hard to restore access to our applications and services and are happy to report they are coming back online now. Thank you for bearing with us.” UPDATED: Vice president of infrastructure Santosh Janardhan went into further detail in a blog post published late Monday. He wrote, “Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.” Addressing speculation about foul play and concerns about the integrity of user data, he added, “We want to make clear at this time we believe the root cause of this outage was a faulty configuration change. We also have no evidence that user data was compromised as a result of this downtime.” Forrester vice president and principal analyst Jeff Pollard said in an email, “Facebook must amplify the mundane truth that a configuration change prevented communication between data centers, over the conspiracy and rumors that will persist. This should also spur organizations to take a long look at their processes and identify ones that require technology—even once unthinkable technology failures—because this outage demonstrates that the unthinkable happens. News of taking an angle grinder to server cages because authentication systems stopped functioning delayed recovery. The phrase, ‘Well, if that happens, we have much bigger problems,’ is sometimes used in tabletops and simulations to avoid dealing with catastrophic, systemic failures, and it’s valid sometimes—just not all of the time, as demonstrated here. Don’t reject the unthinkable just because it’s unlikely.” However, it hardly seems like a coincidence that Facebook’s longest outage in two-and-a-half years struck at the same time the company is dealing with fallout from internal documents published by The Wall Street Journal, the appearance by “Facebook whistleblower” and former product manager for the civic misinformation team Frances Haugen on CBS’ 60 Minutes and testimony on Capitol Hill by Facebook global head of safety Antigone Davis. Prior to Facebook's tweet at 6:33 p.m., the last official communication from Facebook was a tweet from outgoing chief technology officer Mike Schroepfer, who said at 3:52 p.m. ET, “Sincere apologies to everyone impacted by outages of Facebook-powered services right now. We are experiencing networking issues, and teams are working as fast as possible to debug and restore as fast as possible.” Forrester vp and research director Mike Proulx said in an email, “Yesterday’s Facebook outage wasn’t the first, and it won’t be the last. It’s a reminder to advertisers to have proactive mitigation plans in place to avoid the scramble of trying to figure out what to do in the moment. Although many brands had a ‘real-time marketing’ moment on Twitter, this outage has widespread implications to the advertising ecosystem, given the fact that ads weren’t being served for over six hours across Facebook and Instagram, which command the lion’s share of social media ad revenue. This not only affects Facebook’s revenue, but also brands’ bottom lines.” Indeed, based on Facebook’s second-quarter-2021 earnings report, the company averages $13 million worth of ads per hour, or roughly $200,000 per minute. Monday’s issues marked the longest outage for Facebook since March 2019, when Facebook, Instagram and WhatsApp were hampered or completely inaccessible for roughly 24 hours. Speculation around the industry pointed to an issue related to the Domain Name System, with security expert Brian Krebs saying in a tweet Monday afternoon, “Confirmed: The DNS records that tell systems how to find http://Facebook.com or http://Instagram.com got withdrawn this morning from the global routing tables. Can you imagine working at Facebook right now, when your email no longer works and all your internal Facebook-based tools fail?” Forrester principal analyst Renee Murphy said in an email, “It is rumored to be a Border Gateway Protocol problem. This is something worth noting. It might be really hard to get into a router and change the route tables from outside of the organization, but inside, it's a piece of cake. If it really is a BGP problem, that is a big problem. This is not something that should ever happen at a company this sophisticated with this much data. Risk is everywhere, even in the networking protocols.” The outage impacted the 200 million businesses that use Facebook’s platform and tools to connect with consumers and promote their goods and services, as well as over 1 billion people who send 10 billion messages between consumers and businesses on Messenger From Facebook every month. Virtual reality platform Oculus From Facebook and enterprise platform Workplace From Facebook were also felled by the outage, as were internal tools at the company, with Ryan Mac, a tech reporter for The New York Times, saying in a tweet, “Not only are Facebook's services and apps down for the public, its internal tools and communications platforms, including Workplace, are out, as well. No one can do any work. Several people I've talked to said this is the equivalent of a ‘snow day’ at the company.” Forrester senior analyst Alla Valente said in an email, “In Facebook’s quest to integrate its products and underlying technical infrastructure into a single platform is the concentration risk it creates for the company, where a single risk event that produces a cascading effect—in this case, the inability of their machines to talk to one another—brought the company to a standstill. Concentration risk is one of the top systemic risks for 2021 that Forrester identified early this year. And Facebook’s size, market share and ubiquity make it a system in to itself. If the company doesn’t get better at managing its risks across the organization, it stands to lose the tight hold it’s been struggling for years to maintain.”

Platforms

Case of the Mondays: Facebook, Instagram, WhatsApp, Other Apps Down for 6+ Hours

UPDATED: Facebook denies foul play was involved, claims no user data was compromised

David Cohen

About

Subscriptions

Events

Publications

WORK SMARTER - LEARN, GROW AND BE INSPIRED.

Subscribe today!

David Cohen