Note: This article was originally published on 04/10/2021 at 17:40, and was updated on 05/10/2021 at 14:55 to discuss the resolution.
At around Monday 04 October, 17:00 UK time, facebook.com, messenger.com, oculus.com and other Facebook apps and services were not accessible. Facebook has released the following statement via Twitter:
We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.
— Facebook (@Facebook) October 4, 2021
Update 17:47 – Instagram 5XX Error
Instagram DNS is now resolving. All other Facebook sites are not reachable.
WhatsApp, Oculus and Instagram have echoed similar updates on Twitter.
We’re aware that some people are experiencing issues with WhatsApp at the moment. We’re working to get things back to normal and will send an update here as soon as possible.
— WhatsApp (@WhatsApp) October 4, 2021
Thanks for your patience!
Instagram and friends are having a little bit of a hard time right now, and you may be having issues using them. Bear with us, we’re on it! #instagramdown
— Instagram Comms (@InstagramComms) October 4, 2021
As a result of the outage, Oculus users are finding that they cannot access online features of their devices.
We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience.
— Oculus (@oculus) October 4, 2021
Back online
Shortly after 11:30PM UK time, all affected Facebook services started to become functional again. This marks the end of an approximate 6 hour downtime.
Facebook and services have made announcements on rival platform Twitter once again to confirm this. Mark Zuckerberg, CEO of Facebook, Inc., said the following on his Facebook page:
Facebook, Instagram, WhatsApp and Messenger are coming back online now. Sorry for the disruption today — I know how much you rely on our services to stay connected with the people you care about.
Source: https://www.facebook.com/zuck/posts/10113957526871061
What caused the outage?
Initially, speculation suggested DNS was to blame for the outage. Domain Name Service is the service that converts what you type into an address bar, like facebook.com into IP addresses that represent where the site is stored on the Internet. For a technical breakdown, you can see this Cloudflare Learn resource. And whilst this holds some truth, the stem of the issue is down to changes Facebook made to their Border Gateway Protocol configuration. The BGP protocol is a routing protocol that advertises information on how to reach a server to other routers. In simple terms, it helps build a map of the Internet and tells routers where things are. Facebook made changes to their interior BGP configuration. An error in these changes meant that BGP was not advertising routes, and technically speaking, Facebook and it’s services did not exist as far as the Internet was concerned. This lead to DNS failures when resolving Facebook websites.
Due to increased and repeated DNS requests, some DNS servers found themselves overloaded. AdGuard DNS, a popular ad-blocking DNS went briefly down as a result.
Cloudflare have posted an excellent writeup of the cause of the outage on their blog.
We just had a serious outage of @AdGuard DNS, but it was actually caused by @Facebook. What happened and how on earth @AdGuard may depend on FB? Let me try to explain. (1/9)
— Andrey Meshkov (@ay_meshkov) October 4, 2021
A long wait
Website outages are common. But for a site like Facebook, with over 2 billion users, longer outages like these are less expected. The wait, in part, was reportedly caused by Facebook employees being unable to get into the building, as their own smart control systems also went down. Further reports also suggested that internal communication had to be done via Microsoft Outlook and text message too. This likely added to the delays in fixing the issue.
From bad to worse
Facebook’s reputation already was in hot water after a whistle-blower who made damming claims about the social media giant revealed her identity and gave an interview to CBS News’ 60 Minutes show. Today, Frances Haugen is testifying to the US Senate. This, along with the downtime, lead to a large drop in $FB stock and it’s CEO’s personal net worth, with Mark Zuckerberg loosing $7bn.
Facebook will be hoping no more issues arise in the near future, although given recent press, this is likely not to be the case.