Understanding how Facebook disappeared from the internet

https://blog.cloudflare.com/october-2021-facebook-outage/

by jgrahamc • 5 years ago

1226 253 5 years ago

blog.cloudflare.com

clemenspw• 5 years ago

Thanks, great write up!

throwdecro• 5 years ago

This is a great write-up, but one thing I don't understand is why the effect of withdrawing the BGP prefixes was instantaneous (if I understand that correctly), but it's taking hours (so far) to re-announce the prefixes. Why would it take so long to flip the switch back the other way?

dec0dedab0de• 5 years ago

I haven't been following closely, but I think once they moved the prefixes they could no longer access the routers. Coupled with barebones staff at the data center due to the pandemic, and all internal communication being disrupted. Though I really expected it to be up within an hour or two.

dr_orpheus• 5 years ago

Yeah, I think that is true. If you look at the Update near the end of the Cloudflare article there is a huge spike in the BGP activity (I assume re-announcing all of the routes). So that part of it was relatively instantaneous after they got all of their ducks in a row actually getting to the routers and locating the BGP from some earlier version before it went offline this morning that they could use.

rifkiamil3• 5 years ago

We have had out-of-band management ports & networks design for decades! I know the feeling of driving 8 hours because I lost connection to the device I was configuring. https://en.wikipedia.org/wiki/Out-of-band_management

huuuz• 5 years ago

Reduced staff because of the pandemic? 18 months in? This looks like a sweet deal for those workers more than anything else.

Wolfspirit5• 5 years ago

I'm not sure if that is true (and I hope it is not cause that would be fatal) but I read somewhere that with facebook being down also means all internal infrastructure of facebook isn't available at the moment (chats, communication) including remote control tools for the BGP Routers. Therefor they require people to get physical access to the router while many people are working from home cause of the pandemic.

rifkiamil3• 5 years ago

Thats why we have patten called https://en.wikipedia.org/wiki/Out-of-band_management

dirkg2• 5 years ago

also people's badges stopped working and they were unable to get physical access to sites.

kube-system• 5 years ago

Given my experience with DNS issues, I am guessing that they are running into dependencies along the way that assume/require DNS be available to function.

theshadowknows• 5 years ago

Yeah as part of my job I often have to work with our DNS team to provision say a subdomain or get some domain verified. They’ve got like…three people…trying to service thousands of teams across the enterprise. I do not envy their job at all.

bink• 5 years ago

With routing it's even worse than that. If they had no out-of-band method to connect to these routers and they botched the routing config then they had no way to route any traffic to them at all. At least with DNS you can still connect to the IPs.

I would find it a bit surprising if Facebook didn't have OOB access to their data centers, however.

adamcharnock• 5 years ago

I’m pretty new to BGP, but I’d imagine that cutting off access to an AS is fast because all it takes is for the neighbouring routers update their routes. At which point any traffic that makes it that far is simply dropped.

Whereas to make an announcement, the entire internet (or at least all routers between the AS and the user) need to pickup the new announcement.

(Note: I still need to read the article)

bsedlm2• 5 years ago

(I'm trying to better understand this)

I think it's not so simple because authoritative DNS systems are involved.

So it's not just a BGP error. It's a BGP error which disconnected authoritative DNS for all facebook. I'm not quite sure why that makes it so slow to fix. is it just because internal difficulties due to having no DNS at all?

Hikikomori4• 5 years ago

Once they start adversiting again it should only take a few minutes at most for most ISPs to get it.

PeterCorless• 5 years ago

If an authoritative DNS entry was removed, it can take up to 72 hours for that change to be propagated around the world, though usually just a few hours for some other authoritative DNS systems to get you mostly back:

https://ns1.com/resources/dns-propagation#:~:text=DNS%20prop....

tester756• 5 years ago

Why it takes this long?

Hikikomori4• 5 years ago

Restoring is just as simple as flipping the switch again, but access to that switch is another matter when your internal network is also down and you cannot even get access to your office or datacenters.

cyanydeez3• 5 years ago

engineers will tell you that not everything is reversible even if theres no specific cqpacity issue

foobarian• 5 years ago

What I don't get is why there is no dead-man switch mechanism in place to roll back the configuration automatically unless someone confirms it positively. Kind of how screen resolution rolls back if you don't ack it. I used to always run a "(sleep 600; iptables -F) &" when messing with remote personal stuff just in case I lock myself out.

I suppose with something like BGP it would be very difficult to get such a fallback working given how distributed the system is, and even more difficult to keep it exercised and tested.

gorgoiler• 5 years ago

At a guess: reconnecting traffic at the billions-of-people scale has the potential for finding all sorts of weird behaviours. For all we know, it has been connected and disconnected 10x already during the outage, with each reconnect overwhelming some new, deeper level of the system each time.

Reconnect and the GLBs fall apart under load as the entire world’s cadre of recursive resolvers hit you.

Fix that. Reconnect again. This time your LBs have marked half the servers as offline because their heartbeats have been failing.

Fix that. Reconnect again. Now all the memcache data is hours old and so the site business logic fetches straight from databases, knocking them over.

Fix the databases. Reconnect again. Ad nauseam.

Godel_unicode3• 5 years ago

I find this kind of uninformed conjecture amusing on a thread full of people complaining about cloudflare doing the same (they didn't). There's no evidence of this kind of flapping behavior in any of the telemetry I've seen posted by network engineer friends, and the blog post explicitly calls out when they saw the BGP updates that brought the site back online.

acomjean• 5 years ago

Reminds me of when I was working at a startup we DOSed ourselves. We had devices that monitored power minute by minute. A load balancer was mis configured and we went down for a some hours. We came back up and the devices all saw that and flooded us with all the data they’d been storing since we were down.. down again… bring it up and down again…. we needed a better fix. I was up till 2 am with the other developer coding a fix.. the next morning we talked to the CEO (CTO was on vacation)who told us upgrading the was database was going to be too expensive… good times. We did get a firmware fix ( to be honest we were running out of cash..)

I can’t imagine trying to restart something as big as facebook…

nimbius• 5 years ago

Unpopular opinion but I think a talent exodus and turnover are likely at fault here as well. The people who stood up this juggernaut are no longer here, and Facebooks ability to consistently attract talent that is competent enough to maintain such a formidable beast is hampered by repetitive revelations that it operates at the net-loss of humanity as a whole.

Facebooks no longer an innovator, just a mining operation with a dwindling population of hateful elderly and bots.

bsedlm2• 5 years ago

I'd say the same thing about Google.

tjpnz• 5 years ago

I'm sure the people who go to work for Facebook are fully aware of what they're getting into.

goatlover• 5 years ago

Isn’t Facebook huge in some Asian countries? Supposedly 3.5 billion people use one or more of its services. WhatsApp, Instagram and Oculus certainly aren’t just used by the elderly.

termau• 5 years ago

I feel like it just confuses the issue with a bunch of unnecessary babble about DNS, theres better ways to read about how BGP works without confusing a bunch of different things. The only part of the article that was relevant was 'Routes were withdrawn' - the rest being a consequence of that.

foolfoolz• 5 years ago

fast off board slow onboard is a pattern you find all over. it has roots in fraud but there’s many reasons. getting more access is a privilege escalation and requires some trust to achieve

cortesoft2• 5 years ago

You have to be careful turning something as large as Facebook back on. If you turn on announcements one place first, the entire internet will try to reach you through a single transit and overwhelm it.

dijit• 5 years ago

The kind of tail you’re talking about is baked into DNS at least.

I don’t know enough about BGP to make an informed decision; but at the point the outage is noticed it’s entirely possible that the system has been unavailable for quite some time already.

belorn• 5 years ago

It just a guess, but from experience with BGP and associated redundancy systems that most likely was in place, if everything doesn't return immediately then you have a big fight on your hand to not only stop the non-functional redundancy but also reestablish the peer connections with associted hearthbeat/processes for establishing and maintaining the peer connection. My understanding from what people write about configuring BGP and the system around it seems to imply that the best practice in this circumstance is to kill everything, fix the original error and then turn on things slowly again. Then fix the broken redundancy configuration. Then test the redundancy system regularly in the future.

nabaraz• 5 years ago

Is there a way to secure BGP? Some kind of BGP table backup and restore?

Wolfspirit5• 5 years ago

I think that it is possible to restore the previous state but the question is, if it makes sense. When do you decide that it was a failure? Facebook explicitly (even so automatically) told all others that they shouldn't use that routes anymore.

When it comes to Facebooks side I guess they do have backups of their BGP config. Applying them (probably remotely) however seems to be harder then expected when the whole infrastructure is down.

cyanydeez3• 5 years ago

reading krebs suggest they couldnt do anything because all the security ran through the same computers keeping facebook.com

they basically ran into a footgun

mnordhoff3• 5 years ago

"Because of this Cloudflare’s 1.1.1.1 DNS resolver could no longer respond to queries asking for the IP address of facebook.com or instagram.com."

The instagram.com zone itself uses a third-party DNS service and didn't go down. (But e.g. www.instagram.com is a CNAME to a zone on FB DNS.)

yftsui• 5 years ago

That’s pretty much why during the downtime visit instagram.com showed a 503 from AWS instead.

mschuster91• 5 years ago

Wonder why they're still using AWS given that FB operates its own data centers...

jgrahamc4• 5 years ago

Thanks. I'll correct that.

AlbertCory2• 5 years ago

It sorta sounds like Facebook is Too Big To Fail.

Yet another reason to dismantle it.

literallyWTF2• 5 years ago

I would rather purge it from this planet and throw Zuck into jail.

literallyaduck• 5 years ago

Is it harmful?

https://www.theguardian.com/technology/2021/sep/29/facebook-...

Did Facebook lie to congress?

https://www.forbes.com/sites/zakdoffman/2019/03/02/facebook-...

Also short comments without discussion and evidence do poorly.

I believe it should be halted for public safety which is well within the rights and capacity of our government. I believe a fair trial should happen before anyone receives punishment, but I don't believe we could find an impartial jury as Facebook is ubiquitous.

threeseed3• 5 years ago

This comment makes no sense.

Not everyone uses Facebook services and the rest of the internet worked just fine.

AlbertCory2• 5 years ago

Actually, no, the rest of the internet was pig-slow.

robaato• 5 years ago

From article: [This chart shows] the availability of the DNS name 'facebook.com' on Cloudflare's DNS resolver 1.1.1.1. It stopped being available at around 15:50 UTC and returned at 21:20 UTC.

Wolfspirit5• 5 years ago

Seems like they're back online

ruoso• 5 years ago

... but as of 22:28 UTC Facebook appears to be ...

Someone assumed London==UTC, when London is 1 hour ahead :) that was actually 21:28 UTC

Wolfspirit5• 5 years ago

Timezones are the most annoying thing... right after encoding

doublerabbit• 5 years ago

right after encoding

No joke. Today I ended up writing a whole essay explaining the issue I was having and almost sending it off to the core developers because I thought I had discovered an issue with the actual language. The bug was because I had forgot to convert too&from utf-8 in these two procedures:

    proc 2Hex { input } { binary encode hex [encoding convertto utf-8 "$input"] }
    ;# Converts base32 string data to base16

    proc 2Base { input } { encoding convertfrom utf-8 [binary decode hex "$input"] } 
    ;# Converts string hex data to base32

On the plus side, I now have written documentation of the internals of my program.

paxys• 5 years ago

I personally find timezones more annoying. At least with encoding once you figure things out it will work indefinitely. Timezones can simply change from under you with or without notice.

interestica• 5 years ago

No matter what time of year it is, people tend to use 'EST' for 'Eastern Time' even when we might be in Eastern Daylight Time rather than Standard.

It's especially annoying when dealing with multiple countries that may or may not be using Daylight Saving Time.

Wolfspirit5• 5 years ago

Even google isn't quite sure about the summer time. Not sure if that is just a Google German thing...

A few weeks ago I tried to find out what the current time in CET is. Asking google for "CET" gave me: "23:27 CET". Asking google for "CET time" (I know that "time" is twice in this case) gave me "00:27 CET".

The last one is wrong and should be CEST or even more correct would be just the same result for CET as I asked for

delecti• 5 years ago

I make a point to give local times in just "PT". Not least among my reasons is that I can't ever remember which half of the year is Daylight time and which is Standard.

gcbirzan• 5 years ago

Actually, the article seems to confuse the times quite a lot. It's talking about ~16:50 UTC at points, but the outage started at 15:40, which they not only mention in the article but you can also see on the graphs.

amachefe• 5 years ago

Facebook.com is up now, looks like the issue now is the billions of request that is DDoS on the DNS servers

tremon• 5 years ago

Why? Do you think there's billions of widely-used recursive resolvers in the world? Each resolver only needs to contact the Facebook DNS servers once per hostname, the end-user requests are all served from cache.

cwilkes• 5 years ago

Updating BGP configs should go through a flowchart like this:

Do you want to update BGP?

No: exit

Yes: type this random 100 character phrase to continue no copy paste

toomuchtodo• 5 years ago

“Are you sure? Can people you need to recover from this change already in the building?”

_joel• 5 years ago

Maybe a timed rollback with the previous state stored on the device that needs to be rolled back, althogh if you're doing this at facebook scale I'm sure that's a little more difficult than it sounds, perhaps.

cyanydeez3• 5 years ago

not just in the building, but are there any microsoft enabled security doors between you and them

rawoke083600• 5 years ago

or like an iptables or Windows desktop screen resolution change :)

alksjdalkj• 5 years ago

Question about the WARP map - I assume the grey countries are places where Cloudflare doesn't have any presence, but what about Egypt/Oman? Why are they green? And why is Australia orange and not red?

rswail• 5 years ago

Because most of the outage happened in the early hours of the morning AU time, so not that many people online.

davidjytang• 5 years ago

Also curious about the green part.

jgrahamc4• 5 years ago

It's an anomaly. I think that was right on the edge of "not enough traffic to report on".

jgrahamc4• 5 years ago

We didn't have enough data from those locations.

ormax3• 5 years ago

Egypt has over 100 million people. Facebook/WhatsApp is quite popular in that part of the world, so I would image it had significant traffic..

WaitWaitWha3• 5 years ago

This provides a good set of details (mostly educational) what happened up to but not including the how and why the BGP routes were withdrawn (who sent the UPDATE packets to the neighboring ASes?).

The most "natural" occurrence that I can think of is best-path change. If a "better" route between AS is added, the now-second-best routes are withdrawn.

Correct me if I am wrong, but there is no way of determining the source of the withdraw message (UPDATE)...

WaitWaitWha3• 5 years ago

Love to hear the reason for the downs.

asadjb2• 5 years ago

That info. would probably come from FB though, not Cloudflare. At least that's how I understand the incident.

WaitWaitWha3• 5 years ago

"Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. "

and

"... its root cause was a faulty configuration change on our end."

From https://engineering.fb.com/2021/10/04/networking-traffic/out...

robocat• 5 years ago

Also see Krebs’ article on the outage (although he hasn’t yet updated it after the problem was rectified): https://krebsonsecurity.com/2021/10/what-happened-to-faceboo...

Ansil8498• 5 years ago

Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation? First with the Twitter posts and now this blogpost. They're a passive observer, not involved in this event, and yet they keep putting themselves in the middle of it.

Cloudflare staff are not Facebook staff, they do not know how things actually went down to trigger the events that transpired. They are essentially doing glorified bikeshedding, and providing an explainer of how BGP works that can be gleaned by reading virtually any introductory text on it, and using it as a marketing opportunity.

missedthecue• 5 years ago

I mean... they're in the business of keeping websites online. It's natural that they document these events both for their own research and to market their product.

rjvir• 5 years ago

It's merely an informative blog post on a topic many people are interested in. I see nothing wrong with Cloudflare just explaining what happened, even though they had nothing to do with it.

Ansil8498• 5 years ago

I see nothing wrong with Cloudflare just explaining what happened, even though they had nothing to do with it.

You kinda inadvertently highlighted the issue: because they had nothing to do with it, they do not know what actually happened. They can pontificate about likely causes, just like others in the industry can, but they have no idea what actually caused the issue.

cortesoft2• 5 years ago

It is little stuff like this that makes it come off a bit self-aggrandizing:

We keep track of all the BGP updates and announcements we see in our global network. At our scale, the data we collect gives us a view of how the Internet is connected and where the traffic is meant to flow from and to everywhere on the planet.
Fortunately, 1.1.1.1 was built to be Free, Private, Fast (as the independent DNS monitor DNSPerf can attest), and scalable, and we were able to keep servicing our users with minimal impact.

It isn’t a big deal, and the posts are still interesting. It just makes me roll my eyes a bit.

samhwr• 5 years ago

It’s massively viral news about the exact stuff they specialise in. I’m not too surprised that they’d be putting out content. After all, Facebook is hardly in a position to do so ;)

Ansil8498• 5 years ago

It's not surprising, no, but it is irritating because it's just crass marketing.

Blackthorn• 5 years ago

No surprises here. They do it all the time for websites they cdn. Cloudflare might as well be a marketing company for how much they puff about themselves.

threeseed3• 5 years ago

Cloudflare is also incredibly dodgy with their billing practices which you wouldn't pick up from their marketing.

If you downgrade from a Paid to a Free account they automatically and silently add a paid add-on for 10 Page Rules.

wslack• 5 years ago

They seem to be treating it as a marketing opportunity, as well as explaining these topics for journalists. If their post didn't add to the public conversation, it wouldn't have been so viral.

fareesh• 5 years ago

As someone who is always interested in learning new things, their blog post is informative and helpful. Not everything is about optics, their intentions are of no interest to me, because I got some value out of the content.

dirkg2• 5 years ago

how are they 'interjecting' themselves? They run a massively popular dns service as well as many other sites, and they are simply reporting on what happened and why people who use their dns etc might have had outages. In addition to being a very detailed and well written article on the whole incident. Not really sure what you are driving at.

Ansil8498• 5 years ago

Not really sure what you are driving at.

At the fact that they are presenting themselves as authorities on the incident, when they have no internal knowledge of what triggered the events, because they are not Facebook engineers. They provide an explanation of BGP that you can glean by reading virtually any other introductory explainer, and turn it into a chance to promote their own service.

unityByFreedom2• 5 years ago

Not at all. It's interesting analysis. It's possible to bloviate in such a post but I don't see it here.

hinkley3• 5 years ago

They've done this in previous outages and people didn't like it then, either.

Generally big companies don't talk smack about other big companies having internet-wide issues, unless those issues are directly caused by the other company.

For instance, when Google talked about Cloudbleed, which was when Cloudflare vomited millions of secrets all over Google's caching heirarchy and Google had to manually clean it up.

I think perhaps the Cloudflare people have gotten confused and think that means it's okay to talk about other people's stuff. Instead of interpreting it as it really is, which is that Cloudflare is the last company that should be criticizing everyone else, lest someone bring up their previous missteps.

Ansil8498• 5 years ago

I still remember Cloudflare's PR efforts to downplay Cloudbleed. A key reason why I meticulously avoid using any of their services.

Godel_unicode3• 5 years ago

Given that warp is a pretty major player in the VPN space and lots of their customers are likely to blame them for not being able to get to Facebook, I think having a detailed "wasn't us" on their blog that their sales engineers can point to is reasonable.

jgrahamc4• 5 years ago

which is that Cloudflare is the last company that should be criticizing everyone else

There is no criticism of Facebook in our blog post.

xtat2• 5 years ago

Cloudflare are also good at disappearing people from the Internet

BFG-One7• 5 years ago

I don't trust any analysis from CloudFlare. These are the same people preventing people with vpns or tor browsers from reaching sites while screaming and yelling OMG DDoS!! then blaming the customer of CloudFlare for not knowing how to configure it.

DantesKite• 5 years ago

No. I don’t mind at all. It’s a great marketing opportunity and they would be stupid not to take advantage of it.

Ansil8498• 5 years ago

Totally. Just like it's weird that more cigarette companies don't advertise outside schools.

_aj8o• 5 years ago

So ... cloudflare has inserted themselves far more deeply into the discussion awhile back.

With the new DoH Chrome setting turned on by default (DNS-over-HTTPS), AFAIK they are now the default resolver bypassing your network settings.

[1] https://developers.cloudflare.com/1.1.1.1/encrypted-dns/dns-...

[2] https://duo.com/decipher/google-makes-dns-over-https-default...

NikolaNovak2• 5 years ago

I kinda see your perspective; but I also mostly view them similarly to e.g. BackBlaze blogs - sure it can be viewed as taking smack about Samsung and Seagate drives failing, but I seem them as "things fail; what can we learn from it / here's our observations". And yes a little bit of self-advertising, but again, most tech blogs are, one way or another.

Ansil8498• 5 years ago

It just comes off as crass. Next time Cloudflare's CDN eats itself and starts vomiting up private customer data, Facebook can do a blogpost titled 'Understanding how Cloudflare exposed the private information of untold numbers of its customers'.

dilyevsky• 5 years ago

Probably re-enabling it gradually to avoid hammering DNS servers too much due to client caches being totally cold

MrStonedOne2• 5 years ago

They talk about what their services observed related to bgp traffic from facebook. They are an authority on that.

They talk about their dns traffic changes from facebook's outage. They are an authority on that.

Lazare2• 5 years ago

They are essentially doing glorified bikeshedding, and providing an explainer of how BGP works that can be gleaned by reading virtually any introductory text on it, and using it as a marketing opportunity.

I mean, I'm not really sure what the purpose of a corporate blog is except that? You make posts about whatever will garner attention in order to get some views, maybe spread some information, and - of course - turn it into a marketing opportunity. That's the job, no?

Is anyone else kinda put off by how Cloudflare keeps interjecting themselves into this situation?

Personally? No. Although amusingly, a non-technical friend of mine took the twitter posts they made as a sign that Cloudflare had caused the outage somehow, so it's certainly possible there's a risk there.

And from one of your other comments:

Next time Cloudflare's CDN eats itself and starts vomiting up private customer data, Facebook can do a blogpost titled 'Understanding how Cloudflare exposed the private information of untold numbers of its customers'.

Yes, they should totally do that (at least if they have anything informative to contribute, as I think Cloudflare does here). Why would this be a bad thing, or a reason not to talk about Facebook's issues? And I mean, at one point today something like 6+ of the top 10 links on HN were about Facebook, so I mean, everyone else is talking about them. Why not Cloudflare? And if and when Cloudflare has their next big issue, everyone will be talking about them either way.

sudhirj• 5 years ago

The CEO recently pointed out on Twitter that the primary purpose of the company blog was hiring, and that the company would write with that in mind. The post taught me something I didn’t already know, and left me more impressed than I was before with Cloudflare. Mission accomplished.

didntknowya• 5 years ago

it's kinda their whole objective- to keep sites up and running. so of course they'll do commentaries- it's for marketing and helps stimulate discussions in their industry.

joebob42• 5 years ago

Ironically, I read this and thought "I like this and these people are doing the sort of things I like doing, maybe I should think about working with them if and when I need another job."

Sample size of 1, but I imagine that's more or less the reaction they're hoping for.

stubish• 5 years ago

I like having a knowledgeable 3rd party report. Keeps the 1st party honest.

prawn2• 5 years ago

How'd you feel about Krebs' post about the outage?

https://krebsonsecurity.com/2021/10/what-happened-to-faceboo...

Ansil8498• 5 years ago

Krebs did not promote a DNS service he works for a total of nine times in his blogpost, so I did not feel like I was reading an infomercial for how great Cloudflare's continued MITMing of the internet is.

gcorrao• 5 years ago

I thought of this at first, but after reading FB official "tech" explanation on the outage (https://engineering.fb.com/2021/10/04/networking-traffic/out...) I very much prefer Cloudflare's take on this.

nancyp2• 5 years ago

I'dnt expect fb employees to write up about BGP while they are in middle of a fire drill.

yholio• 5 years ago

The whole blog post seemed like a click and bait to me.

Nothing is "explained" other than what we knew already, that some unlucky SOB shot themselves in the head with a BGP shotgun.

I understand, they have no way of knowing what happened inside Facebook. But they could give some detail about exactly what the BGP updates were, the structure of the IP space served etc.

BFG-One7• 5 years ago

YES! THIS!

zanethomas• 5 years ago

All very amusing. But most amusing was that facebook employees' badges wouldn't work. Hilarious. Can't get in the office if facebook.com is not reachable. (or whatever the unreachable service was)

unityByFreedom2• 5 years ago

Ah, so no playing oculus either?

14• 5 years ago

Curious about this as well could not find anything on initial searches but I honestly hope so. The one reason I have not grabbed the oculus is because the Facebook requirement.

drzaiusapelord• 5 years ago

Oculus customers were only able to play offline games during this time. Which I find concerning. I don't mind that FB employee badges didn't work because what would they do at their desks anyway and all exit doors have bypasses anyway so they can, and were, open, just rank and file employees couldn't access some spaces. Some of those spaces are secure and "fail closed" by design. Yes, its inconvenient but unlocking all the doors to potential terrorists or workplace shooters if the internet connection gets snipped isn't what you want in an office.

What's scary is that my gaming machine can't be used because of some weird obsession with centralizing oculus through facebook. As a customer that's inexcusable but now imagine a developer who hosts her own servers for her game and her customers can't get to it. Facebook is only providing middle-man authentication, its not even hosting these games. If they were hosted by FB it would be fine, but they aren't and what we're suffering under is FB being the gatekeeper to the rest of the internet. That is a scary prospect.

I also question why we all think its acceptable to have these incredible BGP outages every 3-6 months. We built our digital world on the equivalent of ever changing spinning plates and manage it in the most cost-efficient way possible. Maybe the alternative would been worse, but its crazy to me as to what we consider normal and acceptable in capitalist culture.

I'm already seeing /r/oculus say its no different than Valve's scheduled maintenance windows, which is obviously false. Its incredible what rationalizations we'll accept instead of questioning the status quo of capitalist culture and our giant corporations that rule so much of our lives.

jeremyis• 5 years ago

My friends said they couldn't play Oculus

ksaj2• 5 years ago

It wouldn't be the first time it's happened, and you would expect their network and security folk to understand single points of failure.

swayvil• 5 years ago

A single point of failure implies a single point of control. A dungeonmaster. There can be only one.

ffwszgf• 5 years ago

You would think those in charge of security would know better with their mid-six figure salaries. But then again these days they stopped hiring based on merit but rather certain other “metrics” no wonder they were so incompetent...

buryat• 5 years ago

you would learn not to trust a first message you read on hackernews about what's going on inside

jiocrag• 5 years ago

Far fetched, obviously, but.... Jurassic Park, Dennis Nedry-type situation? Shut down the master systems to hide the theft (or in this case, destruction) of vital info.

No key-card access means even non-need-to-know internal employees wouldn't see the deed, and plausible deniability is spawned for everyone else.

kevin_thibedeau• 5 years ago

"I'm sorry Senator. We lost that data in the Oct 4 incident."

TrispusAttucks• 5 years ago

Considering they just had a whistle blower leak incriminating documents... I'd say this is plausible.

Edit: Also a great way to suppress the news of the whistleblower. Take the site down so the outage outranks news searches keyword "Facebook"

pier25• 5 years ago

This is all very interesting, but why couldn't FB just push an update remotely to solve this?

I've seen mentions everywhere that to fix this they required physical access to the BGP routers.

ninkendo• 5 years ago

How would they push an update if they couldn’t reach their own network?

My understanding is that if a bad BGP route was pushed and your whole network (typically accessed via VPN while the engineers WFH) is probably unreachable from any of you’re employees’ homes, it’s hard to get the right people on site to make a fix, or talk the remote hands folks at the date center through it over the phone. Troubleshooting is hard, troubleshooting while you have no access to your own network has got to be just that much harder.

gukov• 5 years ago

Everyone knows deploying anything critical on the first and the last day of a work week is not a good idea. Is Facebook run differently?

lokar• 5 years ago

This and other stories seem to assume that a BGP issue is the root cause. That may be, but there are other possibilities.

Some other internal issues could have broken many systems, including BGP.

They could have had some sort of internal systems failure, and intentionally withdrew BGB to cut off the flood of connection attempts (making recovery easier).

Many possibilities.

ofou• 5 years ago

A network intern it's saying oh-oh I mess up something...

cottsak• 5 years ago

amazing write ups from the CF guys

nkssy• 5 years ago

Is there a petition to make this permanent? Asking for a friend

The article itself was a good exploration of the impact of BGP and what happens when network advertisements stop and the associated network disappears in a puff of global forgetfulness.

Might be time to poke around BGP as well. A lab setup might be a good toy.

nancyp2• 5 years ago

My CCNA friends liked the thread. Oh wait she was on 60 minutes last night.

paul79862• 5 years ago

This happens the day after the whistleblower's 60 Minutes interview? Maybe another rogue (whistleblower is surprised Facebook is a for profit business focused on profit .. not saying good or bad given how big facebook is but really how do you police the world and what side do you take in all the madness/negative parts that make up humanity thus facebook) Facebook employee took it offline ... any chance of that?

quickthrower2• 5 years ago

I'm on 1.1.1.1 and well..... FB was down.

dboreham• 5 years ago

They used the "foot blunderbus".

geocrasher• 5 years ago

How I explained this to friends and family: Imagine that the only way to get to Los Angeles is to use a GPS enabled device. All of the maps know how to get there from anywhere. But imagine that every GPS took Los Angeles off the map. Los Angeles is still there, but nobody knows how to get there. That's what happened to Facebook. We don't yet know why, however.

js2• 5 years ago

Those short TTLs that FB likely has on its DNS records probably bit it in the ass today. If it had longer TTLs, caching would have helped it. Curious though that recursive resolvers won't serve an expired cached record when they can't reach the authoritative server. (I know that unbound can be configured to do so, but not sure about others.)

asadjb2• 5 years ago

I don't think so. The root cause (as I understand) was that FB stopped advertising it's IP address space to the world. Even if you had the IP addresses of the FB servers, you would not find any route to access them.

The fact that DNS was also not resolving was a symptom of the DNS servers also being unavailable since they were part of the same IP address space that was unadvertised.

cblconfederate• 5 years ago

Maybe FB should switch to a single IP address to avoid burdening the internet with their DNS requests.

codedeadlock2• 5 years ago

How can you explain yesterday's outage (Facebook, Instagram, WhatsApp) to your parents?

You are feeling hungry and went to food court. The food court (open area) has a lot of options. You sit down in front of Domino's (Facebook), since you want to eat garlic bread. Now, you can't order from the counter directly. The waiter will come to your seat and ask for the order. You ordered garlic bread from the waiter, but the guy at Domino's counter went missing. Your order was not reaching to the chef in kitchen as Domino's counter guy was not present.

This explains why Domino's (Facebook) ecosystem was down, but what about other vendors? They had nothing to do with Facebook.

To understand this, we need to go back to our food court again. Now, there are a lot of hungry people sitting outside Domino. Since they were not getting answer from one waiter as why their food is not on their table, they started disturbing all the waiters. Due to this, majority of the waiters were trying to figure out where the Domino's counter guy went and other food joints (read websites) were not able to fulfil their own orders.

So although only Domino's was down, it appeared as if whole Food Court (Internet) was facing issues.

Counter Guy at Domino's - Facebook Nameservers Waiters - DNS Servers (Cloudflare, Google, Akamai)

miked85• 5 years ago

I think that this may be more confusing than the actual explanation.

codedeadlock2• 5 years ago

Thanks for the feedback. I will simplify it further!

ccakes• 5 years ago

You need to call Facebook to find out what your friends have been up to but their number has temporarily disappeared from the phone book.

phtrivier• 5 years ago

That would have been accurate for a DNS outage ; but with my layman understanding of BGP, I would say the analogy would be something between "...but their phone line is broken" and "...but they disappeared from the phone book because they don't have a phone line any more" .

Is that right ?

philposting• 5 years ago

I explained it as Facebook the city still existing, but they'd taken down the signposts.

rvnx• 5 years ago

Another way to see the situation:

Internet is just a bunch of computers interconnected via tons of cables (hence the name; "inter-networked computers").

To be reachable, every equipment and computer constantly need to tell the others about their existence (to publicly announce on which network cable they can be reached at).

Facebook engineers wanted to optimise that system but accidentally broke it during the update.

As a consequence, after a few minutes, other computers didn't know on which cables they can reach Facebook.

Facebook had to call the technicians sitting in the datacenter to cancel the last change that was done (because the Facebook engineers couldn't themselves connect from the office) and everything was fine again.

Philip-J-Fry• 5 years ago

I'd just say that you had an address for Facebook in your address book. The page somehow vanished and you don't know their address any more. So you start phoning other people and knocking on their door to try and find what their address is. Everyone else is doing this and no one knows what their address is. So you've got millions of people phoning each other and knocking on doors.

Facebook being down was already an issue, but everyone phoning and knocking on doors was causing disruption to everyone else.

srcmap• 5 years ago

If your parent is startrek fan, tell him "Commander Data" transmitted a "sleep" command to BGP (Borg Gateway Protocol) collective.

As result, the borg collectives can access to network, even 7of9 can't enter the campus.

BFG-One7• 5 years ago

cool, but were did mark zuckerberg (dominos chef) go? he felt dissed so he ran and hid?

corobo• 5 years ago

Analogies serve to confuse half the audience and make the other half go "actually it's more like .."

juanuys• 5 years ago

How about this:

Mum, dad - you know how I always tell you to turn your stuff off and on again?

Well, by turning it off, Facebook also turned off the On Button.

adminscoffee• 5 years ago

i've honestly been coding all day, i didn't notice until i got on here and say everyone talking about it

PaulRobinson2• 5 years ago

It's been 20+ years since I had reason to know the internals of BGP, but I still carry a slight nervousness whenever I think about it.

In many ways, it's beautiful in that it allows Autonomous Systems to come together and independently make the Internet.

However, I always felt it was brittle in that it relied every AS to have edge routers aware of the entire AS routing map, and so when those routers went down hard, when they came back up they were blank and needed to learn the Internet again. I noted back then that on some routers a single spoofed UDP packet to a router getting a table from its peers could cause it to stop doing that and start again. In a time when those updates could take hours... well... that kept me awake at night sometimes.

Propagation of routes was global and error prone. Even today, you can see on the spy glass websites rogue AS routers advertising crazy routes either by accident or on purpose as a hostile act (to try and get traffic for a target network through itself).

There are tens of thousands of engineers globally nursing and managing this stuff to keep the whole thing going.

Like I said, it's been a while since I last looked closely, and I imagine multiple improvements have been made in this space since that time, but if there was ever a protocol that needs a long, cold hard look for a replacement, it's quite possibly BGP.

It's also the one - and only - non-crypto situation where if somebody asked me if a blockchain could be really very useful, I'd say "probably, yeah".

nroets• 5 years ago

So much of the internet is about translating addresses from one format to another:

Human readable to IP: DNS

Private IP to Public IP: NAT

MAC to Private IP: ARP

Public IP to AS: BGP

With a block chain DNS it may be possible to replace all of them with a single protocol, but it may well turn out to be an academic exercise.

grlass• 5 years ago

Namecoin [1] was an early fork of bitcoin, that did blockchain based DNS, with the ideas being actively discussed by Satoshi Nakamoto [2].

That deals with DNS (though how much has it been used?), but what about the layers of the stack?

[1] https://en.wikipedia.org/wiki/Namecoin

[2] https://bitcointalk.org/index.php?topic=1790.222

dgb232• 5 years ago

These are abstraction barriers that isolate locations, layers and protocols from each other. I don't understand this as translation but rather as a separation of concerns.

oblio• 5 years ago

So much of the internet is about translating addresses from one format to another

Most of computing is about mapping A to B and maintaining databases of those mappings :-)

alfalfasprout3• 5 years ago

Problem is the blockchain stuff usually ends up being computationally expensive for what it's trying to do.

I could maybe see a use case for DNS but then either each computer has to have the full blockchain locally or rely on providers to do that for them in which case we're basically at the same model we have now (with root servers and DNS servers).

Obviously you don't want to publish private IPs, etc. to other computers... so inside a local network you're not buying yourself much.

For routing tables I could see more of a use case for blockchain technology since you can actually verify the routing tables and prevent BGP spoofing, etc.

collinmanderson• 5 years ago

I believe IPv6 intended to replace both ARP/MAC and NAT, but has mostly failed to achieve that.

vlan0• 5 years ago

Like I said, it's been a while since I last looked closely, and I imagine multiple improvements have been made in this space since that time

For anyone that's interested in what these may be. Take a peak at MANRS. https://www.manrs.org/

edf13• 5 years ago

Didn't we used to have long TTLs on DNS hostnames to help with this very reason?

teh_klev• 5 years ago

Once upon a time this would have been the norm. If we were performing any DNS related maintenance we'd drop the TTL to 5 mins a day or so before the maintenance window was due to start. Once happy we'd not broken anything then we'd bump back to 21600 or whatever. But I guess the move fast/break all the things crowd no longer have any patience for this kind of thing.

belter• 5 years ago

Lets assume somebody messed up their BGP config. Does anybody know if they use Juniper? Why they would not do the changes via "the commit confirmed" option?

The idea is you do something, you messed up and are locked out, the system will revert to previous config after a few minutes by itself. The story of people physically accessing systems only makes sense, if this was a hack...

===============

Commit Confirmed

"Suppose that despite all your efforts to insure your new configuration is correct before you commit it, something is overlooked and when you commit, you are locked out of the router."

"Rather than just a simple commit, you can make the candidate configuration active with a commit confirmed command. With this command, the router waits 10 minutes for a second commit. If it does not receive that confirming command within those 10 minutes, the router automatically does a rollback and commit so that the previous configuration becomes active again."

https://www.networkworld.com/article/2345600/managing-a-juno...

"Committing a Configuration"

https://www.juniper.net/documentation/us/en/software/junos/c...

===============

afinlayson• 5 years ago

I imagine Zuckerberg doing the same thing Taika Waititi's character "Antoine" did at the end of Free Guy while watching the 60 minutes video.

alfalfasprout3• 5 years ago

Stuff like this happens... but the real question is why FB had to send employees to physically go to datacenters to fix this. Sounds like their OOB management had a dependency on FB being online which seems like a bad design. If you're that paranoid about security issue revokable one time pads to a select set of employees.

pinix• 5 years ago

Let's say that when Facebook cannot announce their DNS prefixes, the DNS resolvers from other ASes cannot reach their DNS servers. But why? Does Facebook change the IP addresses of their DNS servers and other web endpoints constantly? Shoudn't Facebook IP addresses be cached by DNS resolvers?

Understanding how Facebook disappeared from the internet

Related Stories