“An ISP in Asia is leaking routes to a Tier 1 transit provider”

https://www.cloudflarestatus.com/incidents/bzknm1t91kjq

by nothingbutslide • 11 years ago

194 100 11 years ago

cloudflarestatus.com

_jomo• 11 years ago

Apparently this is AS4788 (Telekom Malaysia) leaking routes.

https://twitter.com/rrbone_net/status/609282081420652544

bbrazil2• 11 years ago

Nanog discussion: http://mailman.nanog.org/pipermail/nanog/2015-June/076187.ht...

sbarre• 11 years ago

What is the impact of this problem? Any TL;DR from an expert on here?

buro9• 11 years ago

I am no expert.

But a route leak forces traffic to take a different path, and this potentially results in a pipe being inundated with a significant volume of traffic that cannot be handled by ISPs along that route (or simply a dead-end).

i.e. imagine an ISP in Rio suddenly declaring that they are the best route to reach the networks that contain facebook.com and google.com ... that ISP will DDoS itself or one of their downstream partners. An ISP may not even notice this, but their peers and partners probably will.

A couple of years ago a BGP route leak took out the entire internet for a few hours for people in Australia. They're fairly common unfortunately, but tend to have limited impact and are resolved quite quickly.

More info: https://labs.apnic.net/?p=139

tluyben2• 11 years ago

Thanks for asking that; this is the first time I open HN and think ; WUT???

bbrazil2• 11 years ago

People cannot access some internet sites, depending on who they have internet connectivity with.

https://blog.cloudflare.com/route-leak-incident-on-october-2... has more information from a previous time this happened. A key idea is that internet routing depends a lot on trust, and it's possible for a single misconfigured site to cause serious issues across the internet.

knodi123• 11 years ago

I thought the point of the internet was that it was fault tolerant enough to be deployed on the battlefield?

hhw2• 11 years ago

Any networks that both accept the prefixes and see these advertisements as the best path will send their traffic towards the route leaker, who will lack the capacity to handle this traffic, effectively blackholing these routes.

In theory, a large portion of the Internet would have still seen the legitimate routes as better paths, as they would have a shorter AS path (fewer networks in between) than the leaked ones. However, many networks often implement other BGP metrics for traffic engineering, depending on whether they are seeing the routes from a transit/peer/downstream customer, which may override the shortest AS path.

nissehulth• 11 years ago

Rumor has it that a lot of World of Warcraft players left their homes and went outside.

yebyen• 11 years ago

My connection through a Sprint hotspot is only able to reach www.google.com and plus.google.com (but not news.google.com, reddit, hn, my own website on the other side of the city I'm in, or practically any other website I can think of.)

I got on with tech support this morning and they had no indication that anything was wrong regionally, had me do a factory reset on the device, and verified it was connecting to their network successfully.

So, maybe my issue is resolved, but Sprint is still having peering issues. That is the kind of behavior I would expect to see during an issue like this.

Of course I go into the store and they tell me there have been regional issues for the last two weeks and it's not my device (we can't help you), but everything works fine for me while I'm inside of the store and on the way back. Then 2 miles before back at the office, broken again, call again and phone support guy hasn't heard anything about this Malaysia Telekom debacle or any regional outages in my area. At this point I notice that I can still reach some sites, but most sites are down.

Not completely sure this is related, since I've been having issues since Wednesday afternoon (in fact that would seem to indicate it's unrelated). Also possible I have been having one issue that was fixed by the factory reset, so now I can freely experience the other issue that everyone else around the globe is apparently having today.

I am interested to know from The Expert as well, when Telekom Malaysia fixed their issue (it is actually fixed at the source now, right?) would there be some propagation delay or aftershocks, or should everything just return to normal relatively immediately.

adamlj• 11 years ago

I can't access a couple of US sites (like github, linkedin etc.) from my Swedish ISP Bahnhof.

radiospiel• 11 years ago

Had the same issue from Berlin. (and probably unrelated to the original post anyways.) But now we are back online!

tomjen34• 11 years ago

Probably why I couldn't access Stackoverflow from Denmark earlier.

peterwaller2• 11 years ago

This is rather comic: https://twitter.com/TMCorp/status/609167065300271104

(That's the twitter account for the currently implicated provider which messed things up, and for the record it has a minion in a Hawaiian grass dress saying "Happy Friday!", posted about 10 hours ago.)

mfoy_2• 11 years ago

Haha, that's pretty great. Whoever runs the Twitter account must feel pretty bad about the timing on that...

nathas• 11 years ago

http://devopsreactions.tumblr.com/post/87284390953/friday-de...

brazzledazzle• 11 years ago

Some of the replies are so unnecessary and trite.

jameswyse• 11 years ago

Looks like it's party day at Telecom Malaysia. https://twitter.com/TMCorp/status/609167065300271104

johansch• 11 years ago

They added the route at 16:44 local KL time on a friday afternoon. The network team started partying early? :)

jessaustin3• 11 years ago

The bumiputra probably shouldn't schedule any deployment for Friday. b^) Then again GLBX probably should know that too.

ars• 11 years ago

Malaysia is an Islamic country, so Friday is the equivalent of the American Saturday to them, i.e. Friday is the first day of the Weekend, and Sunday is a normal workday.

piyushpr134• 11 years ago

did this happen last night too (about 15 hours from when this post was made)? I am in India and was not able to reach my servers in Singapore or do a git push. All other sites worked okay. I scratched my head and changed my dns to 8.8.8.8. It still did not work.

davidgerard• 11 years ago

It's not DNS - it's how the actual packets are supposed to get between you and the server, that got messed up.

hhw2• 11 years ago

Telecom Malaysia (AS4788) was leaking a full routing table to a Tier 1 network, Global Crossing (AS3549), who in turn was advertising the prefixes to its peers like Level3 (AS3356). Large portions of the Internet would have been affected.

This is a double fail, both for Telecom Malaysia for leaking a full routing table, and for GBLX who apparently isn't filtering prefixes from their downstream customers or even restricting to a max number of prefixes.

namecast• 11 years ago

I just remembered, GBLX was bought by L3 a few years back - I'm guessing someone has a "tighten up route maps for GBLX ASN" ticket open in their backlog at the Level 3 NOC.

The people managing peering for AS 3356 and AS 3549 should be the same group, no? ("That DB we don't share the URL of" seems to imply as much.)

peterwaller2• 11 years ago

Could someone elaborate on the scope of the problem? Is it a sensible question to ask "How many routes were leaked?" how big were the prefixes of the networks leaked?

elktea• 11 years ago

Full table was leaked

lucb1e• 11 years ago

I see only this:

Your IP address 5.79.68.161 has been flagged as a scanner. Scanners are not permitted. If you are seeing this message in error, please contact security@statuspage.io.

Guess I better send an e-mail to see this status page..?

decasteve• 11 years ago

After I switched Tor circuits it worked fine for me.

ryanlol21• 11 years ago

Fuckups like this should result in criminal charges (and immediate depeering). DoS attacks are illegal in most countries and this is definitely gross negligence.

alice715• 11 years ago

Shit happens. Not that it was intentional. Anyway, who cares about software stuff, nobody dies.

Unlike in engineering sector where a negligence could result in injury or death; those could be charged.

therealmarv• 11 years ago

This reminds me of arguments some people use in the EU against net neutrality... they want fast lanes for Industry 4.0, online emergency and self driving cars. Incredible thoughts... I know.

ryanlol21• 11 years ago

You're completely ignoring the potential monetary damages. A plenty of financial systems use the internet. I've also heard of a couple of other businesses utilizing the internet.

"Shit happens" would've been applicable if this was solved in 5 minutes. Accidents like this are trivial to prevent with proper policies.

AndrewDMcG• 11 years ago

It's actually interesting why this idea is so totally wrong.

So Telecom Malaysia messed up a config, and Global Crossing accepted their updates automatically.

Global Crossing didn't have to accept the bad update. They generally trust updates from other organisations that are generally trustworthy. They apply checks and restrictions proportionate to the risks involved.

These mistakes happen rarely. If they were to happen more often, major operators would apply more checks and restrictions. If they were to stop happening, operators would apply less checks and restrictions, because they have a cost in manpower, complexity, and loss of flexibility.

That's how the internet works. You could almost say that's what the internet is--the idea of being actively managed by people who know what they're doing and are not bound by exhaustive predefined policies is defining of how the internet came about and how it came to be dominant.

If you want a network guaranteed to be resistant to this kind of f---up, build one. The internet is that network which does not work that way, which is flexible, expandable, mostly "good enough" but not ever designed for absolute reliability.

ryanlol21• 11 years ago

This isn't about how the internet works. This is someone messing up and breaking the internet.

If I'm driving a car without paying attention to the road and kill a bunch of kids, that's my fault.

If Telecom Malaysia pushes new configs without testing them and breaks the internet, that's their fault.

Obviously these two things aren't even remotely comparable in seriousness, but it's clear that in both cases the people messing up should be held responsible.

tomjen34• 11 years ago

It's actually interesting why this idea is so totally wrong.

No it is not. You accept their claims of "mistakes" I see no evidence of that - how, exactly, do you leak a full table by accident? and this is too big a security hole to leave to hackers. Leak a bunch of table, shut down an entire country.

MichaelGG3• 11 years ago

If anything, the outrage should be directed at Global Crossing/L3. They shouldn't be allowing some little ISP to screw up their route tables like that. Because this means that the Malaysian government could secretly announce just a few blocks, route them back out so they still work, and easily do MITM.

OTOH, so much in telecom is hung together on the assumption of basically good actors everywhere.

mentat3• 11 years ago

Criminal is what jurisdiction? You going to send a drone strike?

ryanlol21• 11 years ago

Reading comprehension. I never said what they did was criminal.

deepnet• 11 years ago

Can confirm UK ISP was unable to reach arxiv imgur mit or nytimes.

Ycombinator and reddit were A OK though.

pyvpx5• 11 years ago

that is because cloudflare has direct peering with many access networks. most providers see cloudflare routes directly and not through a transit such as Level3.

amalcon• 11 years ago

nanog thread: http://seclists.org/nanog/2015/Jun/586

Basically what's happened here is that Telecom Malaysia told one of Level3's networks (AS3549 - Global Crossing) that it was capable of delivering traffic to... anywhere on the Internet. Global Crossing apparently didn't have the usual sanity checks in place.

Because of how BGP works, once GBLX decided the route seemed reasonable, it immediately proceeded to dump huge amounts of traffic on the tiny Telecom Malaysia. This isn't incorrect behavior on GLBX's part (accepting the routes was incorrect, but sending the traffic was correct once they were accepted). This is way more traffic than Telecom Malaysia was prepared to handle.

Every tier 1 network gets rid of traffic as soon as possible (because it reduces their costs), completely ignoring performance or whether a route seems sensible. Telecom Malaysia therefore claimed to the most attractive route from anywhere near Malaysia to most of the world.

Level3 is one of the biggest telecoms in the world -- and in fact the biggest by far in that region. This means that most of the Internet probably stopped working for anyone anywhere near Malaysia.

mikedb• 11 years ago

Is it actually normal to have some checks for this now? When this happened a few years ago in Australia the apnic writeup (someone has linked below) essentially said there was no scalable solution yet.

pyvpx5• 11 years ago

when the number of originated or announced routes are "reasonable", prefix lists, as they are called, are generated and applied to BGP sessions. when the originated or announced routes is substantial and/or fluid, there are little to no prefix lists; allowing anything to be announced and accepted.

in an ideal world that has not existed for over a decade, IRR data could be used to automagically generate accurate prefix lists but for all the usual reasons (inaccurate, stale data; obtuse interface; etc) that just isn't done on a wide scale.

You can put a prefix limit on a session (that's the "little" in little to no) but that doesn't stop someone from announcing 10,000 more specific routes to popular (YouTube, Facebook, Twitter, Akamai) destinations.

tomjen34• 11 years ago

Do we know that this happened by accident? Because if all it takes is a few routing packages that is a very attractive option for any state that wants to attack another.

SG-2• 11 years ago

since it's a full table leak it's likely an accident, if someone wanted to jack a specific network or ISP they'd only advertise those.

MichaelGG3• 11 years ago

Hard to say, but accidents do happen.

However, it'd be a good idea for CAs to look at any "domain validated" certs issued during this time and re-confirm. Really, any service that does insecure account resets or takeovers should be a bit more wary than usual.

MichaelGG3• 11 years ago

Every tier 1 network gets rid of traffic as soon as possible

Really? That's not been my experience with L3 and a few others. They seem to prefer customer routes always, even when prepended. But I don't really understand this too in-depth.

sumedh• 11 years ago

I was able to open reddit but I was not able to open wikipedia from Australia. So I did a tracert.

6 29 ms syd-sot-ken-int2-be-20.tpgi.com.au [203.219.35.68]

7 197 ms 10ge3-7.core1.sjc1.he.net [72.52.66.21]

8 199 ms 10ge1-4.core1.pao1.he.net [72.52.92.113]

9 * Request timed out.

ryan-c• 11 years ago

Looks like it made it to California (San Jose and Palo Alto) with what appears to be a reasonable round trip time. Weird.

Edit: It seems like that must have come through the Southern Cross Cable[0] which is nowhere near Malaysia.

0. https://en.wikipedia.org/wiki/Southern_Cross_Cable

h43k3r• 11 years ago

I am very much interested in learning about such issues. Any links or blogs that put light on such issues.

ethbro• 11 years ago

Google has a couple good starting points: http://www.enterprisenetworkingplanet.com/netsp/article.php/... http://blog.pluralsight.com/bgp-border-gateway-protocol

georgerobinson3• 11 years ago

There is a great chapter on BGP by Hari Balakrishnan. I believe the chapter is available on MIT Open Courseware. It should be under the Computer Networks class.

Edit: found it. See Lecture 4: http://ocw.mit.edu/courses/electrical-engineering-and-comput...

gordon_gecco• 11 years ago

https://www.tm.com.my/OnlineHelp/Announcement/Pages/INTERNET...

ckvamme• 11 years ago

Here is an interactive snapshot of this outage:

Capital One uses Level 3 GLBX as a primary ISP: https://oxqyi.share.thousandeyes.com

(Jump to BGP route visualization and you can see AS 3549)

On the routing plane, you can see that the London GLBX monitor had issues to many services and that AS4788 Malaysia Telecom was advertised in the route.

Here is an example from a LinkedIn snapshot: https://wbpkq.share.thousandeyes.com

prusswan• 11 years ago

just want to ask if there's anything a home user could do to detect unusual routing behavior/phenomenon? I'm envisioning something like a browser plugin that logs/monitors outgoing connections and traceroute data

With that, instead of just looking at 404s we can make more informative observations like this request got stuck at this node, or that request is routed through a new node which has never been seen before recently..

“An ISP in Asia is leaking routes to a Tier 1 transit provider”

Related Stories