A Note from one of Cloudflare's upstream providers

http://cluepon.net/ras/gizmodo

by jauer • 13 years ago

644 87 13 years ago

cluepon.net

malachismith• 13 years ago

Fantastic response. Thank you so much for writing and sharing. Cuts through all the spin and FUD effectively.

e40• 13 years ago

I never read the Gizmodo piece, because I think they're mostly trolls, but I really liked this response. Kudos to him.

guelo• 13 years ago

The Gizmodo piece was better journalism than the nytimes or bbc articles which regurgitated Cloudfare's over the top PR uncritically.

devindotcom• 13 years ago

I disagree. Calling the story a straight-up lie was a mistake, and link bait to boot. I thought Biddle failed to understand what the risk to "The Internet" truly was, and how an attack like this (based on a known architecture vulnerability), if repeated or made more common or effective, could cripple a tier 1 provider and cause serious routing issues, among other things.

lucb1e3• 13 years ago

Better, but still a long way before it's factually correct.

mtgx• 13 years ago

NYTimes especially has been hyping up cyberattacks and cyberwarfare a lot lately.

markisonfire• 13 years ago

In all fairness, the Gizmodo article was pretty well researched and informative. It was more like an anti-hype article - because that's what the headlines were. Interesting things did happen, but the internet was not "in danger" of being down.

nullrouted• 13 years ago

For those of you who don't know Ras is one of the people that run/ran nlayer (now part of GTT) which is Cloudflare's primary provider.

sargun• 13 years ago

RAS has been pretty awesome in the community. He was actually the cofounder of the company we called nLayer. He's given a bunch of very valuable "101" talks at NANOG (think of this as the Hacker News for networking people), and until the acquisition, he was pretty active on the mailing list.

dudus• 13 years ago

It amazes me how little I know about overral internet traffic infraestructures

windexh8er• 13 years ago

That's why it's frustrating for those who have a network arch/eng background when we run across the sensationalized pieces. Or when all of the sec-only wonks make this out to be something to combat with a firewall, and put out pieces that never mention BGP, but are somehow helpful.

End rant, but at least there's more of an awareness of good network folk with things related to this sort of story. Not that it's the ideal path, but still.

niggler• 13 years ago

Who is guilty of the sensationalized piece? My impression was that it was cloudflare itself.

kilovoltaire• 13 years ago

Same here. Can anyone point to a good starting point for someone who understands TCP/IP to start understanding the Internet topology?

WestCoastJustin• 13 years ago

You might want to check out the NANOG mailing list. There is also a list of Internet exchange points [1], these pages are packed with details about how many of them function.

TCP/IP and Internet topology are different things. Think of it this way, internet topology is the highway, and TCP/IP is a vehicle that travels the highway (along with UDP, ICMP, etc).

[1] http://en.wikipedia.org/wiki/List_of_Internet_exchange_point...

dmourati• 13 years ago

drpeering.net

I like his map of US peering as well:

http://drpeering.net/white-papers/shared_images/USInterconne...

codesuela4• 13 years ago

Ars Technica had a pretty good article about peering and transit: http://arstechnica.com/features/2008/09/peering-and-transit/

don't know if that is what you were looking for but it does a pretty good job explaining that part.

jauer• 13 years ago

I'd start with the slides from a powerpoint by ras on traceroute: http://www.nanog.org/meetings/nanog47/presentations/Sunday/R...

He really gets into how, with traceroute, you can get a glimpse of how everything fits together, beyond just "oh, this hop and now that hop". Also, Andrew Blum's book Tubes is a good casual look into how the world of data centers, IXPs, carriers, etc all fit together.

rdl• 13 years ago

For humor, there's always http://www.routergod.com/ Sadly, it hasn't really been updated in a while, but it was technically fairly accurate.

lucb1e3• 13 years ago

Lol, opposite reaction here. He said pretty much what I thought of this and what I've been telling some others on other places. And in the last few paragraphs he said something that announced the coming of the really juicy stuff, but then it was about the exchanges which I already knew about :(

morganwilde• 13 years ago

My first reaction to these kinds of news is fear. As someone who builds stuff for the web, I really hate the idea of some malicious being trying to purposefully ruin your work.

My second reaction is that of "bring-it-on". Basically this is an impulse for improvement, and as with any major threat you either stand your ground or get run over.

lsc4• 13 years ago

> This is definitely on the large end of the scale as far as DoS attacks go, but I wouldn't call it "record smashing" or "game changing" in any special way. It's just another large attack, maybe 10-15% larger than other similar ones we've seen inthe past

Heh. Nice. Yeah, I expressed skepticism that 300G/sec qualified as "largest ever" - I mean, I personally have been hit by 10G+ attacks, and Cogent mostly shrugged. (I mean, my cogent side was down until the target was blackholed at the cogent border.) I know 10 gigabits is a lot less than 300 gigabits, but I am a nobody compared to the people involved in this little kerfuffle.)

duaneb• 13 years ago

I thought that it was the largest ever, but it wasn't larger enough to really be notable, just sort of the natural progression of internet traffic.

knighthacker• 13 years ago

Exactly. The author already said "10-15% larger than other similar ones we've seen inthe past". Doesn't that make a record-changing by definition? :)

druiid• 13 years ago

Yeah, I highly doubt 300G/sec is the largest attack ever as well. I've seen (well, we only saw a gigabit of it.. our host saw the rest) a 41G/sec attack. I imagine large attacks like this happen all the time but don't make the news because they just get fixed and the companies move on because it happens so much.

vidarh• 13 years ago

Note that CloudFlare itself doesn't claim it is the largest ever, but references NY Times' claim that it was, and then later goes on to say "that would make this attack one of the largest ever reported."

Other than the headline and the reference to the NY Times article, CloudFlares claims and the linked article are pretty much in line.

cft• 13 years ago

Can you describe Cogent's reaction to your request to blackhole the route? We had a very small incident with XO recently (perhaps 500Mbps reflected DNS flood), and it took them 24 hrs to get back to me to blackhole the target! We just packet filtered in our edge switch. Was Cogent any faster?

dsl2• 13 years ago

If you run a network of any size you should be speaking BGP with your upstream, even if its using a private AS. You can then announce a prefix to them specially tagged with a "blackhole community" that drops traffic at the edge of their network.

The exact details vary by network, but here is how Hurricane does it http://www.he.net/adm/blackhole.html

polarix• 13 years ago

What if this had been Akamai and not Cloudflare? Would we even hear about it? Why is Cloudflare in the news regularly and never Akamai? Do they just like drama? Is that good or bad for their business?

kyrra• 13 years ago

Cloudflare uses it as advertising. "Look at this attack we mitigated for our client!", which then gets picked up by HN and similar sites because Cloudflare does write fairly good blog posts about it. As others have said, Cloudflare does a fairly good job explaining parts of the internet that most of us don't get to see, so it gets attention.

youngtaff• 13 years ago

It gets picked up by HN as CloudFlare post it to HN themselves.

That said they're interesting articles if you ignore the hype.

drakaal2• 13 years ago

More like they invent drama to cover their bad days. Akamai tells its clients when they are under DDoS, Akamai didn't really notice anything going on. CF just had a bad day and decided to make a story out of it. One other time when CF claimed a DDoS it was because the Google Bot Decided to up it's crawl rate, and took down many CF sites. CF told clients it was DDoS and then later admitted it was an issue with Google.

kaolinite• 13 years ago

Definitely good for their business that many people think it was a groundbreaking attack and CloudFlare stood strong against it.

trotsky• 13 years ago

As an observation, the same day the new york times story ran sharespost announced that they had just added cloudfare to their secondary market platform. This typically means that an insider such as a founder or early investor plans to auction some of their shares. They can't put that in the media story as that breaks SEC rules, but positive press is very likely to drive up share pricing.

eastdakota3• 13 years ago

We have no plans to allow CloudFlare's shares to be traded via Sharespost or any secondary market. We'll investigate if Sharespost has done something without our permission.

bicknergseng• 13 years ago

IMO and not very familiar on the similarities between Akamai and CF, if Akamai handle's similar issues without driving pr/marketing efforts based on them that's their loss and probably a bad business play.

In my mind, this is similar to an antivirus company saying "hey look at all these nasty viruses out there, but we find and destroy them effectively."

This really just seems like effective case-studying on CF's part. It's arguably their job to hype it as much as possible (though of course they are responsible for the inaccuracies).

larrys• 13 years ago

"if Akamai handle's similar issues without driving pr/marketing efforts based on them that's their loss and probably a bad business play."

Akamai has been around since 1998.

Cloudflare since 2009 (both according according to crunchbase which jives with the date the domain were registered iim).

Consequently the established company has more to lose and less to gain from the publicity then the newer company.

For a newer company any publicity is good publicity even if it's over a negative event because you have less to lose and more to gain add: and people become familiar with your name.

Taking this to an extreme example (to make a point) let's say you start a new hamburger restaurant. You have no customers. On day 5 some people get sick (just sick not deathly sick). All the sudden you are in the local paper with a headline and a story that people just skim but they see your name. Almost guaranteed you will pick up business from the mention. Even though it's bad PR all the sudden down the road people will remember you and either forget or not care about the negative story they read 6 mos. prior (add: assuming they even read the story and didn't just see the headline).

saraid216• 13 years ago

> In my mind, this is similar to an antivirus company saying "hey look at all these nasty viruses out there, but we find and destroy them effectively."

I'd upvote such an article if it had the technical depth/layman readability of a Cloudflare puff piece.

snowwrestler2• 13 years ago

Because most companies who employ DDOS mitigation services want discretion, and smart DDOS protection companies are discrete. Even Cloudflare does not discuss most of their high-end customers. In this case Spamhaus gave them permission to write it up.

Cloudflare differs from most DDOS mitigation companies because of their low-dollar, self-service tier. This gives them a reason to blog, and customers and attacks to talk about. Most DDOS companies only provide bespoke service for $$$, and those contracts usually come with silence requirements.

Akamai is not 2nd on the list of DDOS mitigation companies, BTW. I know a couple companies who left Akamai for DOS Arrest, one of great companies you'll never see mentioned in a NYTimes article (because that's how their customer want it).

hhw2• 13 years ago

Exactly. DoS Arrest was using BGP anycast for DDoS mitigation long before CloudFlare even existed. We had customers on their service back in 2008.

I'm somewhat skeptical of CloudFlare's low-cost approach to DDoS mitigation. Going for volume on low MRC clients means that you have a lot of potential targets on your network. And attacks against any one customer can always impact every other customer, which puts you constantly at risk, even if you yourself are not attacked all that often.

zobzu• 13 years ago

That was a very nicely worded response that echos what, I hope/believe, most of us think.

Plus; that was in ASCII. Damn. :)

tiedemann• 13 years ago

Word. Respects given to serving in text/plain (as is proper in this context).

edmond_dantes• 13 years ago

Uber respect if the file had gzip content-encoding.

lelandbatey• 13 years ago

As someone who's fallen in love with writing absolutely everything in straight plain-text, seeing this brought a warmth feeling to my heart : )

stevewilhelm• 13 years ago

This incident drives home the fact that there is no one entity responsible for "The Internet." It is run by a network of for-profit companies, governments, and non-profit public and private standards bodies.

smackfu• 13 years ago

>The next part is where things got interesting, and is the part that nobody outside of extremely technical circles has actually bothered to try and understand yet.

Isn't this talked about in CloudFare's write-up?

mpchlets• 13 years ago

Nice response, nice sentiment, nice format. I often dislike the sensationalism surrounding attacks or viruses - it makes people distrust and gives others excuses for issues.

finnh• 13 years ago

> But, having a bad day on the Internet is nothing new.

That's my new quote of the year.

eastdakota3• 13 years ago

This is a great writeup and aligns with what we saw at CloudFlare. The most interesting part of the attack was that the attackers went after the IXs.

eastdakota3• 13 years ago

PS - This was written by nLayer/GTT's CEO, one of CloudFlare's bandwidth providers. They have been a terrific partner and were extremely helpful as we mitigated this attack.

spiantino• 13 years ago

Would you mind explaining this piece of the story in a little more detail?

"When the attackers stumbled upon this, probably by accident, it resulted in a lot of bogus traffic being injected into the IXP fabrics in an unusual way, until the IXP operators were able to work with everyone to make certain the IXP IP blocks weren't being globally re-advertised."

It's pretty fascinating and I think most of the HN audience, myself included, would be able to understand the actual technical detail.

Confusion• 13 years ago

Not that it matters much, as the mail speaks for itself, but according to the website: CTO

Paige• 13 years ago

Richard you can hit me with your cluepon any day.

drakaal2• 13 years ago

Straight up Lie. Talking with Google Engineers, and Amazon Engineers they say this is not happening. So unless someone picked a fight with CF, it is more likely CF is just having a bad day.

barkingcat• 13 years ago

I think this just reminds everyone that as large as google, facebook, etc seem to be, they are just a small part of this huge global network we humans have created.

As large as Google, Facebook, Amazon, etc is on the web, the major telcos have to be even larger (in terms of network size, capacity, amount of fibre, switches, datacentres) in order to carry the traffic.

EricBurnett• 13 years ago

That's only true to a point. In addition to the consumer-directed packets, large volumes of traffic for Google and Amazon never leave their networks. Shuttling data between datacenters for Google; moving data within the many cloud services of Amazon; or transferring between the two companies (e.g. GCS to S3). This means it's no longer a given that telcos must be larger.

The closer we get to living "in the cloud", the more our traffic can be seen as a window into operations taking place within and between cloud services.

donavanm• 13 years ago

Related, http://cloudscare.com

A Note from one of Cloudflare's upstream providers

Related Stories