Cloudflare blocks an almost 2 Tbps multi-vector DDoS attack

https://blog.cloudflare.com/cloudflare-blocks-an-almost-2-tbps-multi-vector-ddos-attack/

by sendilkumarn • 5 years ago

185 92 5 years ago

blog.cloudflare.com

donkarma• 5 years ago

I always thought there should be more terabit attacks with the level of home connections nowadays

leros• 5 years ago

I would imagine ISPs have some sort of bot prevention measures that would get triggered if you went all out on using a home connection.

pixl972• 5 years ago

In general, no. Unless you start affecting their internal network. If you keep the traffic rather moderate a home connection can spew traffic for months on end.

jchw2• 5 years ago

A good mitigation strategy is giving people 1Gbps down, over DOCSIS 3.1, that nobody can ever actually hit, and overselling significantly on top of that. Then, doing the same with upload, but only offering around 30Mbps up.

At least that’s how it feels in the U.S.

t0mas88• 5 years ago

Some have, but it's usually signature based. If a customer has an infection with a known worm (all I've seen were windows based) it's matched by some signature and the connection is isolated. From then on all web traffic is redirected to the ISPs service portal helping the customer install an antivirus solution.

Never seen it applied to DDoS kind of things.

ransom1538• 5 years ago

They do! I have a fast fiber connection. I have had an ISP sec/ops guy literally call me and ask about my traffic patterns. He was more curious than anything -- but they do monitor strange patterns. I agreed to turn off my crawlers and explained it wasn't a botnet.

wilde• 5 years ago

I assume you live outside the US? Our home connections here haven’t improved in two decades.

14• 5 years ago

Can’t they try take the bots offline? Do the bots hide their IP address or could they not start contacting the owners of said ip addresses and tell them they need to remove the infected device from the internet? I know it wouldn’t be that easy but is there nothing they can do to fight back and start getting rid of these bots?

pixl972• 5 years ago

How long does it take to contact thousands and thousands of IP owners looking for infected device? Many of which are behind NAT devices which require even further tracing.

What about the ones overseas that just don't care?

zeusk• 5 years ago

In past, they have taken bots offline (mainly by taking over the Command/Control server) but most of these "bots" are just malware infected connected devices operated by clueless average folks - hard to update, hard to take down.

buro92• 5 years ago

The article mentions that these were UDP attacks... which are usually reflections based on spoofed IP addresses. So who should Cloudflare contact? In the meantime another few hundred small attacks arrive. It's more constructive to improve the capability to mitigate attacks as they and other network providers have agency over that.

josefx3• 5 years ago

The UDP packets still have to pass through the network and networks can attach all kind of tracking headers to these packets. So you should be theoretically able to track down the sources of long running attack if every network provider along the line cooperates.

spiffytech3• 5 years ago

Do the bots hide their IP address

For this attack and many like it, yes, the bots hide their IP.

Per the article, this attack was a combination of DNS amplification and UDP flood. UDP packets don't use a connection like TCP (where the recipient verifies it can talk back to the sender); instead, the packet just declares where it came from, and the recipient fires-and-forgets a response to that IP, blindly assuming that IP is actually the sender.

So for the UDP flood portion, the victim receives a packet with a fraudulent source IP and no way to tell where it really came from.

For the DNS amplification part (also done over UDP), the attacker finds an open DNS resolver online, sends it a request to resolve a record, and fakes the UDP source IP, telling the DNS server to send the response to the attack victim. Not only does this mean the DDoS packets aren't coming directly from the attacker, but DNS responses can easily be much larger than DNS requests, so an attacker multiplies how many gigabits of traffic they hit the victim with, versus just sending UDP packets directly to the victim.

Here's Cloudflare's primer on DNS amplification attacks: https://www.cloudflare.com/en-gb/learning/ddos/dns-amplifica... and UDP floods: https://www.cloudflare.com/en-gb/learning/ddos/udp-flood-ddo...

As far as solutions go, the answers are broadly 1) get everyone in the world to stop putting up UDP services that send large responses to unverified requests (this attack used DNS, but this happens with other protocols too), and 2) convince ISPs everywhere to deny outbound UDP packets which claim a source IP from outside that ISP's network. Since this is one of those "you have to be perfect, but the attacker only has to find one weakness" scenarios, these sorts of attacks will keep happening until it becomes impractical to find enough abusable networks/services to mount high-volume attacks.

dgudkov• 5 years ago

The second point sounds like something that can be fixed with regulation and/or fines.

__turbobrew__• 5 years ago

How do you spoof the source IP? If these attacks mainly originate from IoT devices in home/business networks shouldn’t the ISP block outgoing packets which have a source IP which does not match the IP of the home/business routers interface?

I haven’t tried but I would hope my ISP would drop any packets I send out from my home network which do not have the public IP address of my router, but I haven’t tested it.

Ansil8493• 5 years ago

The entire attack lasted just one minute.

Did the attack last one minute because Cloudflare 'mitigated' it after that, or because the attackers stopped?

buro92• 5 years ago

Botnets tests their capabilities all the time. This could have been a command and control test, a test to see what they could muster, or a demonstration.

When testing they seldom run for a long time.

Cloudflare's mitigation would've dropped in on the metals and still been visible to Cloudflare's monitoring... so the attackers stopped after a minute.

remram2• 5 years ago

So those nice graphs on Cloudflare's blog are exactly the information the attackers wanted? If that's the case, by publishing such detailed post-morterms, Cloudflare is just inviting future test attacks.

toast06• 5 years ago

I used to run the servers for a popular website. It was common to get DDoSed targeting our servers (or more frequently, just a single one out of the group) for exactly 90 seconds (plus or minus a few systems that had poor ntp synchronization). Whether or not that took my servers down, the attack would stop.

To my knowledge, we never got any communication from the people behind the attack, seemed like people just kicking the tires on DDoS as a service. Ocassionally, we'd get a longer interval, sometimes 60 minutes.

raspyberr• 5 years ago

I've read that Cloudflare also hosts a lot of DDoS-for-hire services. That seems like a conflict of interest.

winternett3• 5 years ago

This is 2021, where almost everyone creates a global problem, then makes money off of being the one to "mitigate the problem"... The people dedicated to not creating new problems, but trying genuinely to fix problems simply fail and/or run out of money are increasingly ignored because they don't have the biggest marketing budgets. Honesty isn't making money any more... A huge problem.

The absence of any real accountability, and admiration of hypocrisy, is what threatens us most heading into the future.

systemvoltage2• 5 years ago

I am not convinced, do you have any sources that prove your conspiracy?

kordlessagain2• 5 years ago

By that logic (abuse of) the global internet is a problem, but the underlying technology isn’t, if it were localized.

teddyh• 5 years ago

“It's a gigantic social phenomenon. People find ways of getting money by impeding society. Once they can impede society, they can be paid to leave people alone.”

— Richard Stallman, 1986 https://www.gnu.org/gnu/byte-interview

Ansil8493• 5 years ago

They sure do [1].

[1] https://krebsonsecurity.com/2015/01/spreading-the-disease-an...

jdc2• 5 years ago

If I'm understanding this correctly, then what Cloudflare is doing is hosting websites of DDoS services rather than hosting DDoS attacks themselves.

aliswe3• 5 years ago

2015. not saying that anythings changed but worth noting.

quote from said article for perspective:

The Web site crimeflare.com, which tracks abusive sites that hide behind CloudFlare, has cataloged more than 200 DDoS-for-hire sites using CloudFlare. For its part, CloudFlare’s owners have rather vehemently resisted the notion of blocking booter services from using the company’s services, saying that doing so would lead CloudFlare down a “slippery slope of censorship.”

As I observed in a previous story about booters, CloudFlare CEO Matthew Prince has noted that while Cloudflare will respond to legal process and subpoenas from law enforcement to take sites offline, “sometimes we have court orders that order us to not take sites down.” Indeed, one such example was CarderProfit, a Cloudflare-protected carding forum that turned out to be an elaborate sting operation set up by the FBI.

systemvoltage2• 5 years ago

I think this an uncharitable simplification of a complex issue. Cloudflare tries to balance itself between censorship and overreach of what their customers are doing with their service (booting off Parlor earlier this year for example) as well as what the law-enforcement legally requires them to do. If Al Queda hosts a website on AWS, the problem is exactly the same.

And now, we have people essentially conspiring that Cloudflare creates their own DDoS attacks just so to prevent it based on a glib oversimplification.

tpmx• 5 years ago

They certainly don't host DDoS network ops. What you're talking about is hosting web pages.

wpietri4• 5 years ago

They're not just "web pages". They're a key part of the financial infrastructure sustaining the problem that Cloudflare gets paid $600m/year to fight.

Does that imply that Cloudflare is intentionally boosting the problem? No. But let's be clear here: anything that makes DDOS attacks less of a problem means less money for Cloudflare. So whatever their intent, Cloudflare is helping to support the problem that they owe their existence to. It's very much a conflict of interest.

eastdakota• 5 years ago

You’ve figured us out! Damn it; it would have been such a great plan if we hadn’t decided to give DDoS mitigation services away for free. Dagnabit!

https://blog.cloudflare.com/unmetered-mitigation/

axutio• 5 years ago

I mean, I don't buy the conspiracy theory at all, but I would expect it to still be true that you benefit from DDoS attacks even if you offer protection for free. As the need for DDoS protection increases, you convert non-users to users, moving them one step closer to paying customers.

I'm sure that the fact that it's highly illegal and unethical are reason enough for Cloudflare to not sell DDoS capacity, but the perverse incentive is still there.

johnklos• 5 years ago

You want everyone to believe that none of us are capable of seeing the first derivative of an action and reaction?

Cloudflare facilitates DDoS, yet Cloudflare "mitigates" for free. "How could this EVER be a business model?", you disingenuously ask.

Simple - if DDoS are common, then more and more people and companies become afraid of them. After a while, everyone wants DDoS mitigation. More and more people move to Cloudflare.

Whether paid or not, you now control more people. Duh.

short123• 5 years ago

What is with ddos these days?

Are they doing it for money ?

It just seems silly with services like cloud flare

catlikesshrimp2• 5 years ago

You can hire a ddos agaisnt your across the street competitor. It could be the other pizzeria, the other hardwareshop. Use your imagination

short123• 5 years ago

That used to be a thing but is it anymore?

There is so much mitigation so it's pretty much ineffective

1270018080• 5 years ago

If you have that much computing power at your disposal, you might as well just mine cryptocurrency, right?

codsane• 5 years ago

Not necessarily as this is Mirai, a botnet of IoT devices. There’s probably not much else you could do with them.

remram2• 5 years ago

You can probably do both at the same time, as cryptocurrency uses a lot of CPU/GPU/memory/... but little bandwidth, and DDoS typically uses bandwidth but little CPU/GPU/memory/...

taf2• 5 years ago

Assuming this is about telnxy outages this week and their migration to cloudflare. https://status.telnyx.com/

Maybe premature for cloudflare to be declaring victory?

BuildTheRobots• 5 years ago

Whilst I'm a big fan of people updating status pages, copy/pasted updates really rub me up the wrong way.

schleck8• 5 years ago

There is a truly excellent video on Mirai's (the botnet or atleast code in question) origin. It was created in the Minecraft server community by teenagers. The botnet was huge to a point where Akamai had to get help from Google to mitigate an attack on krebs' security blog. It also was used to attack Dyn, the infrastructure provider, and resulted in a huge outage affecting Netflix, Twitter etc.

Sadly it's only in German, but if you are on desktop, you can auto-translate the subtitles.

https://www.youtube.com/watch?v=uletKRPMnuo

maxgashkov• 5 years ago

I was responsible for a website (one of a many of this kind) that provided access to a niche auction platform. At some point in the beginning of 2010s it became a subject of a precisely coordinated series of timed attacks designed to disrupt bidding of one of our prominent clients in the specific auctions. It was enough to bring down the service for ~5 minutes to prevent the client from winning.

Eventually we migrated behind CF and the problem was solved but I couldn't help but wonder if there are some applications for which even a few seconds disruption (I assume that's the minimum time Cloudflare needs to begin effectively mitigate the attack of this scale) will be disastrous and what could possibly be done in this case?

iimblack• 5 years ago

Stock trading comes to mind

toast06• 5 years ago

If you can't handle a few seconds disruption, you really need actually private networking. Dedicated lines (or at least dedicated wavelength on shared fiber) and redundancy and very fast failover.

Volumetric udp reflection isn't really too bad to process anyway, as long as you've got the bandwidth --- fancy tricks get you from the UDP stack dropping useless packets to dropping useless packets without the UDP stack, possibly at the edge without using up nearly as much internal bandwidth.

Where it gets pretty hard to manage would be application level bursts, IMHO.

krebsonsecurity• 5 years ago

CF: Would it be asking too much to have a date and time stamp on your blog posts somewhere?

dmd• 5 years ago

It's right below the title, where you'd expect it.

IYasha• 5 years ago

Whatever, guys... Nothing, NOTHING will make me think better of CloudFlare. I won't forgive you, CF, for captcha, tracking and blocking me from accessing a critical server from an airport! Burn in hell!

Cloudflare blocks an almost 2 Tbps multi-vector DDoS attack

Related Stories