“We have been experiencing a catastrophic DDoS attack”

https://status.linode.com/?

by wowaname • 10 years ago

187 141 10 years ago

status.linode.com

ryanlol6• 10 years ago

Update - We have been experiencing a catastrophic DDoS attack which is being spread across hundreds of different IP addresses in rapid succession, making mitigation extremely difficult. We are currently working with our upstreams to implement more complete mitigation.

That's pretty harsh.

diegorbaquero• 10 years ago

3½ hours already. Distributed from and to many IP addresses. Seems like an attack on them rather than on specific users. :/

tempestn3• 10 years ago

Or an attack on them as a vector to attack a specific user or users.

encoderer• 10 years ago

I've done some googling before asking here: Can anybody explain why Linode is so often targeted like this? We moved Cronitor off Linode in spring 2015. During the christmas holiday when they suffered a 2 week DDOS I thought of the family time I'd be missing that year as we did a crash migration to AWS had we not migrated when we did. I have to imagine this has been horrible for their business.

I would use Linode if I needed to lease computational power, because it is still a great value vs AWS, but I could not run a high availability service there. It would feel like professional malpractice at this point.

nxzero• 10 years ago

Ask HN: Who is behind the Linode DDOS attacks?

https://news.ycombinator.com/item?id=10797795

ryanlol6• 10 years ago

https://news.ycombinator.com/item?id=10797827

Found the culprit.

mappu3• 10 years ago

We stuck with Linode after that, but the important parts now failover to Vultr.

reefoctopus4• 10 years ago

How did you set it up to failover? Our MySQL database is very large, and I'm not sure of the best way to ensure it is up to date on another host.

noir-york2• 10 years ago

We stuck with Linode after last Christmas too, but this new attack looks like the final straw.

How have you found Vultr?

funkyy• 10 years ago

I am also surprised as this is not the first time I am reading about it on HN. Linode seems to be highly vulnerable to certain attacks as we could see in the past. I hope they will fix it and provide a permanent solution as I was hoping to use them as a part of my network, but I see more and more signals they can't handle serious traffic. Hopefully, they will redesign their infrastructure to handle it.

I am with all of you guys that are affected by this. I am looking forward to them to resolving this soon.

morecoffee2• 10 years ago

Isn't the problem money? They could certainly fix it but would have to raise the prices on each VPS?

Sir_Cmpwn• 10 years ago

Disclaimer: Linode employee, not in networking

My understanding is that Atlanta in particular has some poor upstreams and is making our job pretty difficult there. Notice that it's almost always Atlanta that's getting hit. I would suggest just using other datacenters or making sure your high availability model includes several DCs (which I would suggest at any hosting provider, really).

busterarm3• 10 years ago

We were sort of in the same boat as you last year. We were already prepping our failover and it was about 95% ready to go when the DDOS started on our Christmas break (and our senior guy who did all of our deploys was out of communication). December 23rd was a difficult 10 hour, remote day but we got things finished up and could relax afterwards.

spydum• 10 years ago

As others have said, maybe it's to affect a customer. However, hosting industry had a lot of roots in the adult content industry, and those guys didn't have many ethical guidelines. It was not uncommon for hosting companies to DDoS each other to drive a competitor out of business. Not saying that is the case here, but perhaps?

VonGuard• 10 years ago

OK, who hosts at Linode and is very popular/pisses people off? 4chan? 9gag? Reddit? Something Awful?

ryanlol6• 10 years ago

I think 9gag used to be on linode? I'm pretty sure none of the rest are though.

discr3t3• 10 years ago

Yeah Reddit is on AWS[1] and 4chan appears to be self-hosted (or at least it was for most of its existence)[2]

[1] http://highscalability.com/blog/2013/8/26/reddit-lessons-lea... [2] https://www.4chan.org/news?all#106

busterarm3• 10 years ago

WPEngine & their customers.

dopamean• 10 years ago

This is becoming less and less true every day since last Christmas.

morecoffee2• 10 years ago

It's not hard to compare this to brush fires. They happen periodically, and only the big trees tend to survive them. Linode is getting pretty unlucky here, but I would imagine that all the small time (and even the medium sized) hosting provides are going to succumb eventually. Is the end game just going to be Google vs. Amazon?

wowaname2• 10 years ago

I'd love to see a network infrastructure and transport protocol that's more resistant to many (D)DoS attacks, because it seems like things will only worsen if it never becomes more difficult for people to attack others' servers online.

apapli• 10 years ago

Good luck. A DDoS is basically lots of traffic. Perhaps run IPX? (Joking)

blfr• 10 years ago

There are other companies which manage to deal with DDoS attacks (OVH, CloudFlare) but you're right that none of them can be described as small.

i_feel_great2• 10 years ago

Well, I had Linode shortlisted for an upcoming project. I hate to take them off the list because it is not their fault, but I don't want this kind of unreliability.

tomschlick5• 10 years ago

Atlanta is their smallest datacenter IIRC with their lowest bandwidth capacity.

I would still recommend them as we have had great luck with their Dallas DC. It seems to be the most resilient of them all in terms of random network outages and also DDOS attacks.

However the one in December did take them offline for a few hours but after which they apparently implemented more DDOS protection. Keep in mind this could happen to anyone (Digital Ocean, Vultr, etc) and mostly their mitigation techniques seem to be to kill your VM until the attack is over.

reefoctopus4• 10 years ago

the one in December did take them offline for a few hours

That is a bit of an understatement. Our servers were down every couple of hours for a week, and the attack bounced around to pretty much all of their datacenters.

poopythecat3• 10 years ago

How do you know atlanta is their smallest datacenter?

Is there a list somewhere that shows what the capacity of each of their datacenters is?

Atlanta got trashed the worst back in December but they beefed it up a lot. But how much? Is Dallas still bigger?

Im considering migrating everything to dallas.

delroth• 10 years ago

it is not their fault

I don't understand this line of reasoning. It's not like DDoS attacks are some kind of 0-day failure mode that nobody has seen before.

Would you also say "it is not their fault" if their uplink provider had a fiber cut and they didn't have redundant uplinks? I'm guessing not: it's well understood that has a service provider you need to plan for this kind of unavailability and pay more money for redundant links. So it seems really weird to have this double standard for a different kind of availability failure mode.

Just like network availability or datacenter power availability, you need to invest technical and financial resources into DDoS defenses if you want to be resilient to incidents. If you don't do that as a hosting provider, I definitely won't feel sad for you.

toomuchtodo2• 10 years ago

There are a handful of environments that can sustain a large, coordinated DDOS attack. Can you sink 10-20Gb/s of traffic forever? Not cost effectively.

i_feel_great2• 10 years ago

I hate to blame the victim. I do not know enough about the circumstances, or the industry in general, to call negligence on Linode.

julianz• 10 years ago

I have a Linode VPS that is fantastically reliable. It's showing no downtime or problems during this DDOS. They've been great to deal with.

edoceo2• 10 years ago

My box in ATL is just wicked slow, on the network. Looks like saturation to me. Got stuff in their other DCs which seems OK for now.

finid• 10 years ago

This can't be good for (Linode's) business.

Vultr had problems a few weeks ago, but I don't think it was DDOS-related.

Somehow things have been quiet on the DigitalOcean's end.

thezach• 10 years ago

I used to be on Linode... then one of their techs tripped on a power cord causing a significant outage in a data center... and no I'm not joking.

reefoctopus4• 10 years ago

This wasn't something recent was it? Which datacenter?

jp_sc• 10 years ago

You are confusing them with Dreamhost: http://www.manton.org/2011/03/brent_on_baked.html

ausjke• 10 years ago

with linode for 10+ years and never heard about this, can you back this up with some evidence?

Scarbutt2• 10 years ago

What kind of DDOS attack is the most likely happening here, a simple L3 spoofed ICMP flood?

jonahx2• 10 years ago

Can anyone recommend a good article that explains how attacks like these work, and what is required to stop them?

Also, we're on Heroku and they advertise Ddos mitigation as a feature, but "mitigation" sounds non-commital and I'm curious how they'd fare against a similar attack?

whorleater• 10 years ago

Cisco has the easiest to grasp fundamentals [1], but to really understand DDOS attacks you'll have to dig down into learning about how TCP works and network layers.

Mitigation sounds non-commital because it is. I don't personally have any experience with mitigating a DDOS attack on Heroku, so I'm not qualified to talk about their prepardness, but DDOS mitigation varies quite a bit [2][3], running the entire gamut of "block a single IP address" to "there's hundreds of IP's rotating and attacking". So the answer to that would be: it depends on who you've pissed off or who's feeling particularly nasty that day.

[1] https://www.cisco.com/c/en/us/products/collateral/security/t...

[2] http://www.continuitycentral.com/index.php/news/technology/6...

[3] https://www.radware.com/newsevents/mediacoverage/changing-ch...

viraptor3• 10 years ago

It's non-commital, because at some point, when you have enough zombie hosts properly distributed all over the world attacking you, your only defence is - have more bandwidth than the attackers. If your peers can't filter out the traffic before it hits your network and it simply saturates your pipes, there's nothing you can do inside the company anymore.

jonahx2• 10 years ago

Thanks for the reply. Can you elaborate on what you mean by "peers" in the above? Eg, who (or what) would Heroku's peers be?

882542F3884314B• 10 years ago

The discussion [1] below was helpful to summarize the last DDoS attack and new controls implemented.

[1] https://news.ycombinator.com/item?id=10998661

throwsep3• 10 years ago

I wonder if this is a diversion to keep Linode's security team busy so they won't notice someone compromising the Xen nodes with XSA-185/6/7/8?

HappyTypist• 10 years ago

This happened last time.

mappu3• 10 years ago

Fascinating, seems believable.

XSAs are released alarmingly frequently. I've always wondered if KVM is really more secure or if it just gets less scrutiny (certainly there's a class of qemu vulnerabilities that impact both hypervisors).

rodgerd• 10 years ago

KVM is a much smaller code base, which probably helps.

Scarbutt2• 10 years ago

They switched to KVM.

kawsper• 10 years ago

As far as I know, not all VPSs are running KVM yet. I migrated my last VPS to KVM just yesterday.

technion3• 10 years ago

I've noted the AWS security bulletins[0] list nearly every Xen advisory with "AWS customers' data and instances are not affected by these issues, and there is no customer action required."

It would appear? you'd need to go back for quite a few months of being unpatched to find a genuine issue. Unless something about Amazon's mitigations don't apply universally.

[0] https://aws.amazon.com/security/security-bulletins/

lmz• 10 years ago

AWS (and other large hosts: https://www.xenproject.org/security-policy.html search for predisclosure) get notified before the public.

exhilaration• 10 years ago

Amazon and other big cloud providers get xen security fixes first, there was an HN discussion about it. Some they've probably already implemented the fix.

mratzloff2• 10 years ago

Apparently they came to the same conclusion—just a bit late.

https://status.linode.com/incidents/frq9sz7dyb1x

hyperpape2• 10 years ago

What in that article makes you think that? I don't see it. They do say "we'll have to upgrade Xen nodes", but they don't mention the DDoS or link them.

pmalynin• 10 years ago

This explains why Package Control, Ansible Docs, etc. were down. Sucks. I knew this has happened before, just didn't know it was Linode.

chrischen• 10 years ago

Does the US have a competent cyber-crime division that can handle stuff like this?

icehawk219• 10 years ago

Quite the opposite. Our government at least appears to work very hard to make sure these attack venues remain wide open.

SEJeff• 10 years ago

Yes, the FBI. They're fantastic at it and part of their job is helping businesses recover from compromise and going after the attackers. However, they're overworked government employees with not enough resources.

matt_wulfeck2• 10 years ago

I think they very much want to, but what can you do if all of your IPs lead to tor exit nodes?

From what I've heard, the FBI will collect a whole bunch of information and then sit on their hands because of the above reason.

naveen99• 10 years ago

overworked government employees

An oxymoron ?

matt_wulfeck2• 10 years ago

I disagree with people saying these types of attacks can't be prevented if you switched hosts. I'm sure Google+cloudflare[0] would keep your website online. AWS also if you had the cash.

The amount of distributed traffic happening right now against linode would probably only represent a 5% increase in traffic to a popular Google product. At least you know they have the expertise. Nothing against the very smart and talented linode engineers, but the two companies are on very different levels of traffic engineering.

[0] https://www.cloudflare.com/google/

the_duke• 10 years ago

Google and AWS probably have sophisticated DDOS mitigation (can anyone comment on this?) and you can scale up pretty quickly.

But if your service is the direct target of the attack (as opposed to the whole provider) and your servers are getting hammered...

Even if you architecture allows quick horizontal scaling, you still face a tough decision.

The attack could go on for days, and the hosting costs can go really high really fast. Which can be catastrophic for a small company.

user59944612• 10 years ago

Attacks are rarely targeted to the hosting providers. They usually target a specific customer.

Google/AWS probably have 100 times the capacity (and redundancy and architecture reliability and failover and awesomeness) of linode. That means that, first, they can't be put down easily, second, a DDoS is limited to a small subset of the infrastructure and doesn't bleed to every customers and services.

As for traditional hosting companies (OVH and the likes) When you're being DDoSed, they'll null-route your IP space. (i.e. they advertise your IPs as dont-exist-on-the-internet-anymore). The traffic is dropped while in transit on the internet because it can't go anywhere. It doesn't reach the hosting company anymore.

Note: being null-routed means your site and all your services are off the internet and thus effectively dead.

As for CloudFlare. They have many locations all around the world and they can absorb a lot of traffic, to the point they themselves cannot be DDoS. They have active monitoring and mitigation against common attacks and known malicious sources, which may prevent the attack without even you knowing about it.

When you're under attack, you can block subnet/AS/countries in cloudflare settings, or request a challenge/capcha from every visitors. Cloudflare will reject visitors (with or without challenging them) at their edge location before any traffic can get to you. It is very effective from my experience.

Generally speaking. The only way to stop a DDoS is to do it before it reaches your datacenters so you need help from your ISP/provider/CDN.

Edit: The attack that put down linode last christmas was against linode itself and not a specific customer. Part of the mitigation included linode moving its critical services behind cloudflare :D

jtl999• 10 years ago

DigitalOcean has had a few DDoS attacks targeting their SFO1 datacenter over the past few months, but fortunately each one seemed to disappear in under half an hour.

hehheh2• 10 years ago

DigitalOcean is the source of many DDoS attacks. They've got a lot of compromised servers on their network and ignore abuse complaints.

simonmales• 10 years ago

Play by Play of Linodes 'twelve day attack' over Christmas and New Year 2015

https://blog.linode.com/2016/01/29/christmas-ddos-retrospect...

m0atz• 10 years ago

Any details on the actual size of the attack and the attack vectors used?

constit-protec• 10 years ago

Godaddy has been getting attacked a lot recently as well. Who is likely behind attacking servers, whether other server companies or governments?

bogomipz2• 10 years ago

Organized crime syndicates? Vigilante hacktivists?

hehheh2• 10 years ago

GoDaddy, among others, host servers that are a major source of DDoS attacks. I am not sad that they are also victims.

mikebutash• 10 years ago

When you are the dns root record and in many cases hosting too for some 60-70 million domains, someone in the internet is wanting to attack someone at godaddy all the time. There isn't a time they aren't being attacked somehow in all reality, and I'd presume it's much the same for Linode.

The means of really combating a ddos is costly and extensive, this is why most use a service like prolexic or silverline, and typically with some massive infrastructures that comes with it. Anything less than n-by 40gb disposable internet pipes, preferably regional as you are, you can/will be smited at will.

ablagoev• 10 years ago

I've always wondered, while in similar cases GCE/AWS can handle the traffic, is it not chargeable? So, while you will probably not get affected by the DDoS, aren't the costs going to cut your head off?

technion3• 10 years ago

Related, if the costs reach a point that you cannot pay them, is it not a successful, unmitigated DDoS?

kalleboo2• 10 years ago

Isn't incoming data on AWS free? Or are you thinking of some kind of attack where they're using your infrastructure to amplify outgoing data?

kalleboo2• 10 years ago

Isn't incoming data on AWS free? Or are you thinking of some kind of attack where they're using your infrastructure to amplify outgoing data?

Edit: I guess an attack that causes your infrastructure to auto-scale could get expensive REAL quick...

technion3• 10 years ago

I'm running the maths right now.. and you could convince me to take down my side project by just having a server outside their network put wget in a loop targeted at my S3 resources.

Dowwie• 10 years ago

I have no evidence to support this theory, but I believe that Linode is not an outlier with regards to frequent DDOS attacks. What makes this company special seems to be with how it communicates to its customers when it's under attack.

This leads me to wonder: How much do other providers leave customers in the dark?

bogomipz2• 10 years ago

I really feel bad for these folks. Does know if they have a DDOS mitigation strategy other than RTBH with their transit providers? I would have thought that after the 2015 attack they would have looked into traffic scrubbing with something like Arbor Network or Prolexic. I understand that these are not cheap and Linodes as well as many other hosting provider's margins are probably thin but I would think that it would pay for itself in one or two attacks by minimizing customer churn an event like that causes.

xmatos• 10 years ago

I don't get the hate towards linode here, on hacker news. I've been their client for a couple of years now and I find it an excellent vps provider. Excellent uptime and performance at a pretty good price. AWS has a few outages every year. Google just had one last week. Azure sucks balls. So, why the hate? Is it because it competes with some ycombinator startups?

fizzbatter• 10 years ago

I don't follow this too closely, so this is just wild speculation from me:

But could it simply be severity of the attacks? I keep seeing comments about a 2 week ddos attack last christmas - that's something that i would be shocked to see Google/AWS succumb to. Not that Google/AWS attacks don't happen, i just can't imagine them being down for ~2weeks

(I imagine it was just one datacenter from Linode, not the entire service, fwiw)

warbiscuit2• 10 years ago

They weren't down for the entire two weeks, but various datacenters went up and down for hours, then quieted down for a few days, then was back again, then another hit; stretching across two weeks.

One thing that took them so long was that their upstream ISPs at some of the datacenters were themselves unable to handle the DDOS, so they had to switch ISPs, which took a while.

I don't see Google/AWS as easy to attack; but I'm not sure why similar tier players like DigitalOcean aren't being hit -- or maybe they're just less transparent about things, or are actually a smaller target (didn't think they were?).

edit: here's a postmortem from linode of the christmas attack - https://blog.linode.com/2016/01/29/christmas-ddos-retrospect...

hobs• 10 years ago

No, its because at one time the community actually liked and recommended it and then got burned and acts accordingly.

Linode has a bunch of great features, but after seeing them get hacked a half dozen times over really silly things, more DDOS's than you would be happy with, and frankly I have had interactions with their management (just online) and was sorry to have had said interactions.

You can also read a bunch of implications from former employees about their management, but you can feel free to discount that given how many times ex-employees are a bit pissed.

- Former Linode Customer

zodiac• 10 years ago

What hate? The post is literally from linode itself

boulos• 10 years ago

For us (Google), I think you're referring to: https://status.cloud.google.com/incident/compute/16017

which was caused by our own maintenance several weeks apart (the root cause description is really quite good).

I think the distinction people make implicitly is a 25 minute outage versus 8 hours. DDoS attacks suck, but they're just standard these days. As a customer though, any source of (network) outage usually has the same outcome: "my site is unreachable (and I don't care why)".

The reason we (and AWS and others) offer multiple datacenters/zones within a <1ms boundary (a compute "region") is so you can build a highly available app that can fail over with minimal degredation. For customers that were using App Engine Flexible environment with the regional spreading turned on, only some of their instances were affected, but their apps shouldn't have skipped a beat.

Linode is good at what they do, but any customer in Atlanta just had to wait this entire event out.

Disclosure: I work on Compute Engine.

ripken• 10 years ago

You get what you pay for. TCO of running a highly available data center is more than just the cost of metal, real estate, power, cooling, and cable

Going cheap with one of the non-big 3 public cloud providers will cost you.

ekiara• 10 years ago

I remember a few years ago when I moved my Linode from Fremont to Atlanta to avoid the frequent outages. I've never had show-stopping issues with Linode and the customer service has always been fast and responsive. Now though, I'm thinking of moving to their Frankfurt datacenter.

But now I think I need to setup fail-over with another VPS provider. What's a recommended alternative? Is Digital Ocean the next best choice after Linode?

isarat• 10 years ago

Linode is a cost effective solution comparing to AWS. But the security and DDoS issues could raise eyebrows and bring confidence issues with customers. This has happened even after they posted about enhanced DDoS mitigation strategies like procuring more bandwidth etc.

If you look at the history of Linode discussions under HN, most of them were related to DDoS attacks and service downtimes. https://news.ycombinator.com/from?site=linode.com

I hope they will recover soon and make the service stable.

“We have been experiencing a catastrophic DDoS attack”

Related Stories