Linode under large DoS attack

http://status.linode.com/2014/02/resolved-dallas-connectivity-issues.html

by simcop2387 • 12 years ago

56 51 12 years ago

status.linode.com

simcop23875• 12 years ago

Posted this because I can't get at the management page anymore and my NJ server has now gone down too. No word yet from them if it's being affected now or if it's just me.

rajjalan• 12 years ago

Having same issues as well.

Edit: It is a Dallas instance and we had few minutes of downtime. It is working now normally.

simcop23875• 12 years ago

based on the status updates i suspect that something got out of whack after mitigating the DoS and it's affecting multiple datacenters.

agwa2• 12 years ago

Really, your NJ server too? NJ has been rock solid for me today, unlike Dallas.

simcop23875• 12 years ago

yea didnt' last long, but was a black hole for about 10 minutes, lost all of it's connection to outside services too with no indication of anything wrong on the server itself.

invisiblea• 12 years ago

Fourth time this week?

leoshk• 12 years ago

Linode has been having DoS issues over the past few months. It feels like they've really been slipping over the past six. Has anyone left Linode because of the DoS issues? Are there other good alternatives that folks love out there?

stevenmays• 12 years ago

I just switched to digital ocean for a lower price.

simcop23875• 12 years ago

I haven't but have been looking. Digital ocean is competitive on everything but storage space for me, i'd end up paying the same for 10GB less or pay for more ram/cpu than i need to get the same storage. (though this is largely because of the bulk discount from paying for things a year in advance).

dhimes2• 12 years ago

I chose Digital Ocean recently because my needs are rather modest right now and they are cheaper at the low end. I imagine that Linode will get this worked out. It's probably only a matter of time until DO gets this too, so I wouldn't have picked based on this.

agwa2• 12 years ago

I've got Linodes in every data center except Atlanta. Dallas has been quite unreliable in the last six months, but the other data centers have been OK. (Fremont had some bad DoS and power issues a while back but it seems to have been OK recently.)

I wonder if Dallas is being affected disproportionately because Linode's own website and management console are hosted there, and that's what the DoSers are targeting.

VLM• 12 years ago

Somewhat mystified what strategies a victim can use in an attack other than paying bribes or protection money. Obviously the attackers scale faster than defenders so there's no technological solution. Do a bad job, so you don't make money, so you won't be targeted by people shaking down providers for money? I suppose one strategy would be to cover up problems and not talk about them, so obviously a provider not reporting problems couldn't be having problems?

notastartup• 12 years ago

yeah I was a linode user for few years and am using digitalocean and I've never looked back. I think the droplet API is by far the most powerful feature, it's so simple to use and understand compared to Amazon. For example, if I need a new development box to test something out I can spin up a new droplet and shut it down when I no longer need it and save the money. It's pretty insane how much of a cost save DO offers.

simcop23875• 12 years ago

That's one use case I can definitely see being for DO but doesn't make as much sense for me. I've just got the linode for handling random services (VPN, IMAP, WWW, and a few others) and need it up constantly. For dev work I've always got a local machine that has enough ram and storage to do that instantly right next to me and not have to deal payment for it.

Brian-Puccio• 12 years ago

I switched from Linode to Uptano[0] almost a year ago now. I heard about them here.

They don't have many of the features Linode, DigitalOcean, etc. have. But they've been super responsive to the (two?) questions I had. Their one ~$85/mo server replaced five Linodes I had. Better performance for less money. Though, as I understand it, Linode has upgraded their offerings since then.

[0] https://uptano.com/

nivla2• 12 years ago

This is the second or third time this has happened this week. They put this status on their page right after it was back up and after I had complaint to them about the random outages and no mention of it on the page. Not claiming they put it up because of the complaint but that it coming after they whole debacle made it the least useful.

I usually stick up for Linode for their excellent CS but this was disappointing. My site was in the middle of a large traffic for an event signup when it went down. SSH failed, accessing Linode's website failed, no mention on twitter or other social media, status page only mentions regular maintenance and not even the last incident. No knowing what was going on, left me feeling helpless. :(

Excerpt from the chat:

...The two very large attacks we experienced recently were both targeting the Dallas data center however.

threeseed• 12 years ago

Seriously switch to something else.

They have been doing this for years and clearly they still haven't learn.

thaumaturgy• 12 years ago

When there's a metaphorical fire in the data center, a lot of companies have a tendency to focus first on putting out the fire and second on letting the customers know there's a problem.

So Linode's not alone in that, but hopefully they can get better at it anyway.

I do wish they'd take advantage of their pretty nifty infrastructure and make it so their management system isn't all housed in Dallas. Every time there's an issue in Dallas their management system becomes unusable.

nivla2• 12 years ago

a lot of companies have a tendency to focus first on putting out the fire and second on letting the customers know there's a problem.

Having been told second would have been atleast comforting. I was disappointed because there was no mention or report of the last DoS attack or downtime, as if it never happened. The last incident lasted for about 17mins but since it was off peak hours, it din't bother me much. Then the same thing replayed today but this time at the peak of the traffic. You would atleast expect a hosting company to have their main page/portal balanced across different regions but nope. Its just sad because except their handling of situations like this, there is nothing really to complaint against them.

markdown• 12 years ago

It's not a one-man shop, is it?

There ops/sysadmins shouldn't be handling customer relations/support.

It takes about 3 seconds to stand up and yell across the room to the frontline staff, telling them that servers are down and you're working on it.

Then the frontline staff should only have to make a few clicks for announcements to go out (pre-written templates outlining what's going on and what's being done to fix it) via social media, etc.

kbar132• 12 years ago

Blame networks that don't implement BCP38

http://www.bcp38.info/index.php/Main_Page

codexon2• 12 years ago

Which is the majority of networks. It will never happen, BCP38 is a pipe dream.

haberman• 12 years ago

Why? (For those of us who have never heard of it)

seancoleman• 12 years ago

I think it's interesting that this is hosted on a TypePad blog (You can tell from the favicon).

kochman• 12 years ago

This is standard operating procedure for many companies in the VPS and server market. It's to ensure that they can keep customers updated even if all of their infrastructure goes down. Some companies take it a step further and put their status page on a completely separate domain, e.g. DreamHost's http://www.dreamhoststatus.com

hrrsn• 12 years ago

And a separate network :)

$ host dreamhoststatus.com

dreamhoststatus.com has address 96.126.109.82

$ nslookup 96.126.109.82

Non-authoritative answer:

82.109.126.96.in-addr.arpa name = li366-82.members.linode.com.

USNetizen3• 12 years ago

I used Linode in the past for a number of clients, and had no issues. Then, all of a sudden, in the past 6 months or so my clients have been repeatedly hit with attack after attack after attack. Linode "support" blamed us, as if it were our fault and charged us hundreds of dollars in bandwidth overages.

Needless to say, all of my clients now reside on other platforms and are entirely attack-free now and have been since moving off of Linode.

Their service has gone so far downhill.

jaskerr• 12 years ago

Care to clarify to which platforms you have moved your clients? I've considered Linode, but haven't "pulled the trigger" just yet, and all this turmoil has me reconsidering. Thanks in advance.

USNetizen3• 12 years ago

I've moved to Windows Azure (for my Microsoft-using clients) and Amazon for the rest.

Linode support has been absolutely crude and even berating in their responses to me. They have lost all sense of professionalism.

It is sad to see a once promising hosting company go so far downhill so fast, but such is the nature of trying to be the lowest-cost option.

jethro_tell• 12 years ago

When you say linode 'blamed you' do you mean that your customers were directly attacked (their ips) or that the platform was attacked? Does linode have an anti ddos product that the sell?

grey-area2• 12 years ago

I haven't noticed any downtime on the London datacenter, is this just affecting Dallas or is it more widespread?

teh_klev• 12 years ago

Re-read the status page, three times it mentions "Dallas" :)

ghx• 12 years ago

I've used Linode for the last 4 years, and prior to this week, I've only had 2-3 instances of downtime, usually only for a few minutes. This week, it's happened 3 times, which really sucks.

I've been using DigitalOcean for about a year, and I experience downtime with them (NYC1) about once or twice every month. But they're $5.

This sort of thing happens to every VPS provider at some point, and switching to DigitalOcean isn't going to make a huge difference in that respect.

I do wish Linode were a bit better about updating their status page. I don't like relying other peoples angry tweets to determine if it's Linode, or just me.

USNetizen3• 12 years ago

It may happen to other providers, but it happens to Linode almost weekly - and they just turn around the blame (and charges) on the customer.

akerl_• 12 years ago

Since Linode doesn't charge for any incoming transfer, can you clarify what you mean?

api2• 12 years ago

I've had no appreciable downtime on Digital Ocean, and have a network scattered around: NYC2, AMS1, SFO1, and now Singapore (whatever they call it). My app is low-RAM, low-CPU, but possibly high bandwidth and their stuff is ideal for that.

gtaylor2• 12 years ago

I'm in NYC1 and NYC2 and it seems like there is maintenance at least once a month, if not more. Luckily, my application is very fault-tolerant, but it is surprising how often they are emailing me about maintenance outages (however brief they may be).

jpea• 12 years ago

So, since there's really no difference between $10 for DO and $20 for Linode, what providers are out there that'll have a solid API or provisioning panel, a great network and great service? Doesn't matter if it's 50 bucks a month. Or 100. Anyone? What the heck do people use for business VPS hosting??

Most of what I've seen comparing any of these always brings up price, but that's like bickering over couch change if it matters to your business.

zerop• 12 years ago

Giving public IP to every node is what I dislike about linode, also No security group like EC2. No VPC.. one has to manually configure firewall on each node. 'Linode managed' costs $100 per linode, insanely high. Despite of all this I am on linode only because of CPU :)

mryan• 12 years ago

This has also been affecting WP Engine: http://wpengine.wordpress.com/2014/02/12/intermittent-connec...

Linode under large DoS attack

Related Stories