Digital Ocean NYC2 Droplets Down

https://news.ycombinator.com/item?id=8103766

by heavymark • 12 years ago

52 68 12 years ago

news.ycombinator.com

And counting. Second time in about a week and during prime time hours. If it doesn't get up quickly or if happens again to NYC2 in the next few weeks may be time to reconsider DO for hosting production sites. I love them, and glitches happen, but twice in a row clients have a hard time swallowing.

n0nenet• 12 years ago

I have to agree with you on this one.

marcusr• 12 years ago

It seems to be http (perhaps load balancers) rather than the instances being down - I'm ssh-ed into a droplet and everything is working fine, except remote HTTP connections.

iurisilvio2• 12 years ago

I ssh-ed into some droplets, but others are completely inacessible outside DO. The internal network works fine, so it is just a several network outage. Again.

publicfig2• 12 years ago

I'm struggling to ssh into droplets, but it seemed like when I was ssh'd in, I didn't have an issue at all.

mikeash• 12 years ago

It seems to be intermittently up and down. So your connections all die, and you can't reconnect for a while, then it comes back and everything looks good for a bit until it dies again.

toomuchtodo• 12 years ago

What happens when you try to execute outbound web requests from the instance?

http://netalyzr.icsi.berkeley.edu/cli.html

JohnTHaller5• 12 years ago

I've been unable to SSH in for an hour and a half. But I expected this when I signed up (cheap = cheap) so have backups running that were switched to.

mbreedlove• 12 years ago

I'd love to know what's going on. My website hosted on DO, michaelbreedlove.net, is unavailable. I also have a few services running on subdomains of the same DO VPS instance; they're all accessible. I also have SSH access. They seem to be having some serious HTTP routing issues.

publicfig2• 12 years ago

I just created a droplet today, and was trying for the life of me to figure out what I would have done so early on in setting it up to cause it to become unresponsive. Glad to see it's not just me. It's not really a great sign though, as I was hoping to use this fairly frequently. Is this a common occurrence, or do I just have bad luck? If so, I'll definitely have to find another solution.

zhodge3• 12 years ago

I'm not too qualified to comment on the hosting market, but I do want to vouch for DigitalOcean as I've never had uptime issues with them until now. They also have always had timely and helpful support and their pricing is obviously competitive.

That being said, near 100% uptime is really desired in this space and I've been scratching my head for the past hour or so wondering why I couldn't connect to my mongo process running on my droplet...maybe I should have tested DO sooner, but the fact that only mongo seemed to be failing (and not postgres for example) made me think I had client and not server issues.

gingerlime• 12 years ago

I've been using Linode for many years, and DO for some time now (a year? not sure, since I can't even login to their control panel right now and check). I've seen issues on both providers, and dealt with support of both not too infrequently.

I'd still say Linode is miles ahead in terms of expertise and responsiveness. Most of my servers are on Linode for this reason, and only a few on DO.

As a customer I enjoy this competition and would like to keep both players compete - because I'm benefiting from it. So I'm likely to stick to both. But I would still recommend Linode more easily than I would DO.

heavymark2• 12 years ago

The big issue here is NYC2 droplets went down only 7 days ago. This time its even longer at over an a half. When it happened last week it was unfortunate, but things happen. When it happens twice in a week that is a much bigger problem.

jqueryin• 12 years ago

Having previously left Linode for Digital Ocean due to lower pricing and SSDs, I can safely say Linode both dropped their prices and matched the DO plans with SSDs included. If you want to jump ship, that's the route I'd go for a comparable.

I still love DO, but downtime is a no-no when it comes to VPSes.

ramatgan• 12 years ago

In the process of switching over now.

JohnTHaller5• 12 years ago

When you charge so little and the SLA is just credit during the time of the outtage (pennies), downtime is pretty much a given. It always surprises me that this surprises people.

uptown• 12 years ago

Here's their status panel: https://status.digitalocean.com/

dangoldin• 12 years ago

One of my instances on NY2 is fine and the other one is sporadically accessible via SSH. Hope they figure this one out soon.

jcampbell1• 12 years ago

Linode's new offering probably the better choice these days. I get the feeling that DO is just overwhelmed with problems related to their hyper growth - 10% per month on a huge base.

http://trends.netcraft.com/www.digitalocean.com

mctDarren• 12 years ago

They've done so well to this point keeping up with the growth curve. Hope they don't become crippled by success.

bmoresbest55• 12 years ago

This is a little ridiculous. I hope not to have this problem again. Second strike her. Unfortunately.

potomushto• 12 years ago

https://cloud.digitalocean.com/ now is unavailable too.

alokyadav152• 12 years ago

Yeah , NYC2 seems affected https://twitter.com/digitalocean/status/494170340973887488

kl4m• 12 years ago

Their status page indicates "We have identified a slow connection to our NYC2 data center". Assuming they have redundant connections, this seems like a big understatement for being down for 1 hour and going. Even their control panel is down for me. ( http://i.imgur.com/3ltg00o.png )

bigredtech2• 12 years ago

I'm a big fan of Linode. Have had one of their mid-range servers for about 2 years now and have had no troubles. When it came time to upgrading, or any other questions I had, their support staff is 1) quick, 2) knowledgable, 3) thorough & helpful.

I've used DO in the past for quick spin-ups, and they're great for that. But for long running applications I'd stay away. Was actually considering DO for a new project, but this kind of tipped the scales the other way.

If you're looking to switch, I think Linode gives you the best bang and reliability for your buck. Have also used Rackspace for a previous startup. A bit more pricey, but also a great service if you're considering switching.

meritt3• 12 years ago

Linode can't silo resources on a per VPS basis and constantly will harass you for over-utilizing your share of arbitrarily enforced limits. Just a heads up if you actually utilize your instances.

harpastum• 12 years ago

Are you talking about the "Linode Alert" emails? Those are user-configurable — in the email, they specifically state "This is not meant as a warning or a representation that you are misusing your resources. We encourage you to modify the thresholds based on your own individual needs."

If that's not what you're talking about, I apologize, but I've been using Linode for years (often with very compute-intensive projects) and I haven't seen what you're talking about.

opendais6• 12 years ago

Really? I find that odd. I do load testing on their instances all the time. which maxes out cpu/disk to give me an idea of performance changes for stuff I have high throughput on.

Hell, I've gotten DDoS'd for like 2 Gbps and they were like "Oh, ya, its fine."

So, can you explain more about what happened?

bigredtech2• 12 years ago

Have run a web-app with 30K+ users. Only been 'harassed' when I've been warned that I'm running the CPU at 350% of max rate and I should be concerned. That's ok in my book.

Ronsenshi3• 12 years ago

I find that surprising. Linode support never contacted me regarding resource usage. Not even when one of the servers has been under DDOS. I contacted them first notifying of such event. And that was it.

I agree with harpastum. I think you're confusing automatic email notifications based on your own settings with Linode support. I feel like you have to seriously abuse their infrastructure to get them to contact you.

Disclaimer: has been a happy Linode customer for over 4 years now.

nilved2• 12 years ago

If you'd used their servers for a bit longer, you'd have plenty of trouble. Linode is notorious for dreadful security practices. Their main ColdFusion app (yes, really) has been hacked on numerous occasions, including at least once incident that involved customer data loss. Linode dodged the issue and didn't tell their customers until badgered.

I can agree that DO isn't suited for production apps, but Linode isn't either. They're both VPS playgrounds.

antsar• 12 years ago

Is there a better alternative you would suggest? While Linode has done better than DO, that fiasco left me feeling somewhat uneasy as well.

Ronsenshi3• 12 years ago

Same question as antsar. What would you suggest then?

olso• 12 years ago

I use DO because they have servers in Amsterdam, which happens to be very close to my country.

I never had problems with DO that weren't resolved promptly.

Just a note, Linode is very US-centric anyway... (and that one London location doesn't make it international very much)

Sir_Cmpwn2• 12 years ago

I am getting pretty sick of it, too. I might go back to Linode. I was already upset that they dropped Arch Linux support.

opendais6• 12 years ago

Linode has price matched them so its not a price issue anymore. [e.g. $10 Linode is roughly the same as $10 DO]

Glad to see the ProjectWonderful ads providing something, even if its small btw. :)

Sir_Cmpwn2• 12 years ago

It definitely helps. The biggest source of income has been (surprisingly) Gittip, though.

JohnTHaller5• 12 years ago

You should probably reconsider hosting production sites on super-cheap VPSes with a 1:1 SLA refund. They're going to go down. And, when they do, you'll get the 7/10ths of a cent refund per hour promised for your $5 instance (1.4 cents per hour for $10, 2.8 cents per hour for $20, etc). But if you depend on a site for your business, ask yourself how much it costs you per hour of downtime and plan appropriately.

For context: I'm a happy Digital Ocean customer (as well as Rackspace and others). I like their services and consider them a great value for the money. I've also gotten multiple developer friends into using them. But I don't let them be a single point of failure for anything critical.

kbar13• 12 years ago

downtime is an issue no matter how much you pay your infrastructure provider. You should account for downtime in your application and your infra by taking advantage of low-hanging high availability fruit.

Things like saltstack (saltstack.org) make this mega-easy.

JohnTHaller5• 12 years ago

It's easier for apps since you can build in failover support based on multiple servers pretty easily (I just implemented this in our 'live' installers recently on Windows). It's a bit harder at the website level for an average user, though. I know far too many people that have a single point of failure website on a super-cheap VPS that's just down whenever the VPS is down. Most of them would have no idea how to implement a load balancer and failover support. Though I could probably get them into something like CloudFare or similar.

neom• 12 years ago

we've had an ongoing issue with a juniper fiber core in nyc2 it seems. We've engaged juniper to help us, I don't expect this to be an issue again.

alokyadav152• 12 years ago

its working now , atleast for me .

JohnTHaller5• 12 years ago

I'm back up as of 2:59pm EDT, so I was down for about 1:35. This is the only outtage longer than a few minutes I've had with DO thusfar.

larrys• 12 years ago

"but twice in a row clients have a hard time swallowing."

May I ask what your clients are paying?

Maybe the answer is to charge them more instead of giving them a low price (an assumption on my part of course which is why I ask what you are getting for this) and some implicit guarantee of uptime.

For example we had some customers some time ago who were pooled on a shared IP address for sending SMTP service. Occasionally they would experience problems with mail reputation on that shared IP. If they complained about that would simply explain that that was the drawback of a shared IP and that they could always upgrade to a service level where they got a unique IP and could establish their own reputation.

Donzo• 12 years ago

I've had great experiences with them. I recommended DO to everyone.

Now I'm tempted to drive to New York and punch them each in the face.

PEAK downtime.

EDIT: $160 refund. Maybe just a Charlie horse.

opendais6• 12 years ago

Folks, I strongly, strongly suggest if you use Cloud/VPS providers to look into something like Cassandra, Galera, Mongo, etc. that can do WAN replication and using 3 DCs.

Yes it costs money but things like this are avoidable. Every single Datacenter on the planet has downtime and/or performance issues from time to time. Hell, one of the places I have a dedi at had its network performance drop by 50% for about 15 minutes while they did emergency maintenance.

Things like this are going to happen as long as you depend on one provider.

rrss1122• 12 years ago

We're getting the same thing. I don't care how cheap it is or how nice the customer service is, but if they are not up 99% of the time they will not be my web host.

wglb3• 12 years ago

Er, no.

My droplets are up. Never went down today.

Also, please don't post that things are down here on HN. This is not a support forum. Not for DO, not for AWS.

iurisilvio2• 12 years ago

You came here after it was fixed. It is fine now. Also, it was just NYC2, maybe only part of it.

wglb3• 12 years ago

Mine are on NYC2. Terminal sessions up all day. On three of them.

Check http://downforeveryoneorjustme.com instead of HN.

hadoukenio2• 12 years ago

Why do people have their eggs in one basket? More specifically, whenever hosting comes up on HN, it's always DO vs Linode. Why not Zoidberg^Wboth?

In the age of DevOps, it's not hard to configure different providers. I have been using Linode and DO for a while now. I noticed DO go down last week, checked my site, and it was UP because it was being served via Linode.

I flinch every time I hear someone say to me that their service has redundancy because they use multiple Amazon regions. Multiple providers people! And preferably multiple providers in multiple states, paid for by mutually exclusive credit cards!

Redundancy doesn't mean having multiple hosts by the same provider, within the same data center, within the same rack, provided by the same power strip.

tl;dr: Plan for outages.

opendais6• 12 years ago

It is always DO vs. Linode because both are relatively stable, relatively good for their price points. Tbh, the only real reason I use Linode over DO is the fact I've seen higher restarts on my DO nodes and the fact I've attracted DDoS's at a size Linode shrugs about.

anderspetersson• 12 years ago

How do you do this on the DNS layer? Very short TTL and switching on failure detection? Do you use both providers at the same time when both are up via DNS round robin or similar?

hadoukenio2• 12 years ago

There's nothing stopping you from adding multiple A records for a domain. In fact, that's what a lot of people do:

  dig a www.google.com

  www.google.com.         300     IN      A       74.125.237.178
  www.google.com.         300     IN      A       74.125.237.177
  www.google.com.         300     IN      A       74.125.237.176
  www.google.com.         300     IN      A       74.125.237.179
  www.google.com.         300     IN      A       74.125.237.180

Browsers these days will resolve and try each one, on a first-come-first-served basis. If one fails, it will try the request on the next server.

stevekemp• 12 years ago

If you have your own IP-space you can use BGP to route IPs to different locations.

If you're relying on DNS then short TTLs are pretty reliable, but there can be extended outages due to erroneous client-caching.

Digital Ocean NYC2 Droplets Down

Related Stories