DigitalOcean lost our server

https://murze.be/2016/02/today-digitalocean-lost-our-entire-server/

by ayi • 10 years ago

290 205 10 years ago

phamilton• 10 years ago

I'm glad the author didn't turn this into a negative review of DO. "What will you do when your server is lost?" is exactly the right question to learn from a scenario like this.

sarciszewski• 10 years ago

That's what I half-expected when I read the title, as well.

freekmurze3• 10 years ago

I'm well aware that this could happen with any hosting provider. Sooner or later everyone will encounter this kind of problems.

Forewarned is fore armed!

cortesoft2• 10 years ago

I think it was an ad for some of their own services instead.

pdabbadabba2• 10 years ago

Agreed. There may also be a customer service lesson in that refund email. Typically, your customer is not happy with you even after they receive a refund, so exclamation points and language like "Booyah!" is really not a good idea for that sort of message. It's always going to sound a little bit tone deaf -- and very tone deaf when the refund was for a major incident like the loss of an entire server.

Keep it professional.

dubcanada• 10 years ago

It's the standard DO credit email, which typically is from referring friends or doing something else to acquire credit. Not getting a refund because DO accidentally deleted your droplet.

Though their should be another avenue for such things.

click170• 10 years ago

Not to mention that service credit is more than I've ever gotten from Amazon when they've lost one of our VMs due to underlying hardware failure.

gdgtfiend• 10 years ago

This seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"

bluedino• 10 years ago

Remember the blog posts during the early days of AWS? People didn't realize their new elastic cloud servers could go into thin air

Piskvorrr• 10 years ago

Dissolve into thin air? That's what clouds do, right? (One woulda thunk the word "cloud" in itself would be warning aplenty ;))

__michaelg• 10 years ago

Agreed. This shouldn't be an issue for anyone using cloud services -- or any computer really. Except for some super duper redundant mainframes maybe, you should always be able to deal with losing a component not matter what it is. If you really care, always have multiple instances, in multiple (availability) zones, etc.

theptip• 10 years ago

Exactly, to use a now-trite metaphor: your cloud instances are cattle, not pets.

cortesoft2• 10 years ago

It is no different if you are running your own hardware - your server hardware can fail no matter who owns the machine.

lotyrin2• 10 years ago

Yep, somehow cloud customers have decided that by being in the cloud everything is redundant and durable and has several other magical properties.

At one of my positions we had stupidly put too many eggs in the basket of a single physical machine. Its disk controller failed in a way that it trashed the data volumes. I was unable to convince anyone that "move to Amazon" was not a one-step solution to "how do we make sure this never happens again".

techjuice• 10 years ago

This is so true, I wish there was a way to make this more noticeable (flashing red lights on the order forms, regular email warnings about the single point of failure,etc.). This simple fact is taught in entry level system administration courses (RAID, Disaster Recovery Plans, Replication, High Availability, Fault Tolerance, etc.) but in reality some seem to not plan for it when moving to cloud services. I always suggest people set their apps up to be able to run out of multiple availability zones in case one goes down if it is very important to them. For those that are not able to I will suggest they at least setup replication to another server within the same data center and an offsite location in case their server has hardware failure, an account gets locked or other possible situation to help take of the just in case scenarios that occur to everyone.

Dylan16807• 10 years ago

If your server does something weird and corrupts the filesystem, you still have access to it. That can be a huge difference when it comes to restoring the most recent data.

watty• 10 years ago

Isn't "non-issue" a big of an exaggeration? If the dry cleaners lost my clothes, the bank lost my money, a valet lost my car, or gmail lost my inbox I'd be angry.

chc• 10 years ago

Yes, but those aren't expected outcomes when using those services. That's the difference gdgtfiend was pointing out — you should expect cloud servers to sometimes go away.

geofft• 10 years ago

If the bank lost the particular dollar bill that you deposited last week, but offered you a new dollar bill, would you get angry? If you rent a car from the airport every time you visit a city, and it's always been the same car, but you show up one night and they they tell you the car was in an accident and they'll get you another car, would you be angry?

nkozyra• 10 years ago

The expectation with these services you mentioned, however, is reliability.

The implicit (and explicit) volatility of cloud hosting should change that expectation with such services.

"Non-issue" is an exaggeration because it was potentially a catastrophe. But this is sort of the way these services "work." Servers/droplets/instances are ephemeral and replaceable, and their underlying data is not guaranteed in a failure.

bobdole1234• 10 years ago

The dry cleaner and bank don't tell you to make sure you have backups.

endymi0n• 10 years ago

While my first intuition was to agree with you, there's certainly an upcoming generation of developers who have never operated their own root servers and the abstraction level in the cloud nowadays is so high it makes you easily forget that "Droplets" are just VMs are just servers running software. On the other hand, hardware reliability has increased in recent years due to RAID, fully redundant networking & power adaptors, you name it.

These facts combined certainly make it easy for the younger generation to forget that redundancy and backup still don't replace each other (never did).

beat• 10 years ago

Well, there's one way to teach these kids...

(Could be worse. Their first lessons could be like mine, in C! If anything will make you paranoid, writing C will.)

XorNot• 10 years ago

This seems like the wrong lesson is being taught though somewhere - the delete button is right there. It should be much more obvious that is very easy to make a server go away.

nailer5• 10 years ago

You should be designing for failure in "the cloud"

DO should be designing for failure: VM storage should be on a SAN. A single physical server failing should not cause data loss. This is basic stuff.

vardump6• 10 years ago

That's a very unreasonable expectation at cloud price levels.

Does any cloud provider have SAN backed VMs? If any do, at what price?

random_upvoter• 10 years ago

DO should do what's written in their terms of service, and the customer should read them carefully.

tbyehl3• 10 years ago

SANs aren't magical boxes incapable of losing data.

nickbauman• 10 years ago

CAP theorem applies everywhere, but most software practices still do not assume that you must code for your app to be able to handle performance variation and partial failure state both of which are common scenarios of cloud infra.

Fizzadar• 10 years ago

I have to disagree - DO's "cloud servers" are equivalent to virtual private servers, which you would never expect to loose in this manner from other providers.

The lack of explanation is what worries me most - it leads me to think this might have been a case of "we forgot to replace a bad drive, then the second in the pair failed".

bitshepherd• 10 years ago

Welcome to "The Cloud". VPS can be effectively rebranded as "cloud", except when a server has problems you shoot first and ask later.

RubyPinch• 10 years ago

I'd expect to lose them.

Fire happens, floods happen, electrical faults happen, mistakes happen. I can't really expect any human or humans to be perfect.

halfdan• 10 years ago

..and the second due to a flood of people from HN.

Server seems down :-/

d0ugie• 10 years ago

Where's that internet permanence when you need it?

captn3m0• 10 years ago

Cache Link: https://webcache.googleusercontent.com/search?q=cache%3Ahttp...

freekmurze3• 10 years ago

My blog is up again.

autotune• 10 years ago

Error establishing a database connection.

I award OP the "Most Accurate Title Of The Day" Award.

freekmurze3• 10 years ago

Haha :-) It's up again.

bsg75• 10 years ago

TL;DR:

Backup servers no matter where hosted, because two is one and one is none.

brayton• 10 years ago

They had to make space for the YC portfolio companies getting $250k of credits ;)

fweespeech4• 10 years ago

I'd still not host production on DO unless your setup allows you to failover to a backup DC with a different provider.

andrewthetechie4• 10 years ago

Why?

r1ch• 10 years ago

I definitely wouldn't recommend DO's built in backup service in production based on my experience. It brings my site down every week since all I/O freezes during the backup. Sometimes the droplet doesn't recover and needs a hard reset. Apparently it's a known issue but hard to fix.

mrnismo92• 10 years ago

You should always build for failure, especially if you are running on IaaS. Good post & a great reminder for many!

justinhj• 10 years ago

This has happened to me with Amazon servers too, it's just the nature of the cloud. I think Digital Ocean handled this as well as can be expected.

Glyptodon• 10 years ago

Between the lines takeaway: hardware raid is dangerous.

protomyth2• 10 years ago

I'd be curious what their RAID is setup. I agree and run ZFS for our data storage.

andrewthetechie4• 10 years ago

Check out https://www.digitalocean.com/community/questions/how-did-do-... for your answer. It's a bit old, but still true.

ekimekim2• 10 years ago

Anecdotally: Every company I've worked for that uses hardware raid controllers, it feels like the controllers break about as often as one of the disks. Sure, still an improvement over no RAID, but still ridiculous compared to any decent software raid (md or ZFS).

scurvy3• 10 years ago

You lose a RAID card as often as spinning metal disks? Sure that's not a wild exaggeration? Sure sounds like one. In 15+ years of doing this, I've only been a part of a RAID card swap three times. Once was just to be safe with an overly paranoid customer.

That said, I'm a fan of moving away from RAID. It's just more complexity in the face of a movement towards simpler architectures. The one benefit of a good RAID card: stupid fast, safe writes with a BBWC.

seanwilson• 10 years ago

What if you need to scale up to more servers? What if your server gets hacked and you have to recreate it? What if you accidentally delete the server or perform an upgrade that completely breaks it?

None of these would be an issue if you've scripted server recreation from scratch and permanent data is stored in high availability shared databases and services like S3. If you're relying on weekly backups to save your server state and customer data you're just asking for trouble. I like that Heroku recreates your server once a day to force you to do this properly.

z3t42• 10 years ago

I don't know how to write this without upsetting quite a lot of people, but RAID!???!

I run Redundant disk controllers and hard drives on ZFS. And it's cheaper than a decent RAID system.

First thing I do with a new server is disabling RAID.

helper• 10 years ago

How many servers? 1, 100, 1000,10,000? Where are you hosting them, in your home or in a datacenter? How much are you paying for bandwidth and power? What SLAs do you have in place for network and power? How much do you pay for your remote hands?

I hope you see what I'm getting at. While I love ZFS and think self-hosting can make a lot of economic sense, without knowing all the variables you can't really make a fair comparison.

z3t42• 10 years ago

I understand there are huge benefits of cloud hosting. But why are they running RAID? Why can't my setup be used at scale!?

fidget• 10 years ago

As if this can't happen on physical machines also.

chrismartin3• 10 years ago

Probably due to optimization of the conversion funnel, it's VERY easy to build things on DigitalOcean without understanding that - unlike many other *-as-a-service - droplets do not include backups and that is your responsibility. As a sysadmin, this is fine since I don't trust a single provider for both prod and backups anyhow, but I fear for those who are just getting started and don't realize this.

I agree that DO handled this situation appropriately, but they should warn people a little more deliberately when they sign up and create a droplet. A checkbox along the lines of "I, user, understand that I'm responsible for backing up my server", with a few links to some how-to articles of the excellent quality that DO is known for, would suffice and empower without reducing the conversion rate.

mikeash3• 10 years ago

I would really hope that the presence of a "Backups" checkbox on the droplet creation page is enough to tell the (presumably tech-savvy) user that backups don't happen unless you check that box.

aesthetics1• 10 years ago

That's fine, but they didn't introduce this functionality until recently. One day, I logged into my DO console (I don't have a reason to do this very often), and saw a new backup tab. Older customers might not even be aware.

I don't host anything I couldn't live without though, so I'm not really worried.

bowlich• 10 years ago

I still don't trust them to back it up. I checked the box, still make my own local backup.

joshmn2• 10 years ago

SolusVM — software that powers the majority of VPS range (I would love to argue how DO isn't anything cloud) provides providers with backups out-of-the-box. It's a matter of them setting it up, and usually almost always is. If it's not, well, you're probably with someone who hasn't an idea what they're doing.

mdellabitta• 10 years ago

Hey, anybody remember LeafyHost?

http://arstechnica.com/civis/viewtopic.php?f=25&t=238085

trustfundbaby• 10 years ago

The weekly backup deal is something I really do not like with Digital Ocean, I think Linode does it better, I've been planning to move for a while now, I've just been super lazy, but this might be the push I need. 7 days of lost data is unacceptable to me, especially when I'm paying for backups.

Three backup slots are executed and rotated automatically: a daily backup, a 2-7 day old backup, and an 8-14 day old backup. A fourth backup slot is available for on-demand backups.

chrismartin3• 10 years ago

I see DO's weekly backup as a convenience feature for if you need to restore your server, not as your primary backup method. Use the DO snapshot to restore most of the way, then run an up-to-date differential restore from your other (real) backup solution (that is hosted somewhere else).

gedrap2• 10 years ago

But wouldn't it be much more convenient (largely because of the complexity) to have the complete backup solution at the same place (i.e. DO)?

Obviously, you want some offsite backups rather than completely relying on DO but that should be for disaster recovery (massive failures in DO's DCs, etc) rather than a more routine one.

ryanlol3• 10 years ago

If you're going to switch to linode you might just as well use any public torrent tracker to store your backups.

vidyesh8• 10 years ago

What did Linode do this time?

gedrap2• 10 years ago

Agreed. When you actually need to restore something from backup, often you need a fairly recent state. Hours or days ago (and not 7 days ago).

Having something 7 days old might be useful in some cases (especially e.g. DB dump) but it is far from 'good enough'.

I am not sure what's the reason why they do not provide more recent backups (for additional fee, obviously).

Would it be so complicated that the engineering effort required wouldn't make it worth it? Or such a service wouldn't be popular enough among DO users?

sauere• 10 years ago

I had two DO machines with a unrepairable file system after a crash and reboot. Shit happens. Never rely on a _any_ VPS. I treat them as throw-away boxes that can fail anytime.

codazoda• 10 years ago

Although I have backups of my critical files, I don't want to try to rebuild from those files if I can help it.

After learning that Digital Ocean does no backups of their own (even for critical hardware failure on their own side) I've enabled weekly backups for an additional $1 per month.

Glad you posted this so that I know that option is available and really necessary. And who wouldn't pay $1?

Edit: It costs 20% but I only pay $5 for my personal server.

toomuchtodo2• 10 years ago

If you're running in "the cloud", you should _always_ be able to destroy your instances and have them auto-rebuild from configuration management and a persistent object storage system (git repo + S3 or Google Nearline).

notwhereyouare• 10 years ago

do you have any guides or tutorials I could possibly follow to have a setup like that? I'm working on configuring a site and would like the ability to 1) scale as quickly as possibly or 2) rebuild in event of total failure.

I just don't even know where to begin. Is this something I could stand up locally and push to like s3, spin up a new server and install 1 thing and have it pull the configs and software installs down?

Any suggestions?

vardump6• 10 years ago

Absolutely right. That's what the purest essence of the "cloud" is.

twunde• 10 years ago

DO just gave a talk (~2 weeks ago) where they mentioned that they were moving to software RAID.

vardump6• 10 years ago

Sounds great, for example ZFS has proven to be more reliable than PCI-e card based hardware RAID solutions.

EMC and Netapp are of course another matter, they're great. If you can afford them.

brobinson• 10 years ago

ZFS has proven to be more reliable than PCI-e card based hardware RAID solutions

I'm a fan of ZFS, but this is the first time I've heard this claim. Do you have any links to articles/stories talking about it?

vinceguidry4• 10 years ago

If you're relying on backups for servers other than your database then you're keeping state on your servers and that's a Bad Thing. You should regularly destroy your own servers and recreate them using your configuration / deployment scripts if the prospect of this happening worries you. Do it before your business starts to rely on it.

For database servers you need to have procedures in place to quickly switch the production application over to a database you just spun up and migrated the data from a recent backup to in order to test your data backups. You want this to happen as smoothly as possible in the case of a failure. Keep data backups in three different places and test your procedure on all of them.

On a production system, do not keep the database on the same server as the application.

snuxoll2• 10 years ago

It's so insanely easy to get started with puppet or ansible I wouldn't start rolling out a product without using them from the get-go at this point. At the very least properly package your software (read: DEB or RPM, not docker containers) so it's easy to reinstall quickly if need be.

vinceguidry4• 10 years ago

Personally, I like platform libraries rather than OS packages. I have several Ruby projects set up as gems. What's neat about that is it allows code reuse across projects.

lqdc13• 10 years ago

What? Why is it a bad thing? I guess it depends on someone's definition of a database.

vinceguidry4• 10 years ago

Because state left on servers is inevitably unmanaged and will get lost eventually. At my job, I have cron jobs running on the production system from years ago that I have no clue what they do and no time to try to figure it out. I find out when they fail and someone, usually customer service, complains.

If the server goes away, then even once I redeploy, I've lost all those cron jobs. Who knows what will happen then.

On systems I build, cron jobs are added as part of the deployment process and managed as code, in the git repository.

That's just one example. Others include iptables rules, FTP server configuration, startup scripts, and such. If it's not in your codebase, it's unmanaged state on the server that you will lose if your cloud provider takes a shit. All of that needs to be managed as code or as configuration and deploy scripts should contain idempotent commands that add them if they are not present. I consider shelling into a server for any reason to be unideal and will think of ways to avoid it.

If you are asking about database being on the same server as the application, it's because it makes infrastructure management more difficult. Application servers are set up quickly, torn down quickly, and work as soon as you deploy. Database servers are fortresses, you don't build new ones and tear them down half as quickly, unless you've got scaling requirements.

It is tempting to want to put data on the same server as the application early on in order to save on hosting costs, with the understanding that you'll separate them later, but that's a rookie move. It's much less work to separate them now, when there's nothing riding on it, much less headache.

hinkley3• 10 years ago

And the nice thing about DO and some of their cohort is that they charge by the hour for servers. Fire up new droplets and install your upgrades on them, smoke test the system, swap your routing over to the new machines, back up the old droplets (just in case you missed something) and and then destroy them.

If that takes 4 hours you've hardly spent any money at all, and you're running your cluster the way a lot of people already do.

outworlder• 10 years ago

Oh well, It's not like DigitalOcean doesn't tell you to enable backups. And you should not expect that hardware failures are now a thing of the past just because it's in "the cloud".

kentonv2• 10 years ago

I think Google Compute Engine does the right thing here: by default it uses "persistent disks" (network-attached redundant/highly-available block devices) for all disks. The only case I've heard of where persistent disk data was lost was a few acknowledged writes occurring just before an unusual lightning-induced power outage: https://status.cloud.google.com/incident/compute/15056

For added protection, you can take regular snapshots. You only pay to store the diff from the last snapshot (so go ahead and snapshot often), and snapshot storage is geographically distributed.

(Note: I have no idea what EC2 does, maybe it's similar.)

tshtf• 10 years ago

Elastic storage on AWS has a fairly low AFR (https://aws.amazon.com/ebs/details/):

Amazon EBS volumes are designed for an annual failure rate (AFR) of between 0.1% - 0.2%, where failure refers to a complete or partial loss of the volume, depending on the size and performance of the volume. This makes EBS volumes 20 times more reliable than typical commodity disk drives, which fail with an AFR of around 4%. EBS also supports a snapshot feature, which is a good way to take point-in-time backups of your data.

mwcampbell• 10 years ago

I've been suspicious of network-attached block storage ever since the Amazon EBS cascading failure of April 2011 and these three blog posts that Joyent did in response:

https://www.joyent.com/blog/on-cascading-failures-and-amazon...

https://www.joyent.com/blog/magical-block-store-when-abstrac...

https://www.joyent.com/blog/network-storage-in-the-cloud-del...

Besides Joyent, I think DigitalOcean and Vultr do it right, with RAID-backed local storage and a backup option.

kentonv2• 10 years ago

I'm having trouble unpacking useful information out of these rambly blog posts. It sounds like the basic complaint is that abstractions add complexity, which is more trouble when it goes wrong.

I agree that if you have a large-scale sophisticated operation you will probably want to choose local disks and handle availability your own way (and GCE provides those options). But for small-scale operations that can't justify as much engineering, abstractions like persistent disks and automatic migration will save a ton of time and avoid data loss. Ironically, Digital Ocean seems to target this smaller scale, yet they've set the wrong defaults for their target market.

apocalyptic0n3• 10 years ago

Does anyone have any good backup solutions to mitigate this? My agency uses a script we wrote in-house to more-or-less rsync our data to an AWS instance, but it's always seemed a poor way to handle it. I'm using DO's backup service for my personal site which was always supposed to be a temporary solution (the timing of the backups is inconvenient as mentioned by the article).

A better solution would be wonderful.

chrismartin3• 10 years ago

This isn't exactly turnkey but for my personal servers, I use rsync.net (ask for the HN discount) and attic [0]. It's easy to script and call with cron. (Of course, this is still somewhat DIY, and any backup solution is only as good as your monitoring + restore testing.)

[0] https://attic-backup.org/

Tassels4• 10 years ago

rsync.net is the shit. I can vouch for them 110%.

Also backblaze's b2 is pretty damn good/cheap. I use it as a secondary backup.

justizin• 10 years ago

"Luckily, we made the decision at Spatie to host every site on it’s own droplet, so only one site was affected."

I think that's a poor lesson learned here. Were this me, I would have said:

"Luckily, all of our sites run on several servers, access data in a shared, replicated cluster, and a small shell script I wrote kept me from writing this entire blog post."

IaaS has only surfaced what has always been true: your data lives on little physical things that are screwed into a thing and goes through a controller that could fuck up due to cosmic rays.

Do better for your customers.

hinkley3• 10 years ago

My sentiments exactly. If they lost their db replica that would get the blood pumping but the customers would never see an actual outage.

rakem• 10 years ago

I see this as a win for your processes (having backups) and DO for being honest. Stuff breaks, it happens. Good for you for having backups.

therealmarv• 10 years ago

No surprise for me. It also happened on DO for me one time. It seemed I've had my machine on a wrong rack/server there. At the end you get what you pay for I would say... you can have luck on your DO server but there is a reason why Rackspace and Amazon still exist nowadays! If you can afford one of the bigger ones go for it.

xytop• 10 years ago

Rackspace!!1 ha ha ha ha ha.. (neurotic laugh..) They took our whole infrastructure down for a day just because their highly paid Linux support team changed all root passwords and screwed firewalls and added unneed balancers so that servers weren't able to communicate with each other.. We didn't even request anything (as I remember).. Just bought their support and this shit happened. That was a nightmare

vidyesh8• 10 years ago

Oh.ho. When did this happen? I seem to have no recollection of reading about this.

vellis• 10 years ago

Typical PHP user, thinking that cloud is like their shared HostGator box. Pathetic.

halis• 10 years ago

Lucy!? You got some splainin' to do...

ljoshua• 10 years ago

I hadn't heard of the backup providers mentioned in the article. Does anyone have experience with those or with other recommended solutions?

I can re-implement the infrastructure of my VMs fairly easily (thank you Ansible), but backing up content outside of the provider's built-in options is something I haven't played with yet, and would obviously be the crucial piece.

lobster_johnson• 10 years ago

We use Backupninja to drive backups, mostly Duplicity against S3. Some things, like Postgres dumps, are custom commands that just upload individual files to S3. The reason you want to wrap things in Backupninja is to centralize logging, scheduling and monitoring.

Note that we only back up specific things like databases, logs (central syslog server) and data directories. We use Puppet to configure our boxes, and consider everything to be expendable that isn't data.

csomar• 10 years ago

Two things to learn from this:

1. Never have a single point of failure. Relying on DO for Server+Backups is putting your eggs on one basket.

2. Your server state should be programmable. This is not quite easy for complex configurations. But today, DO has an API, we have Docker, and quite modern deployment tools.

Here is setup:

1. Github for the server state. Basically, a repository to configure and deploy my infrastructure.

2. Enable DO backups in case I mess up something and want a quick come-back.

3. File backups through Tarsnap. Since I use Docker, I have a volume container. Backup the volume container with Tarsnap.

peterwwillis• 10 years ago

There's always a single point of failure. Most commonly it's a human.

tehbmar• 10 years ago

Welp, I'm verifying back ups on my server now.

ryanmarr• 10 years ago

The comment thread on this post makes me very happy. Having recently switched all my production instances to individual docker containers, I'm very happy to hear that the consensus seems to be that server instances should be killed and spun back up like bacteria. There's hope for humanity yet.

krinchan• 10 years ago

I use DO as a test bed for a lot of stuff. I ended up destroying/recreating so many droplets I just started using Chef. Also, I come from a major background in AWS, ChaosMonkey in prod, and a very strict push-button + nuke & pave deployment strategy.

So, all this strikes me as "lol you didn't know this?"

20centuryboys• 10 years ago

How exactly do you "ssh" into your DO droplet?

waxjar• 10 years ago

A droplet is just a VPS.

bikamonki• 10 years ago

This is yet another reason to push forward the idea of static websites/webapps. There is no single argument to support the idea of using a database to power a 5-6 page website. Not even a blog with daily posts. How many sites out there have less than 10 pages, are updated maybe once per month, sport a contact form or newsletter subscription box? 90%? And how many of them use an insecure behemonth like WP or Joomla?

My new model: client wants a pretty responsive theme, I get the HTML version of it. Turn into a template I can use to spit out a static site (from my homebrewed CMS). I tell clients your monthly support charges are $0. If you ever need updates I charge hourly (most never call in months). Some clients want "a list of something" say products. If small and won't grow: a static csv file that also gets munched by the homebrewed CMS. Large list? A BAAS service with API.

I really don't care if the servers are gone/hacked/ransomware. Git clone, go.

vidyesh8• 10 years ago

Those 'behemoth like' CMSes have spur out a new range of amateur web-developing agencies who spit out a design, theme and finish the complete project with less cost, less maintenance and fast delivery.

The client (who doesn't want to contact the agency again)is given instructions how to update which makes them happy. And then such projects spread like wildfire.

/rant over.

the_watcher• 10 years ago

Holy hell, obviously where critical data is concerned, redundancy is something you should insist on, but this definitely sucks. Seems ridiculous that all you got from DigitalOcean is a $15 credit though. I'd have expected something like 6 months of paid backups comped.

andrenotgiant• 10 years ago

$15 credit gets you 15 months of paid backups comped on a $5/month droplet, or 7.5 months comped on a $10 droplet.

vidyesh8• 10 years ago

Well if the droplet lost was $5 then as promised 3 months worth of it sounds fine.

nikdow• 10 years ago

Tell me if I'm doing something wrong: my two EC2 servers both use EBS including for the root device, so I'm not using any device-attached storage. If Amazon "lost" my instance, it would appear to me to be just a reboot.

And yes, I do a nightly backup of databases, webroot, /etc and some other directories.

ju-st• 10 years ago

Do you have two separate AWS accounts? Is your backup outside of amazons influence?

tomphoolery• 10 years ago

That really stinks, but this should be a warning to you that in the future, you should have some kind of redundancy plan so data does not get lost. Even if Amazon decided to terminate all of my app instances right now, I would just need to rebuild them. No data is actually lost, and furthermore backups are made frequently so that if a server decides to explode one day, we're still covered. DigitalOcean droplets are really not meant to be holding your database and application server...you shouldn't have experienced "data loss" by losing a droplet because your database should not have been hosted on a droplet.

tzakrajs• 10 years ago

Which is the irony of the entire post because the tldr was: "our DO instance was lost, fortunately we keep multiple backups of the data so we were able to restore without any data loss". It seems more like a passive aggressive note against Digital Ocean.

ricardobeat• 10 years ago

Why not? How is a droplet different than any other server, including your own hardware? All of them will fail eventually, the important bit is having a sound replication/backup strategy as you said yourself.

beachstartup• 10 years ago

you should hear what people say when we suggest that they make their own backups in addition to the ones we provide for them.

adults acting like children. nothing more, nothing less.

take responsibility for your own data. back it up yourself.

jvoorhis• 10 years ago

Sounds like a Tuesday at AWS.

mizzao3• 10 years ago

I'm more worried about the following post in that thread:

<begin quote> --------------------------------------------------------

We were loyal customers of DigitalOcean for over 2 years. We showed up to work one morning and had a client email stating that their website was down. We checked, sure enough. We tried logging in to our DI account and it said it was suspended. After searching around, we realized our CC had expired. No biggie, we'll just update it, turn the server back on and be on our way.

Nope. We had to contact customer service to get back in to our account. After updating the card info we realize that all our droplets are gone. We reach out to customer service again in which case they let us know that when a CC expires, an automated process kicks off and deletes the droplets. We've been a customer for 2 years, surely they could pick up the phone and call us. We've spent thousands of dollars with them.

Plan B, let's restore them from the backups we've been paying for. Nope! When they delete your droplets, they also delete your backups.

This is where we ask to speak to them on the phone and are denied. I then ask if they are insane and why they would delete someone's servers and backups via an automated process without a human at least checking to see if it is a loyal multi year customer who has a simple lapse with their CC expiry.

We had to find a backup on a developers machine from over a month prior and rebuild data using various megtods that took close to a week. In the end, DI gave us a $500 credit.

You get what you pay for. Anyone who uses DI for production or anything more than a hobby app is playing with fire. They do not care, they are apathetic, they will screw you over and the throw you a credit for their terrible service as a half assed apology.

mizzao3• 10 years ago

One time my CC expired on DO because there was a fraudulent purchase on it and I had to get a new one. I got a brief e-mail from DO noting there was a problem charging my card, and updated it the same day.

What if I was on vacation, and didn't see this e-mail for a few days? shudder

In what world is automating the deletion of customer data a good idea?

dualboot• 10 years ago

In what world is automating the deletion of customer data a good idea?

The data destruction world :)

xyzzy4• 10 years ago

The world where employees care more about management satisfaction than customer satisfaction? And it's difficult to prove that bothering some customers here and there is impacting corporate revenue. Policies like this slip through the cracks.

the_common_man• 10 years ago

I had the exact same experience! Except I got no credit.

My CC expired but I was on vacation and didn't have access to email. When I was back, the droplet and all backups and snapshots were completely removed. For the record, I was only 15 days late. Very sad. I tried my best to get the backups from the support engineer but no luck.

vidyesh8• 10 years ago

I really don't know to admire or criticise DO for this.

On one hand horrible situation to be in, at least could have called/mailed/SMSes/notified the customer before nuking the account.

On the other hand, it is a relief to know your data is secure and gone one you stop paying, no way they have a copy or keep it in archive/deep backup.

happywolf2• 10 years ago

Nope, the only assurance you can get is you won't be access your data, I am not sure how one can come down with conclusion of "no way they have a copy or keep it in archive/deep backup"

Having a backup and letting you to access it are two entirely different issues, and most of the time customer service staff don't know how the data management is being done by the engineering/operations team.

joenathan• 10 years ago

I'm looking for a good host, was considering DO, but now I'm not. Any suggestions?

joshmn2• 10 years ago

Needs vary, but here's a brief list of competitors:

* OVH.com — huge, great price point and performance

* Linode.com — also huge, favorite of many developers

* Vultr.com — owned by Choopa (bigger player)

* RamNode.com — doesn't scream "super modern", but reviews are great

There are many others, but those are off the top of my head that correlate with DO.

toadi2• 10 years ago

According to me DO is still a good hosting company. Works good for the price you pay. Taking offsite backups is always a good idea.

rorykoehler• 10 years ago

For what it's worth this happened to me through a client's account (client wasn't checking emails and didn't see warning mails re CC expiring). I reached out to customer support once the site went down and even though we were beyond the cut-off deadline they re-activated all the droplets. Perhaps they have updated their process in this regard. I also learned my lesson and made sure I got forwarding on all mails related to DO.

mrweasel3• 10 years ago

We've been a customer for 2 years, surely they could pick up the phone and call us.

I don't understand that reasoning from customers to be honest. That right there, throws away the $5 and $10 plans. Having some one trying to contact a customer, just because their credit card expired, cost more than the actual hosting.

Top tier service does comes from companies with $5 a month plans. I'm sorry it just doesn't.

josephagoss2• 10 years ago

We've spent thousands of dollars with them.

That's a decent amount of money right? It sounds like they were not using the $5 plans.

aagha• 10 years ago

Because $5/$10/month is not the same thing as Customer Lifetime Value.

techstrategist• 10 years ago

This is a big danger of "move fast and break things". I imagine the process they put in place seemed reasonable internally before catching this edge case, but what a crappy result.

josephagoss2• 10 years ago

Thank you for this, I have a large client where we were considering DigitalOcean and the yearly spend could easily be in the thousands. Now I'll strike them off of the list. Automated deletion of VM's and backups, absolutely unacceptable.

The VM hosting market isn't looking good at the moment, seems like each provider has some dark history at this point, it's really frustrating.

toadi2• 10 years ago

Offsite backups? We use AWS and DO to host applications for our clients. We still take offsite backups. Isn't that what your clients paying you for?

josephmx• 10 years ago

I had the opposite experience, my CC expired with DO, but I didn't have a new one for about 2 weeks. I didn't have any issues with just waiting to the next billing cycle and paying double then to cover the negative.

fweespeech4• 10 years ago

This is why you use two providers with two credit cards.

I'm quite serious. Our production environment operates on two cards with different issue & expiration dates from two different merchants and uses 2 different Cloud providers.

andrewthetechie4• 10 years ago

Check out the followup:

<begin quote> Hey Matthew,

I'm Zach, Director of Support here at DigitalOcean. Thanks for raising this topic.

I was able to locate your account, and I see that I was the one who granted the credit and followed up via email. I hadn't heard back from you until now, and I'm happy that we have an open line of communication. I'm hopeful that this thread starts a conversation, that it clarifies what steps we take, and we might even uncover a different solution that works well for everyone.

To start with, I want to be really clear, this is the absolute worst case scenario. I absolutely don't want it to happen to any customer, let alone someone who has been with us for such a long time. There's really a delicate balance that we must strike between customers who forget to pay and customers who do not want to pay.

Currently, our notifications for overdue balances are sent via email. If a customer account is unpaid on the first of the month, we send ~15 emails total notifying the customer of the situation, and subsequently power off servers 21 days after the account is on hold as a further way to gain your attention. At this time, droplets are removed from your account 14 days following power off, which is an increased amount of time from what you experienced. It was increased from 3 days based on past user feedback.

Why do we do it this way? In the past, we had no hold or suspension process for non-payment, which enabled bad actors to run for months and months without paying. From a business perspective, we made a decision to put a scalable process in place that limited how long an account could go unpaid.

As a support team and business, we are always willing to work with our customers who are unable to pay. We are always available via ticket, our contact form, and have made wide-ranging attempts to help users who aren't able to pay due to banking regulations: https://insidedigitalocean.com...

I'd like some input, so I'll publicly ask for feedback on questions that I've asked privately before. I'd also like any other feedback that we can consider on how to make this a more positive experience for everyone

-Knowing that we do not do phone support, what's the best way to notify you of a past-due balance? -Is SMS effective at times like this?

I would love to hear your thoughts.

Thank you, Zach zach@digitalocean.com

mizzao3• 10 years ago

3 days seemed like an accident waiting to happen.

This updated policy seems a lot better; it would probably be great to ask people for secondary contact information such as SMS for emergencies.

rubberstamp• 10 years ago

Whats the point of raid if raid card failure resulted in complete loss of data? I came across an article a week or two ago that discussed various alternatives, I think ZFS with checksums, which unlike raid will not replicate corrupt data from drive nearing failure to the healthy drive.

scurvy3• 10 years ago

They should have been able to swap out the RAID card and import the configs from the disks. This is pretty common practice with LSI/Avago-based cards. Curious as to why they didn't do that. Maybe the card went really bad and started writing garbage all over the place?

arihant• 10 years ago

I do understand the part where they terminated and killed of the VMs. I do not, at all, understand why they deleted the backups. How can the CEO sit in his chair and make that decision? If he didn't and he doesn't know, why is he still in that chair?

Even damn Netflix preserves data for 10 months. Deleting backups is making 100% sure that the customer doesn't come back to you. It's amazingly stupid.

Unless you're as smart and as big as Google, use the damn phone. Don't use the word automated at all. It's not automated if one fool made a decision and another one wrote the script.

charliedevolve• 10 years ago

If you're paying for backups and they get deleted the moment there's an issue with payment, ur doin' it rong. I don't care whatever the TOS says, they're being completely shitty to customers. Sure there should be regular off-site backups. No one is arguing against that. But the fastest restore is usually from the closest source, and that's partly why you'd pay them for it.

Lanari• 10 years ago

I recommend having a backup using Git on some reliable Git hosting(s). I can't think on anything safer and more practical.

vidyesh8• 10 years ago

To my surprise, this HN thread has no link to any external backup solution guide or little-to-no suggestion of best way to backup your server to an external service or another VPS/backup server.

aprdm• 10 years ago

I use Tarsnap and really enjoy it!

tvvocold• 10 years ago

Lucky to u, they just delete my server just for i set up a VPN server...

scurvy3• 10 years ago

Anyone have any experiences with running VM's with ceph storage? I've used it for other things, but I know that VM hosting is quite popular. Care to share any stories?

kull• 10 years ago

We are also experiencing a huge problem with performance of DO servers as we scale. Especially databases. With stories like that , a decision of moving to AWS seems pretty obvious.

DigitalOcean lost our server

Related Stories