Why the “Digital Ocean killed my company” incident scares the hell out of me

https://blog.checklyhq.com/why-the-recent-digital-ocean-killed-my-company-incident-scares-the-hell-out-of-me/

by tnolet • 7 years ago

316 180 7 years ago

blog.checklyhq.com

Do have some backups outside of your primary cloud provider. You'll sleep better.

This is really the takeaway from this whole incident. Even if the account doesn’t get banned you’re just one accidental action away from your database and backups disappearing simultaneously otherwise in most cases. Or if you are using a blob store there may not be any backup if the original is deleted (it’s like RAID not backups).

cm21873• 7 years ago

And have a separate DNS provider.

DenisM2• 7 years ago

Does that really help? If your DNS provider screws it up, you're in deep trouble regardless of whether they were the same guys hosting your VMs or not.

crazygringo• 7 years ago

No reason to.

If your cloud provider does your DNS and you switch to a different cloud provider...

...then just set your registrar to point to the new DNS provider. You've still got total control.

tinus_hn• 7 years ago

Absolutely, your domain name might be your most important asset. It’s silly to put that in the hands of a party that you might have to fight. Especially as there are hundreds of alternatives.

chrismeller• 7 years ago

The initial account lock and resource power down resulted from an automated service that monitors for cryptocurrency mining activity (Droplet CPU loads and Droplet create behaviors).

So despite the fact that you can get the same or more resources at other providers who advertise their shared CPU resources for less than half that charged by DO I’m still buying a shared resource?

I think this is something that needs to be much more front and center. With AWS I can spin up as many “boxes” as I want on a moment’s notice. As far as anything on DO’s site seems to advertise I can do the same.

Turns out that’s not so. If they wanted to shut down an account because they suspected it was compromised then that’s one thing. Shutting it down simply because of suspected cryptocurrency activity... not so much. If I’m willing to pay for it then that’s what I’m willing to pay for and there should be no limitations.

Now that becomes a little more clear as you read...

determine if automated action is warranted to minimize the impact of potential fraudulent high-cpu-loads on other customers.

So clearly we are not talking the same resources that were expected.

EdwardDiego3• 7 years ago

So despite the fact that you can get the same or more resources at other providers who advertise their shared CPU resources for less than half that charged by DO I’m still buying a shared resource?

The account that got locked down triggered automatic checks that used payment history as a "these people are okay" check. They didn't have a payment history, they were running solely on credits, so got flagged. IOW, they hadn't paid for anything yet.

With AWS I can spin up as many “boxes” as I want on a moment’s notice

You can, but they can also shut you down for "abuse". https://aws.amazon.com/premiumsupport/knowledge-center/aws-a...

scarface743• 7 years ago

You have 24 hours to respond and why would you have a business and not be on at least their business support plan?

DenisM2• 7 years ago

Any stories of being screwed by AWS in the similar manner?

That is, excluding the stories about the same account being used to sell thing on Amazon.coma and run a AWS setup. We know those are risky.

EdwardDiego3• 7 years ago

Why isn't everyone on bare metal?

Is this how we're referring to running on servers we have contractual control over these days? Part of managing risk is being sure your infrastructure will stay there, and if you don't have a contract, your risk is significantly higher.

cdumler2• 7 years ago

Pretty much the standard "the cloud is just someone else's computer" issue: if you don't have the ability to reproduce your work somewhere else and your provider decides to go away, you're going to have a Bad Time(TM).

Also, for as cool as Digital Ocean is, their primary focus is on low-use, shared cloud resources. From my experience, they over subscribe CPU resources so "noisy neighbors" was a problem sometimes. They do not provide or work with people as if they have production services, and they don't seem to like people who really want to use their system. I would never use them for production unless the work was ephemeral.

BinaryArcher• 7 years ago

My account was banned two years ago just for accessing it from abroad. This is real, don't use DO.

They finally gave me temporary access even though they didn't believe who I was. This was the icing on the cake, they gave me access to a cluster, they could've given it to anyone.

It's cheap but seriously not worth it. They also have many more outages than other larger providers.

codingdave• 7 years ago

You don't have to actually have redundant providers and monthly-tested DR runs, etc. But you do need a plan. No matter how small.

If you are just a side project running locally, than plan me be as simple as "If my laptop falls in a river, get a new one and clone the repo again." As you start having production systems, it may still be "clone the repo again", but this time to actual servers.

As you start having customers rely on you, your plan should get more robust. Maybe you don't have a hot DR site set up where you can just flip a switch. But you should know who your backup host would be, and have an account ready with them. You should what steps would be needed to go from repo -> new provider.

None of this needs to be set up and tested ahead of time if you are just getting started. But if you have paying customers, you have to have thought about it. DR starts tiny and scales up, just like everything else.

freehunter10• 7 years ago

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch.

But it wasn't, because every night I run a PG backup and copy it to AWS S3. I just had to download the backup from the other cloud vendor and restore it on my DO server.

Did DO fuck up? Yes. Did it cost me downtime? Yes. Was I mad? Yes, and I still am. But I still do business with DO because it costs ~half the price of a comparable EC2 instance, and writing a 10 line bash script to move my database backups to another cloud vendor isn't that hard. Storing that backup on S3 costs pennies per month.

I don't see "I've never run a business before" as a valid excuse, nor do I see "the cloud vendor is better equipped to handle backups" as a valid excuse. When it comes down to it, you are solely responsible to your customers. They're not going to care whose fault it was, because it was your fault.

Don't trust any of your vendors.

dmix3• 7 years ago

The article addresses this in the "Why not just do x?" section.

- Why were you only hosting one JUST ONE cloud provider?
- Why didn't you have backups outside of JUST ONE cloud provider?

Telling them to just use two cloud hosting providers sounds easy on paper but when you're a cash-strapped startup it's a significant ask. Especially if you need to have the same architecture replicated across both, in order not to have extended downtime.

This is much bigger issue than having external backups (which every startup should still do).

zamu• 7 years ago

Then save it locally as well? There really isn't an excuse to having DO or any provider being your sole backup.

pbourke• 7 years ago

But you don’t need the same architecture replicated across two providers, as your parent post demonstrated. Run Postgres in DO and upload your backups to S3/Azure Storage. This is both cheap and simple to the level that any startup could manage it. If and when it makes sense, you can investigate multi provider replication, but it might just be YAGNI.

EugeneOZ3• 7 years ago

No, it's not so difficult task and not even expensive. I don't think every project really needs it, though. But external db backups - they are ridiculously cheap and every project needs them.

LinuxBender• 7 years ago

These are strange times I guess? Maybe I'm a weird one-off, but I have 3 VPS providers for my hobby sites, my own DNS spread across all three (one remains shadow backup) and offline rsnapshot backups that automation can't touch.

I can't imagine doing less for a business. In my opinion, the business model and investment plan should account for all of this for at least the first five years.

jotm3• 7 years ago

Local backups? A few hard drives and a fiber connection can be cheap. Slow to restore, but not "killed my business" slow

jwr2• 7 years ago

But I still do business with DO because it costs ~half the price of a comparable EC2 instance

I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting.

For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better reliability, rent three.

I looked at this many times and still check the pricing regularly, and it just doesn't make sense for me to switch from these physical servers to virtualized offerings, not in any foreseeable future.

geggam• 7 years ago

Because I can simply turn off everything and reduce my spend to storage alone. If I turn off that server I still pay for it

barkingcat• 7 years ago

Because DO has their $5 and $10 / month plan, and that's really enough for me.

Someone1234• 7 years ago

Because hypothetically your instance should be hardware agnostic. If the physical hardware dies, it should automatically migrate to another physical server at their data center without your intervention. It will only look like an unexpected restart from your perspective.

That's something worth paying for. It would be more comparable to two servers at Hetzner with rapid failover. But even that is more involved since you have to set up the logic of when to failover.

Auto-scaling is a benefit of "cloud" services. But it isn't the only selling point. Hardware abstraction is perhaps bigger.

EugeneOZ3• 7 years ago

Maybe because they had dozens of failures and DDOS outages? Why they have to keep their prices so low - because of reputation. Price is the only attractive thing in services like this.

vorticalbox• 7 years ago

If you use a lambda on aws you get 1 million requests for free each month and after that factions of a penny per invoke.

jimnotgym• 7 years ago

I'm moving Azure, AWS and other VPS workloads to them right now. Very impressive service!

hadlock2• 7 years ago

Most $10MM ARR startups are running on two medium T class aws instances behind a load balancer at $35/mo plus another small server for accessories (redis, message queue, etc) and then a sql backend. That's something like $200/mo for infrastructure, you never need to maintain the hardware and you have the option to autoscale at any time to support $100MM ARR.

Plus you get access to backups and all the other tools and products that are constantly being released. Not having to hire an IT guy at $1200/mo to maintain your physical server and tend to it's networking rules is a significant savings over a developer-managed aws/gcp instance. Plus all the cloud interfaces are standardized so you can just hire someone to come through and fix/upgrade your infrastructure.

Using a home-spun physical server means months if not years of undocumented tech debt for the next guy who comes along to have to maintain whatever kludges were installed long before he/she ever showed up. I just got done converting a bunch of physical servers over to the cloud and spent a year unwinding seven years of technical debt and now the dev/qa teams can actually spin up a new test environment in under 30 days (closer to 3 minutes).

rkangel• 7 years ago

In our case, because it was easy to get thousands of dollars of free credit in Google cloud (the same appeared to be true for AWS). This went up to tens of thousands for Series A.

We had zero serving costs and would have continued to do so for years.

It makes perfect sense for Google/Amazon to incentivise startups to design themselves for their cloud.

ksec3• 7 years ago

For 34€/month, Hetzner ...

Hetzner is only available in EU and specifically Germany. And my bet most of these people want it within US.

rando444• 7 years ago

I'm a bit late to the party, but Hetzner actually deleted all my VMs and backups. Their abuse team didn't like my name, despite me also having my name in my e-mail and my credit card info.

I got an e-mail that my account was flagged, but they didn't even give me a chance to tell them they had my name correct.

I responded to their email within 30 minutes, but everything had already been erased.

Luckily I had some stuff in github, but I will never use them again.

sonnyblarney• 7 years ago

Because AWS et. al. are considerably more than just instances?

There's a whole host of services that make dev and deployment at lot easier.

Though instance costs are more expensive ... that's not the issue, the issue is 'TCO' (total cost of ownership) not 'server cost'.

If your team can move at 2x the speed on AWS, well that's worth a lot.

Cloud services are very useful, so the article raises a very legit concern/paradox.

wink• 7 years ago

My negative record is "getting access to the server" until "hard disk died" is 12h. And no, this isn't 1 out of 1 server.

I'd never again run anything business-critical on Hetzner metal. I'm not even running my irc bouncer there, tbh. The cloud stuff I'm really happy though, higher uptimes than on most physical servers there.

Oh, and remind me again of the time where we regularly called them because of outages because our monitoring was better than theirs and the customer support people hadn't yet learned of the network outage...

Operyl2• 7 years ago

For me, it’s because I have some latency specific loads that I can’t really easily do overseas sometimes.

EugeneOZ3• 7 years ago

It's because price of your site downtime is insignificant. DO (and many other "bulk vps" services) is only suitable for projects like this. When 1 hour of downtime cost much more than year of hosting, you will be ready to pay for high availability and duplication, replication, backups verification.

codegeek• 7 years ago

Great point. I always run a separate backup through bash scripts that syncs with S3 bucket even though I also have the DO backups enabled as well. You always want 2 separate offsite backups for any real world production application.

chocolatebunny• 7 years ago

Cloud or not, you have to trust your vendor to deliver the service you paid them for. That's the whole reason for doing business with them.

freehunter10• 7 years ago

The reason for doing business with them is because they say they will deliver the service you paid them for. But if they don't, what then? You can switch to another vendor, but without a backup you're starting from scratch.

You should hope they deliver the service you've paid for, but you should never trust that they will. Always plan for failure. That's the entire point of this whole "DO killed my company" saga.

proyb2• 7 years ago

I trust UpCloud more than other VPS provider because the technical support (Regional) has been very approachable and you know "who" they are. Beside, they promise 100% uptime SLA when your business is critical.

https://upcloud.com/blog/

behringer• 7 years ago

Might I recommend prgmr.com instead? I use both DO and PRGMR (for resiliency).

noir_lord• 7 years ago

FWIW I do the same I have backups of everything.

All that said in close to a decade (actually might be a decade) I've still yet to have any issue with linode (tempting the computer gods here) or their backups.

I remember when DO was first starting out and they used to knock their own stuff offline updating router tables.

Ever since I have a jaundiced view of their capabilities.

wiredfool• 7 years ago

N=1

Linode backups start to fail when you have more then 3 million files on a drive.

When debugging this, they told me the reason, I did a snapshot that worked, and then they turned off backups because they were failing. They didn’t highlight that backups were now off.

Restores on the larger machines can take 5+ hours, and they will often report that the restore failed if you are restoring a drive that contains docker’s pipe files.

Ymmv, but I’m trying to get off linode.

jotm3• 7 years ago

Do you not have a mirror copy of everything somewhere else? Or is that not possible in your case?

freehunter10• 7 years ago

I only run one server, because like the dead company in question, I don't have the money or manpower to do a full high-availability solution. Just a database backup and my site's code in Gitlab, so DR failover is a matter of 'git clone' and 'pg_restore'.

It might take me a few hours to get back online and piece my docker containers back together but my customers can absorb a few hours of downtime as long as it's a very rare occasion. If they couldn't, I'd charge them enough to afford a real HA solution.

neovive• 7 years ago

If you don't mind me asking, what is the preferred method to backup to S3? Is it possible to scp or rsync a mysqldump to S3 or do you install aws tools on DO and run aws s3 cp as a scheduled job?

freehunter10• 7 years ago

I use the awscli tools and aws s3 cp. I don't think it's possible to scp or rsync directly to S3, at least last I checked.

https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-in...

slenk• 7 years ago

Not OP but I just use shell scripts. You could scp or rsync if you mounted s3 as a drive to your host, but that is not recommended as far as I know

ptype• 7 years ago

check out restic which does dedup backups to s3, very easy to use

anderspitman• 7 years ago

I recommend looking into restic and/or rclone

Alex3917• 7 years ago

But it wasn't, because every night I run a PG backup and copy it to AWS S3

Which is probably much easy on DO than AWS. For whatever reason AWS seems to go out of their way to make it as hard as possible to backup your RDS data to a separate AWS account.

PhasmaFelis2• 7 years ago

I don't see "I've never run a business before" as a valid excuse, nor do I see "the cloud vendor is better equipped to handle backups" as a valid excuse.

It's not unreasonable to be annoyed when a company you are paying specifically to do the things you're not familiar with, catastrophically fails to do them correctly.

freehunter10• 7 years ago

Agreed, but "I'm annoyed" doesn't help your customers. "Don't worry, I'm restoring from last night's backup" sounds a lot nicer.

Again, your customer doesn't care what your excuse is, just like the dead company's founder doesn't care what DigitalOcean's excuse was. "Our processes failed" translates directly into "I failed".

PhasmaFelis2• 7 years ago

I don't see "I've never run a business before" as a valid excuse, nor do I see "the cloud vendor is better equipped to handle backups" as a valid excuse.

If you're paying someone specifically to make backups for you, you should be able to trust that they've taken every reasonable measure to ensure that backups are actually being made and preserved.

freehunter10• 7 years ago

You would think so, but then again you might be wrong. Better to be safe than sorry, no? I expected DO to make the backups I was paying them for and they didn't. Luckily I was making my own backups at the same time. Turns out my backups worked and theirs didn't. If I had just trusted them, I'd be out of business just like the dead company in question.

I'm not sure what's so confusing about "don't trust your vendors" but I've had to make this exact same reply way too many times.

Don't trust your vendors!

xiaomai• 7 years ago

I remember getting the e-mail about their backups not having been working for months. I was shocked at how negligent that was. I run my own backups now.

(Also their backup offering is so much more cumbersome to use than borg backup that it wasn't a great loss to manage these myself).

Still amazed that they didn't notice no one's backups had been running for so long.

duxup4• 7 years ago

Digital Ocean's statement on the events:

https://blog.digitalocean.com/an-update-on-last-weeks-custom...

Lots of support related issues here.

As someone who worked in support for a long time it isn't surprising to see this play out and the whole "Support and Security Operations leadership will create new workflows to allow abuse-related events to leverage the 24/7 structure of Support." while probably a good action to take, is one of those things that you see in support time and again and so rarely do you see an appropriate response as much as a patch for "this one thing won't happen again".

Nobody cares to staff, fund, and support the support teams until something goes wrong, and then it is usually "new workflows" for support.

reading-at-work• 7 years ago

I recently tried to sign up for a DO account to deploy a proof of concept app for my startup, and my account was immediately locked within minutes of opening it. I hadn't even spun up an instance yet. That combined with this means I'll be sticking with AWS for now, I suppose.

mark-ruwt• 7 years ago

I'm a one-man startup and I couldn't disagree with this more.

If you want to "play" startup, feel free to put all your eggs in one basket. If you actually want to build something meaningful and long-lasting, and you have Fortune 500 customers, take a breath and invest time in an infrastructure that can't be destroyed by a single rogue algorithm.

After all of the horror stories we've heard over the years, how is that time investment (and ability to sleep well at night) not worth it?

luckylion4• 7 years ago

take a breath and invest time in an infrastructure that can't be destroyed by a single rogue algorithm

As in? You can have your own cage, but somebody will have a cable that connects to the hardware you own. If they pull that cable because a single rogue algorithm told them that you're most likely abusing their service, that's it.

scarface743• 7 years ago

I do not have all the details, so caveat lector, but apparently it was due to spinning up a dozen droplets and running a batch job.

If this is considered “abuse” Digital Ocean is not a platform you should be running a business on.

If you can’t afford to pay for reliable big boy infrastructure, you might need to rethink your business plan or how much you charge.

mabbo• 7 years ago

It feels like the need for multi-cloud designs and infrastructure are becoming more sensible. If costs are best with one provider, by all means put 95% of your traffic there... But keeping 5% running smoothly on a second cloud, with a simple means to rebalance between them would be a huge win. It would mean the difference between an annoyance (argh, one of my cloud providers doesn't work) and a disaster (my sole cloud provider just ended my company). It would also save you the next time AWS or Google's cloud have an outage.

Are there any good resources for deploying multi cloud architectures?

thruhiker• 7 years ago

For a business, especially a startup, the opportunity cost often makes multi-cloud infeasible. You'd rather spend that engineering time on delivering features or addressing tech debt. I'm of the opinion that multi-cloud doesn't make much sense if your monthly cloud spend is less than at least $500k.

Pick one of the strong public cloud providers such as AWS, Azure, or GCP and then be judicious about which PaaS offerings you use to avoid unnecessary vendor lock in unless it adds enough value to justify the lock in.

the_common_man• 7 years ago

Was there a post mortem by DO?

aquaticsunset• 7 years ago

This was linked above

https://blog.digitalocean.com/an-update-on-last-weeks-custom...

It doesn't explain any detail behind why there was radio silence until social media support stepped in, which is what I'm very curious about, but it does have a timeline of events and an apology.

terryf• 7 years ago

I am old enough to have lived in soviet times for a short while. One thing I remember from back then - in order to get things done, you had to know someone. Perhaps your aunt's friend worked in the politbureau or your mom's classmate was a friend of the director. Through connections like that you could get what you needed.

It was such a relief when times changed and regardless of who you were, you could start exchanging money for things and services.

This story and many others like it remind me of those times - it's back to who you know to get your account unlocked. I've never gotten a story on the front page of HN and probably have 5 twitter followers. What's my avenue of getting my data back?

This isn't just about DO either. Similar stories about google and other services are many to be found.

duxup4• 7 years ago

I'll toot my own theory and it is that nobody wants to pay for enough capable support staff, pay to keep the support staff at the ready often enough, pay support staff who want to stay in that role.

They want to automate that all away as much as possible.

The future is everyone who isn't somebody chatting about how they run their application on X... because they're mysteriously banned form Y and Z and the next guy talks about how he was banned on X and is on Z now... simply for that reason.

pfranz• 7 years ago

I've heard people observe how many support lines have transformed into these impeccably polite, but completely unempowered people--which is more frustrating than being on hold or getting stuck in an automated menu loop because their protocol deflects any anger away from that person (which isn't fair in any circumstance) and leaves you helpless.

ehnto• 7 years ago

I would support your theory. Support staff don't scale as well or as fast as the technology they're supporting does.

I have often wondered about that for Google. Sure, they have bucket loads of cash, but could they even feasibly stand up a large enough support staff to handle all their platforms? They have so many services, across dozens of languages, and serve hundreds of millions of customers in different timezones.

Also, why would they want to? They're saving untold amounts of money by pushing the problem onto the consumer, and if it works 99% of the time then it's probably good enough.

treis4• 7 years ago

I'll toot my own theory and it is that nobody wants to pay for enough capable support staff, pay to keep the support staff at the ready often enough, pay support staff who want to stay in that role.

The crazy thing is that it's really not that expensive. AWS, for example, offers 24x7 support w/ <1 hour response time for production issues starting at $100 a month. At worst it's 10% of your bill.

aianus• 7 years ago

nobody wants to pay for enough capable support staff, pay to keep the support staff at the ready often enough, pay support staff who want to stay in that role

It's not about money. I worked at a unicorn that paid very high wages for support staff AND allowed them to work remotely and asynchronously from anywhere in the world. If you lived in Southeast Asia or Eastern Europe you'd make more than a local doctor just answering emails.

We still simply couldn't hire enough halfway intelligent people fast enough to keep up with the user growth. For each support person we'd hire, there'd be 10,000 new customers joining the same week. "Automating that all away" was the only tractable way to respond to people at all in a reasonable time frame. Obviously the support quality was awful.

4ntonius8lock2• 7 years ago

Personally, I blame the decoupling of the dollar to the gold standard and the distribution of newly minted dollars from the federal reserve into the well connected via banks and corporations that are controlled by a small social segment that all attend the same schools.

Once money became a thing that doesn't cost anything but the changing of zeros, then growth becomes a question not of how to produce something to get money, and but how to get the both the connection and the pedigree needed to receive cheap dollars.

This has created money silos, where the US aristocracy will take hundreds of billions in loses to capture a market and then extract value in monopolistic ways.

You can see this with google, facebook, amazon, etc.

It wasn't always the best companies that won. It was the best companies that had access to the vast capitals pools created out of thin air and who could promise to operate at the monopolistic scales the monied classes were aiming for from the beginning. That is why Ivy leaguers (whether drop outs or not) were chosen as the princelings. They are people who have a lot committed into the system and wouldn't dare betray it: they can be counted on to take things to their logical extreme.

I think also pertinent is the locking out of the middle and lower clases from growth fases of company creation (incentive angel investors and delaying IPOs + legally enforced discrimination against investors based on social class) - Oh and the pooling of legally stolen funds (pensions) into 'safe' stocks. Not to mention the legalization of bribery which has further accelerated our current state of legislative capture.

</completely my theory>

bogomipz• 7 years ago

"This has created money silos, where the US aristocracy will take hundreds of billions in loses to capture a market and then extract value in monopolistic ways."

I was curious about the above sentence. Who is the aristocracy in the context of startups? Are you referring to the VCs? If so don't the VCs not care about the "class" of the founders of a startup as long as they think there's money to be made from their company?

awinder• 7 years ago

This isn't a DigitalOcean option (I don't think anyways, I've never seen it), but on AWS/GCloud/Azure you can absolutely exchange money in order to get faster support SLAs, or even AWS solutions experts hanging out in your slack channel.

Of course, people who don't have money are forced to deal with greater hardships, yet that's always been a footnote to "regardless of who you were, you could start exchanging money".

CM30• 7 years ago

Yeah, support by social media seems to be trend now; those who are loudest and have the largest online followings get support by shouting at companies on Twitter, everyone else is just basically told to pound sand.

Then again, it's not just support for services. Even abuse reports seem to be taken a lot more seriously it's some online 'influencer' at the receiving end rather than an average Joe.

zacharybk2• 7 years ago

Hey terryf - Zach here from DO. If you ever encounter an issue that you don't feel is getting the proper attention feel free to reach out. We constantly engage with developers on social, which is primarily Twitter, and we don't look at follower count to determine who to reply to. If for whatever reason you don't get the help you need, you can always email me directly (first name at).

Thanks, Zach

egorfine2• 7 years ago

Здравия желаю, тов. коллега! I had the very same thought: how big is Google (or DO or anything else) but still you had to know someone to be just safe from the the very service you use.

Sawamara• 7 years ago

There is a heck of a difference between "eh, it only costs $5-$10" and between "the survival of my project relies on the site/service being under my control. If it was the latter, you go in having a plan B and a contingency Plan c.

GCP, AWS, DO, Azure... they all bank on that sweet combination of huge clients and clients who do not really need them but due to opportunity costs, cant be bothered to find viable, self-reliant alternatives.

INTPenis• 7 years ago

Something tells me it scares DO too because I've been getting emails lately asking if I'm happy with their services.

Operyl2• 7 years ago

When I joined Digital Ocean, back when it was 2-3 months old, I hit a KVM bug in their stack with my network traffic that caused them to rate limit my droplet to 1Kbit/s because it kept taking the host machine down I guess(?), and then them ultimately terminating it without a ticket asking me to even investigate. That was enough to pretty much never consider using Digital Ocean for anything “production” ever again. They didn’t terminate my account, and apologized with credit, but wow.

Another thought: That was also back when support responded in a reasonable time frame, and they actually spoke on IRC and everything too.

fhood• 7 years ago

Arguably Raisup should have been better prepared for this eventuality. But, I suspect that many of the people here talking about how their system is infallible made it that way because they, had time and/or money to spare working on it, have a very portable system, or have had something like this happen before.

I feel like HN comments are often sanctimonious to the point of ignorance. A solo dev, scrambling to get his project to the point where there is even the tiniest chance it will succeed, is likely to have a complex network of hardcoded filepaths, hostnames, and other magic numbers, strings, and config files that would make it very difficult to make portable without significant time and effort.

Or maybe the guy was an idiot. But possibly entertain the thought that maybe his situation wasn't exactly comparable to your eh?

DKnoll• 7 years ago

Or maybe the guy was an idiot. But possibly entertain the thought that maybe his situation wasn't exactly comparable to your eh?

I have never encountered a business that does not require backups. Not having a DR site as early in the game as they were might be excusable, but even then suggesting they get one is sound advice. It's an additional expense but it has a cheaper implementation cost while your infrastructure is small. Having a good DR plan is a selling point as well, this is the first thing I ask SaaS providers when I am evaluating them.

What is a total failure is not having backups on another provider. Before anyone says it's just a 2-person startup... surely that makes it easier? I can't imagine their DBs are all that big. Even just a three line cronjob and a Backblaze B2 ($0.005/GB) account could have secured their business continuity in the event their DO account didn't come back up.

But also DO Spaces for our object storage with ~500k media and our database backups.

Based on that comment (from @w3nicolas on Twitter) they actually could have used the same script they used for DO Spaces to back up their DBs to S3 (both use the S3 protocol). The 500k image files should be relatively small and cheap to back up too, appears they are just logo thumbnails for the tracked companies (500k image files, copy says they track 450k companies).

If you are building infrastructure and you don't have IT Ops experience consult somebody who does because they will save you from yourself in scenarios like this. If you are a dev you could probably throw a rock and hit 5 friends with enough ops experience to tell you the importance of redundant backups.

I think this comment might read a bit more abrasively than I meant it, but I don't mean to kick the guy after he was down. If you are in a similar boat to RaisUp with your backups go fix it now. If you rely on any one vendor for business continuity that is a problem. This PSA is sponsored by salty sysadmins everywhere.

vardump• 7 years ago

I have never encountered a business that does not require backups.

Me neither. But unfortunately the most I've encountered don't either have truly full backups (missing stuff), actually working backup process (some issue with backup media, process, etc.) or any idea how to actually restore if something does happen.

Almost no one does drills to restore backups to some test system.

I can't talk about the other cases, but I can tell about one small startup I joined a long ago. The worst one. They did backups on a 3-disk RAID-5 server... with one disk broken and one with SMART warnings. Their backup process also failed to backup anything with too long path names, so in reality almost half of the data was actually missing! There was also some unicode issue that lost files with characters above 128 ASCII.

My first days went to just actually ensuring their data is backed up...

qes• 7 years ago

They've been in business for at least five years. They also decided to make a lot of noise about their Fortune 500 clients (multiple). If they're not well past "scrambling to succeed" then they failed on the business side as much as they did the technical side.

cdumler2• 7 years ago

I feel like HN comments are often sanctimonious to the point of ignorance. A solo dev, scrambling to get his project to the point where there is even the tiniest chance it will succeed, is likely to have a complex network of hardcoded filepaths, hostnames, and other magic numbers, strings, and config files that would make it very difficult to make portable without significant time and effort.

No, no, no. That's the "well, it's never happen before so why should I have planned for it?" attitude. If you are in a solo business it is the _most_ important time to pull out your SWOT chart be honest about threats and weaknesses. A solo company is literally the worse time to not consider weaknesses as you have the least amount of assets to cover calamity. His primary concern right now is to be able to reproduce his work on AWS or Azure. Period.

tnolet• 7 years ago

OP here. A dude in the devops Slack channel had a great comment on the multi-cloud thing. "People can't even get redundancy and failover working on one cloud, why bother with two."

yardstick2• 7 years ago

I’ve asked this in a previous thread but didn’t get much of an answer: what can a business do to proactively fend off issues with cloud providers shutting them down.

I’ve heard GCP (or was it AWS?) is less likely to shut you down if you bill on account vs using a credit card.

Are there any other things we can do to reduce problems?

Would asking for an account manager help? Would buying reserved instances help?

Is there a minimum spend that gets you more attention and priority? Eg 100/month vs 1k/month vs 10k/month etc?

Any other advice?

dathinab• 7 years ago

- Use billing methods which are less likely to be used by scammers (e.g. likely not credit card).

- Keep a steady balance.

- Have luck (sad but true).

Minimum spend and similar might help with customer service (if they can see your spend). But most problems start with automated services.

Don't switch cloud providers every half year because another one is slightly cheaper now (i.e. have longer running/existing accounts), might help.

But most important make sure that you don't get locked in by the could provider

Even a permanent ban from the could provider should just cause some, not to high, short term (mony) loss. This means:

- Do not ~use~ relay on cloud providers proprietary tec (sure use there management console, etc. but don't integrate it into your business to tightly).

- Have control over your DNS names so that you can map them to different IP addresses if needed.

- Make sure you can migrate to a different cloud provider in matters of hours/days/weeks (depending on what you use it for) at any time.

- Have backups outside of the could provider for all your stuff. - don't get trapped by doing backups on a different service of the same provider (e.g. Amazon) ;=)

In the end how much this costs you depend a lot on what you do and how you do it. And yes deciding and not doing any continous shot term investments taking the risk of loading everything is fine, you just should make the decision conscious.

yardstick2• 7 years ago

Some good advice, thanks

turrini2• 7 years ago

I've got a stable setup:

  ---------------------------------------
  Vultr ========================== Linode
  ---------------------------------------
  Los Angeles ==================== Dallas
  Mastercard 1234 ============= Visa 4567
  ---------------------------------------
  VMs with custom Linux (ROOT/ZFS Debian)
  Replicating snapshots every 5 minutes
  CloudFlare + HAProxy Load Balancing

Total cost: 6 VMS at $20 each: $120, CloudFlare $20

Edit: backups to B2 and Wasabi. ~$10 month

jotm3• 7 years ago

Add a local backup. Then again, Im biased :D

lohszvu• 7 years ago

Are you using a managed host for HAProxy? What if it goes down.

turrini2• 7 years ago

I have a failover setup that changes Cloudflare DNS when necessary.

__d• 7 years ago

Where's your DNS?

And good work on the credit cards -- too many people forget that point of failure.

siffland• 7 years ago

If you are running a business then always have a good disaster recovery strategy. Split services between cloud providers and make sure your backups are good and offsite. You can do this all with opensource software if you dont want to pay for anything too (and if you have the staff that can do it).

If your business can be "killed" by your provider losing your data you have set yourself up to fail. I am not saying the provider is not to blame, but the burden should also be on the customer for not protecting themselves, what if DO burnt down?

We have yearly DR exercises, we have to bring up about 40 of our 120 production instances, it is a difficult task, but we know we can do it.

Never depend on your provider to save you, always make sure you have planed this out.

exabrial• 7 years ago

I really liked their blog response. I logged a support case about the incident, and a rep personally called me and spent 30m on video chat with me answering my questions. I don't think that would _ever_ happen from GCP.

whitepoplar• 7 years ago

I want to love DO so much. I think they really do mean well and their products are great, but yeah, this kind of stuff scares me shitless for mission critical workloads.

docker_up• 7 years ago

The one thing that the author forgot or didn't realize to mention is:

Test moving your environment between service providers in production.

You should be able to failover from one provider to another, in a predictable way. You should actually do so every few months so that you're well versed in what needs to happen. If you practice this, you will be prepared in case the absolute worst happens.

TheSpiciestDev2• 7 years ago

All this Digital Ocean talk... I spun up some droplets to play with a few months ago and liked what I saw and then "deactivated" them or turned them off.

What I did not expect to see a few days ago was the bill for the stopped droplets. Not just a little for the dormant disks I have had but seemingly full-prices for the entire droplets, as if they were all still "active". Nonetheless I deleted all my droplets, discarding whatever I had setup a few months ago, out of pure frustration. I never plan to return.

AWS may take me a little more time to setup and their interface may not be as pretty, but they will at least bill me fairly.

freehunter10• 7 years ago

I just tried to turn off one of my droplets and had to click "okay" on a message that said "You will still be billed for Droplets that are turned off because your disk space, CPU, RAM, and IP address will still be reserved."

If you clicked okay without actually reading the message, that is entirely your own fault.

TheSpiciestDev2• 7 years ago

While I don't remember any such message, after a few months it's certainly felt wrong and I've felt the sting. It'll be something I take into consideration whenever I shop around again.

weaksauce2• 7 years ago

I made this comment in an older thread but haven't gotten an answer on... the circumstances seem off to me:

The thing that has me scratching my head is how this chain of events unfolded.

I get that your fraud algorithm flagged it because of lack of established payment. how is this possible if what the tweet referred to as "locking us out of all of our backups and work"? surely an account history of any significance would have an established payment record. From their tweets they mention that they had 5 droplets and some storage of a not insignificant number of records (~500k) and that a script is required to be run every 2-3 months to process some data and that script spins up 10 droplets during that time. seems like it will take 13 hours to process the data based on row count and per record time.[0] I am struggling to see how they didn't have payment history. can you elaborate?

In addition another thing I'd think would help assuage fears of a complete lockout is some process where you can request and download the db or a snapshot of the virtual machine.

[0] https://twitter.com/w3Nicolas/status/1134529322902007809

bcooks3• 7 years ago

Hi weaksauce... sorry I must have missed your earlier question.

The account had been live for some time and in that sense had history but because of credits it didn't have payment history. As some others have commented lots of startups use credits to get their business going and depending on your usage they can last you for quite a while. Payment history indicates a willingness and capability to make payments.

Part of the issue here was what triggers the algorithm used when looking at remaining credits, payment history (none), workload deltas (the new spin ups), and effective run rate (think of that as the amount of money they would be charged for the workload they were spinning up). The bug in this case was both simple and super impactful. Raisup did nothing wrong, everything right in fact. We just blew it.

Thanks for the comment on request for download of backups or snapshot. That is a great idea, I guess we just never expected to actually go shoot a real customer and the fraudsters don't ask for their data.

weaksauce2• 7 years ago

I appreciate the response. a followup question to that would be how they got enough credits to be running for that long without any payment?

cyberferret2• 7 years ago

What scares me the most these days is that a lot of vendors have a policy of "we're shutting you down, but we won't tell you why or how you can fix things so that it doesn't happen again".

I have had friends who have had their Twitter or Youtube accounts shut down with no explanation from the companies involved. Just recently, MailChimp shut down a brand new account for my SaaS (we had only 4 subscribers added, and hadn't sent out any email campaigns yet!) with a refusal to give us an explanation why. We set up another account and did everything exactly the same way (including tediously replicating lots of email templates manually) without the same thing happening. Why was the first account shut down? We will never know, it seems.

I am all for using AI to detect dark patterns and raising a flag, but I think these companies need to bring a more human aspect into the after effects. Surely a legitimate operator will always email in with a "WTF?" whereas a nefarious actor will simply automatically move on to the next thing?

bcooks3• 7 years ago

Hi cyberferret. Two quick trys to provide some clarity for you here:

1. On your vendors don't say why they took action point. I think a lot of vendors, us included, worry about the bad actors knowing the _how_ of our algorithms because they will then have intelligence on what to try and work around to avoid detection. It's a constant adversarial chess match we adjust, they adjust.

2. As for the human element being needed for the after effects I totally agree. That said, I looked at our numbers and while many of the nefarious actors simply move to the next thing, a very large percentage of the, "WTF" replies are actually from bad actors hoping that is enough to be re-enabled and to keep going for a while. Our goal is to bring more people into the after effects and give them better information to make decisions and work with customers to avoid this kind of mess in the future.

cyberferret2• 7 years ago

I appreciate you giving an insight into the reasons behind your decisions Barry - I've learned a bit here myself, and can relate to the quandary that you must have to tiptoe around, in between being open and communicative, versus keeping your cards close to your chest. Thanks, and best of luck.

ksec3• 7 years ago

I think Scaleway and UpCloud are two biggest contender to DO. ( Not sure why Linode is not getting traction, may be its security incidents )

freehunter10• 7 years ago

Before I moved to DO I used Linode. My day job is a security consultant, I switched from Linode because of their repeated security incidents.

tasssko• 7 years ago

Don’t use DigitalOcean for anything serious. My team and i frequently support DigitalOcean customers with issues like crashed applications failed builds and in many cases the challenge is how they (DigitalOcean) provide resource allocations for droplets. 1cpu != 1cpu which is vital for vm to perform at its best. AWS and GCP are better. i’d argue that most IaaS platforms are better than DIgitalOcean. You get what you pay for. i’m not a customer.

mobilemidget• 7 years ago

'before switching to AWS completely'

why rely on a single party, you are putting all your eggs in a single basket again.

egorfine2• 7 years ago

most appeals are true bad actors

is that really the case?

Why the “Digital Ocean killed my company” incident scares the hell out of me

Related Stories