Multiple Digital Ocean services down

https://status.digitalocean.com/incidents/lgt5xs2843rx

by inanothertime • 8 months ago

115 45 8 months ago

status.digitalocean.com

jsheard• 8 months ago

They don't name names but it's probably due to the ongoing Cloudflare explosion. I know the DigitalOcean Spaces CDN is just Cloudflare under the hood.

matt-p5• 8 months ago

Just spaces CDN, not spaces - you'd think they'd just turn the CDN off for a bit.

potato3732842• 8 months ago

You can't just "turn off CDN" on the modern internet. You'd instantly DDOS your customers' origins. They're not provisioned to handle it, and even if they were the size of the pipe going to them isn't. The modern internet is built around the expectation that everything is distributed via CDN. Some more "traditional" websites would probably be fine.

TechRemarker• 8 months ago

Yes all sites showing the CloudFlare error due to the massive outage. Seems their outages are getting more frequent and taking down the internet in new ways each time.

archerx• 8 months ago

I have two projects on DO using droplets and they are still running fine.

hshdhdhj4444• 8 months ago

It seems mostly a CludFlare related issue.

My DOs are working fine as well.

sgc• 8 months ago

Are you using their "reserved IPs"? I was thinking of starting to use them, but now I wonder if it is part of their load balancing stack under the hood.

soheilpro• 8 months ago

Droplets are fine.

This incident affects: API, App Platform (Global), Load Balancers (Global), and Spaces (Global).

giancarlostoro2• 8 months ago

So yesterday Azure got hit hard, today CF and DO are down, bad week or something else?

zx8080• 8 months ago

Year-end promotion cycle is the worst time for end-users and the best one for engineers greedy for promotions.

Lammy• 8 months ago

Don't blame individual engineers who want to do what will be rewarded instead of company performance policies that reward this type of behavior.

red-iron-pine3• 8 months ago

what engineers, mate? they AI now

and they're doing just spectacular

BubbleRings• 8 months ago

I would also like to know people’s opinion on this.

matt-p5• 8 months ago

DDOS, but I don't really understand why in particular.

giancarlostoro2• 8 months ago

Having known people like this, its either flexing about who has the more powerful botnet or advertising who can do what.

red-iron-pine3• 8 months ago

NATO testing internal infra, or Russian hackers stepping it up after aggressive sabotage efforts in Eastern Europe?

watermelon0• 8 months ago

Azure DDoS event happened in October. Blog post about the attack was published yesterday, and was quickly picked up by news sites.

showerst4• 8 months ago

I use DO's load balancers in a couple of projects, and they don't list Cloudflare as an upstream dependency anywhere that I've seen. It's so frustrating to think you're clear of a service then find out that you're actually in their blast radius too through no fault of your own.

wesammikhail• 8 months ago

slight off topic: I used DO LBs for a little while but found myself moving away from that toward a small droplet with haproxy or nginx setup. Worked much better for me personally!

showerst4• 8 months ago

The point of an LB for these projects is to get away from a single point of failure, and I find configuring HA and setting up the networking and everything to be a pain point.

These are all low-traffic projects so it's more cost effective to just throw on the smallest LB than spend the time setting it up myself.

miken123• 8 months ago

It is mentioned in their list of subprocessors: https://www.digitalocean.com/trust/subprocessors

bbss• 8 months ago

Regional LBs do not have Cloudflare as an upstream dependency.

coreylane• 8 months ago

I find stuff like this all the time, railway.com recently launched an object storage service, but it's simply a wrapper for wasabi buckets under the hood, and they don't mention this anywhere... not even the subprocessors page https://railway.com/legal/subprocessors - customers have no idea they are using wasabi storage buckets unless they dig around the dns records. so i have to do all this research to find upstream dependencies and go subscribe to status.wasabi.com alerts etc.

dig b1.eu-central-1.storage.railway.app +short

s3.eu-central-1.wasabisys.com.

eu-central-1.wasabisys.com.

timomeh• 8 months ago

Hey, I'm the person that was responsible for adding object storage to Railway. It was my onboarding project, basically a project I was able to choose myself and implemented in 3 weeks in my 3rd month after joining Railway.

Object Storage is currently in Priority Boarding, our beta program. We can and will definitely do better, document it and add it to the subprocessor list. I'm really sorry about the current lack of it. There was another important project that I had to do between the beta release of buckets and now. I'm oncall this week, but will continue to bring Buckets to GA next week. So, just to give this context. There's no intentional malevolence or shadiness going on, it's simply because there's 1 engineer (me) working on it, and there's a lot of stuff to prioritize and do.

It's also super important to get user feedback as early as possible. That's why it's a beta release right now, and the beta release is a bit "rushed". The earlier I can get user feedback, the better the GA version will be.

On the "simply a wrapper for wasabi buckets" - yes, we're currently using wasabi under the hood. I can't add physical Object Storage within 3 weeks to all our server locations :D But that's something we'll work towards. I wouldn't say it's "simply" a wrapper, because we're adding substantial value when you use Buckets on Railway: automatic bucket creation for new environments, variable references, credentials as automatic variables, included in your usage limits and alerts, and so on.

I'll do right by you, and by all users.

drob518• 8 months ago

Dominos falling into dominos falling into dominos…

aforty• 8 months ago

Cloudflare outage.

igtztorrero• 8 months ago

I knew it, DigitalOcean CDN is using Cloudflare behind the scenes. Why DO ?

mrkramer• 8 months ago

Who is next?

red-iron-pine3• 8 months ago

my guesses would be look at who has a FedRAMP capable service first.

maybe also GCP, hetzner, akamai

foxyv• 8 months ago

Man, it really seems like the cloud providers are having some tough times lately. Azure, AWS, and Cloudflare! Is everything just secretly AWS?

Multiple Digital Ocean services down

Related Stories