Cloudflare was down

https://www.cloudflare.com/

by mektrik • 8 months ago

832 530 8 months ago

cloudflare.com

kaliqt• 8 months ago

NPM is down as a result.

chokominto• 8 months ago

Craaazzzyy

mercurialsolo3• 8 months ago

As is supabase

Andugal2• 8 months ago

Notion is also down as a result

arunaugustine2• 8 months ago

Shopify is down.

Geep5• 8 months ago

Claude RIP

dinoqqq2• 8 months ago

LinkedIn, Perplexity as well

mercurialsolo3• 8 months ago

shopify.com

xyproto2• 8 months ago

Yes.

Weird that https://www.cloudflarestatus.com/ isn't reporting this properly. It should be full of red blinking lights.

csomar4• 8 months ago

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Dec 05, 2025 - 07:00 UTC

Something must have gone really wrong.

shafyy• 8 months ago

Life hack: Announce bug that brings your entire network down as scheduled maintenance.

headmelted3• 8 months ago

It's 1AM in San Francisco right now. I don't envy the person having to call Matthew Prince and wake him up for this one. And I feel really bad for the person that forgot a closing brace in whatever config file did this.

mikkom2• 8 months ago

Company internal status pages are always like this. When you don't report problems they don't exist!

chironjit• 8 months ago

Yeah, their status site reports nothing but then clicking on some of the links on that site bring you the 500 error

63stack• 8 months ago

This is just business as usual, status pages are 95% for show now. The data center would have to be under water for the status page to say "some users might be experiencing disruptions".

csomar4• 8 months ago

They just did an update, and it is bad (in the sense that they are not realizing their clients are down?)

Investigating - Cloudflare is investigating issues with Cloudflare Dashboard and related APIs.
These issues do not affect the serving of cached files via the Cloudflare CDN or other security features at the Cloudflare Edge.
Customers using the Dashboard / Cloudflare APIs are impacted as requests might fail and/or errors may be displayed.

yapyap2• 8 months ago

well it does say that now, so…

which datacenter got flooded?

tommek4077• 8 months ago

Yes, it’s really ‘weird’ that they refuse to share any details. Completely unlike AWS, for example. As if being open about issues with their own product wouldn’t be in their best interest. /s

darccio• 8 months ago

https://updog.ai/status/cloudflare reported the incident 13 minutes ago (at the moment of writing this).

jonathanlydall3• 8 months ago

Now showing a message, posted at 08:56 UTC.

fxd123• 8 months ago

Investigating - Cloudflare is investigating issues with Cloudflare Dashboard and related APIs.

They seem to now, a few min after your comment

redm2• 8 months ago

Im much more concerned with customer sites being down which indicates are not impacted. They are.. :/

javier24• 8 months ago

Yeah. I only work for a small company, but you can be certain we will not update the status page if only a small portion of customers are affected, and if we are fully down, rest assured there will be no available hands to keep the status page updated

s_dev• 8 months ago

rest assured there will be no available hands to keep the status page updated

That's not how status pages if implemented correctly work. The real reason status pages aren't updated is SLAs. If you agree on a contract to have 99.99% uptime your status page better reflect that or it invalidates many contracts. This is why AWS also lies about it's uptime and status page.

These services rarely experience outages according their own figures but rather 'degraded performance' or some other language that talks around the issue rather than acknowledging it.

It's like when buying a house you need an independent surveyor not the one offered by the developer/seller to check for problems with foundations or rotting timber.

lawnchair• 8 months ago

I have to say that if an incident becomes so overwhelming that nobody can spare even a moment to communicate with customers, that points to a deeper operational problem. A status page is not something you update only when things are calm. It is part of the response itself. It is how you keep users informed and maintain trust when everything else is going wrong.

If communication disappears entirely during an outage, the whole operation suffers. And if that is truly how a company handles incidents, then it is not a practice I would want to rely on. Good operations teams build processes that protect both the system and the people using it. Communication is one of those processes.

GoblinSlayer3• 8 months ago

You won't be able to update the status page due to failures anyway.

onion2k• 8 months ago

if we are fully down, rest assured there will be no available hands to keep the status page updated

There is no quicker way for customers to lose trust in your service than it to be down and for them to not know that you're aware and trying to fix it as quickly as possible. One of the things Cloudflare gets right is the frequent public updates when there's a problem.

You should give someone the responsibility for keeping everyone up to date during an incident. It's a good idea to give that task to someone quite junior - they're not much help during the crisis, and they learn a lot about both the tech and communication by managing it.

hinkley2• 8 months ago

They were intending to start a maintenance window starting 6 minutes ago, but they were already down by then.

dinoqqq2• 8 months ago

There is an update:

"Cloudflare Dashboard and Cloudflare API service issues"

Investigating - Cloudflare is investigating issues with Cloudflare Dashboard and related APIs.

Customers using the Dashboard / Cloudflare APIs are impacted as requests might fail and/or errors may be displayed. Dec 05, 2025 - 08:56 UTC

rollulus• 8 months ago

Not weird, that’s tradition by now.

jachee2• 8 months ago

Management is always going to take too long (in an engineer’s opinion) to manually throw the alerts on. They’re pressing people for quick fixes so they can claim their SLAs are intact.

rvz7• 8 months ago

The AI agents can't help out on this time.

rifycombine12• 8 months ago

maybe we can back to stackoverflow :)

Havoc• 8 months ago

It’s wild how non of the big corporations can make a functional status page

dncornholio• 8 months ago

They can. They don't want to though.

javier24• 8 months ago

They could, but accurate reporting is not good for their SLAs

jbuild• 8 months ago

Interesting, I get a 500 if I try to visit coinbase.com, but my WebSocket connections to advanced-trade-ws.coinbase.com are still live with no issues.

emakarov• 8 months ago

probably these websockets are not going through cloudflare

tjpnz• 8 months ago

They have enough data to at least automate yellow.

devmor• 8 months ago

Yes, the incident report claims this was limited to their client dashboard. It most certainly was not. I have the PagerDuty alerts to prove it...

headmelted3• 8 months ago

Claude offline too. 500 errors on the web and the mobile app has been knocked out.

lionkor3• 8 months ago

I had to switch to Gemini for it to help me form a thought so I could type this reply. Its dire.

mercurialsolo3• 8 months ago

claude code works tho

sammy22553• 8 months ago

500 internal server error on most things:

500 Internal Server Error cloudflare

pzs• 8 months ago

Just experienced this and came here to check, because even their website is down. The referenced link also returns with 500.

headmelted3• 8 months ago

"In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Dec 05, 2025 - 07:00 UTC"

No need. Yikes.

imperfectfourth• 8 months ago

downdetector is also down

maxlin4• 8 months ago

it being the first google result and serving the exact same error as the pages one is trying to get info from is too funny

AmateurAlert• 8 months ago

https://downdetector.com/ classic

26d0• 8 months ago

hmm... https://downdetectorsdowndetector.com/

(edit: it's working now (detecting downdetector's down))

aurareturn2• 8 months ago

Ehh, so down detector for down detector is up but it is inaccurate.

xyproto2• 8 months ago

A wrong downdetectordowntector is worse than a 500 one. :D

deveesh_shetty• 8 months ago

You had one job.

Andugal2• 8 months ago

So DownDetector is down, but DownDetectorDownDetector does not detect it... We probably need one more DownDetector. (no)

manyaoman• 8 months ago

So down²detector was fake all along?

vanyauhalin• 8 months ago

So,

This one is green: https://downdetectorsdowndetector.com

This one is not openning: https://downdetectorsdowndetectorsdowndetector.com

This one is red: https://downdetectorsdowndetectorsdowndetectorsdowndetector....

aroman2• 8 months ago

great news, schrodingersdetector.com is available!

maxlin4• 8 months ago

half the internet is down downdetector is down downdetector down detector reports everything is fine

software was a mistake

O4epegb• 8 months ago

This is a fake detector that just has frontend logic for mocking realistic data, you can easily see it in the source code.

xx_ns• 8 months ago

At least it's still right in spite of being down.

strangeness• 8 months ago

Who knows, maybe it will be because of C or C++ this time. Or something else.

etyhhgfff• 8 months ago

They rewrote some of their core components from nginx+LuaJit to Rust for better perf and lateny recently. I guess there are some bugs in the new codebase.

piker• 8 months ago

At least the 500 error announces ownership.

Imagine how productive we'll be now!

rvz7• 8 months ago

Round 2 of Cloudflare outages.

We can now see which companies have failed in their performative systems design interviews.

Looking forward to the post-mortem.

sharts• 8 months ago

HaHa -Nelson

csomar4• 8 months ago

Interestingly, my site running on workers https://codeinput.com is still functioning. Worth mentioning that I don't use Cloudflare firewall/caching (directly exposed workers)

c16• 8 months ago

CloudFlare: You can't go down if you're never up.

pm90• 8 months ago

This is not good. One major outage? Something exceptional. Several outages in a short time? As someone thats worked in operations, I have empathy; there are so many “temp havks” that are put in place for incidents. but the rest of the world won’t… they’re gonna suffer a massive reputation loss if this goes on as long as the last one.

karmakurtisaani2• 8 months ago

Probably fired a lot of their best people in the past few years and replaced it with AI. They have a de-facto monopoly, so we'll just accept it and wait patiently until they fix the problem. You know, business as usual in the grift economy.

5d41402abc4b• 8 months ago

They have a de-facto monopoly

On what? There are lots of CDN providers out there.

rvz7• 8 months ago

The "AI agents" are on holiday when an outage like this happens.

mvdtnz• 8 months ago

This didn't happen at all. You're just completely making shit up.

PlotCitizen• 8 months ago

This is a good reminder for everyone to reconsider making all of their websites depend on a single centralized point of failure. There are many alternatives to the different services which Cloudflare offers.

koakuma-chan7• 8 months ago

My Cloudflare Pages website works fine.

coffeebeqn2• 8 months ago

We just love to merge the internet into single points of failure

berkes2• 8 months ago

But the nature of a CDN and most other products CF offers, is central by nature.

If you switch from CF to the next CF competitor, you've not improved this dependency.

The alternative here, is complex or even non-existing. Complex would be some system that allows you to hotswap a CDN, or to have fallback DDOS protection services, or to build you own in-house. Which, IMO, is the worst to do if your business is elsewhere. If you sell, say, petfood online, the dependency-risk that comes with a vendor like CF, quite certainly is less than the investment needed- and risk associted with- building a DDOS protection or CDN on your own; all investment that's not directed to selling more pet-food or get higher margins at doing so.

inferiorhuman2• 8 months ago

  There are many alternatives

Of varying quality depending on the service. Most of the anti-bot/catpcha crap seems to be equivalently obnoxious, but the handful of sites that use PerimeterX… I've basically sworn off DigiKey as a vendor since I keep getting their bullshit "press and hold" nonsense even while logged in.

I don't like that we're trending towards a centralized internet, but that's where we are.

rvz7• 8 months ago

We are now seeing which companies do not consider the third party risk of single point of failures in systems they do not control as part of their infrastructure and what their contingency plan is.

It turns out so far, there isn't one. Other than contacting the CEO of Cloudflare rather than switching on a temporary mitigation measure to ensure minimal downtime.

Therefore, many engineers at affected companies would have failed their own systems design interviews.

cryptonym10• 8 months ago

Sometimes it's not worth it. Your plan is just to accept you'll be off for a day or two, while you switch to a competitor.

throwaway42346• 8 months ago

Alternative infrastructure costs money, and it's hard to get approval from leadership in many cases. I think many know what the ideal solution looks like, but anything linked to budgets is often out of the engineer's hands.

In some cases it is also a valid business decision. If you have 2 hour down time every 5 years, it may not have a significant revenue impact. Most customers think it's too much bother to switch to a competitor anyway, and even if it were simple the competition might not be better. Nobody gets fired for buying IBM

The decision was probably made by someone else who moved on to a different company, so they can blame that person. It's only when down time significantly impacts your future ARR (and bonus) that leadership cares (assuming that someone can even prove that they actually lose customers).

formerly_proven• 8 months ago

On the other thread there were comments claiming it’s unknowable what IaaS some SaaS is using, but SaaS vendors need to disclose these things one way or another, e.g. DPAs. Here is for example renders list of subprocessors: https://render.com/security

It’s actually fairly easy to know which 3rd party services a SaaS depends on and map these risks. It’s normal due diligence for most companies to do so before contracting a SaaS.

berkes2• 8 months ago

At least this warrants a good review of anyone's dependency on cloudflare.

If it turns out that this was really just random bad luck, it shouldn't affect their reputation (if humans were rational, that is...)

But if it is what many people seem to imply, that this is the outcome of internal problems/cuttings/restructuring/profit-increase etc, then I truly very much hope it affects their reputation.

But I'm afraid it won't. Just like Microsoft continues to push out software, that, compared to competitors, is unstable, insecure, frustrating to use, lacks features, etc, without it harming their reputation or even bottomlines too much. I'm afraid Cloudflare has a de-facto monopoly (technically: big moat) and can get away with offering poorer quality, for increasing pricing by now.

coffeebeqn2• 8 months ago

Vibe infrastructure

MrAureliusR2• 8 months ago

well that's the thing, such a huge number of companies route all their traffic through Cloudflare. This is at least partially because for a long time, there was no other company that could really do what Cloudflare does, especially not at the scales they do. As much as I despise Cloudflare as a company, their blog posts about stopping attacks and such are extremely interesting. The amount of bandwidth their network can absorb is jaw-dropping.

I've said to many people/friends that use Cloudflare to look elsewhere. When such a huge percentage of the internet flows through a single provider, and when that provider offers a service that allows them to decrypt all your traffic (if you let them install HTTPS certs for you), not only is that a hugely juicy target for nation-states but the company itself has too much power.

But again, what other companies can offer the insane amount of protection they can?

zelphirkalt2• 8 months ago

Microsoft's reputation couldn't be much lower at this point, that's their trick.

The issue is the uninformed masses being led to use Windows when they buy a computer. They don't even know how much better a system could work, and so they accept whatever is shoved down their throats.

rsynnott• 8 months ago

Just like Microsoft continues to push out software, that, compared to competitors, is unstable, insecure, frustrating to use, lacks features, etc, without it harming their reputation or even bottomlines too much.

Eh.... This is _kind_ of a counterfactual, tho. Like, we are not living in the world where MS did not do that. You could argue that MS was in a good place to be the dominant server and mobile OS vendor, and simply screwed both up through poor planning, poor execution, and (particularly in the case of server stuff) a complete disregard for quality as a concept.

I think someone who'd been in a coma since 1999 waking up today would be baffled at how diminished MS is, tbh. In the late 90s, Microsoft practically _was_ computers, with only a bunch of mostly-dying UNIX vendors for competition. And one reasonable lens through which to interpret its current position is that it's basically due to incompetence on Microsoft's part.

gbrindisi• 8 months ago

The crowdstrike incident taught us that no one is going to review any dependency whatsoever.

pyuser583• 8 months ago

Lots of big sites are down

belter• 8 months ago

This will be another post-mortem of...config file messed...did not catch...promise to be doing better next....We are sorry.

They problem is architectural.

lucyjojo• 8 months ago

cloudflare is a huge system in active development.

it will randomly fail. there is no way it cannot.

there is a point where the cost to not fail simply becomes too high.

jcmfernandes• 8 months ago

Absolutely. I wouldn’t be surprised if they turned the heat up a little after the last incident. The result? Even more incidents.

bluerooibos• 8 months ago

I'm quite sure the reputational damage has already been done.

How do they not have better isolation of these issues, or redundancy of some sort?

brandensilva2• 8 months ago

The seed has been planted. It will take awhile for others to fill the void. Still the big players see this as an opportunity to steal market share if Cloudflare cannot live up to their reputation.

wooque• 8 months ago

2 days ago they had outage that affected Europe, Cloudflare seems to be going down the drain. I removed it for my personal sites.

dev0p• 8 months ago

Isn't it happening a little too often now? Did someone .unwrap in production again?

tovej• 8 months ago

Internet-level companies are having more outages recently. Is the exposed surface area increasing or is the quality of service suffering?

timvdalen• 8 months ago

Wow, just plain 500s on customer sites. That's a level of down you don't see that often.

disillusioned• 8 months ago

At least they branded it!

ablation• 8 months ago

Yeah that's a hard 500 right? Not even Cloudflare's 500 branded page like last time. What could have caused this, I wonder.

mckirk• 8 months ago

"A cable!"

"How do you know?"

"I'm holding it!"

Hamuko• 8 months ago

I hope it’s not another Result.unwrap().

willtemperley• 8 months ago

Yes Claude is down with a 500 (cloudflare).

ransom1538• 8 months ago

So. I don't understand the 5 nines they promote. One bad day those nines are gone. So they next year you are pushing 2 nines.

kingstnap• 8 months ago

Its just fabricated bullshit. It's how all the companies do it. 99.999% over a year is literally 5 minutes. Or under an hour in a decade, that's wildly unrealistic.

Reddit was once down for a full day and that month they reported 99.5% uptime instead of 99.99% as they normally claimed for most months.

There is this amazing combination of nonsense going on to achieve these kinds of numbers:

1. Straight up fraudulent information on status page. Reporting incendents as more minor than any internal monitors would claim.

2. If it's working for at least a few percent of customers it's not down. Degraded is not counted.

3. If any part of anything is working then it's not down. For example with the reddit example even if the site was dead as long as the image server is still at 1% functional with some internal ping the status is good.

maxekman• 8 months ago

A precious glimpse of the less seen page renders.

jondot2• 8 months ago

its like someone-shut-down-the-power 500s

gwd3• 8 months ago

Unlike the previous outage, my server seems fine, and I can use Cloudflare's tunnel to ssh to the host as well.

Eikon2• 8 months ago

Mine [0] seems to be very high latency but no 500s. But yes, most cloudflare-proxied websites I tried seems to just return 500s.

[0] https://www.merklemap.com/

sushidev• 8 months ago

Are you serious?

moralestapia• 8 months ago

Ooof, this one looks like a big one!

canva.com

chess.com

claude.com

coinbase.com

kraken.com

linkedin.com

medium.com

notion.so

npmjs.com

shopify.com (!)

and many more I won't add bc I don't want to be spammy.

Edit: Just checked all my websites hosted there (~12), they're all ok. Other people with small websites are doing well.

Only huge sites seem to be down. Perhaps they deal with them separately, the premium-tier of Cloudflare clients, ... and those went down, dang.

reddalo2• 8 months ago

My small websites are also up. I wonder if they're going to go down soon, or if we're safe.

shultays• 8 months ago

zoom

otherme1232• 8 months ago

readthedocs down is hurting me the most. My small websites are doing OK.

atraac• 8 months ago

All those enterprise architects must be fuming now

aurareturn2• 8 months ago

My company's services went down as well.

SherryWong• 8 months ago

LinkedIn and MEdium are also down as a result

chinathrow2• 8 months ago

Looks like (some) sites behind Cloudflare still work if they do not have caching on.

jonathanlydall3• 8 months ago

It's not simply about caching as we have CDN and reverse proxying which are still running without issue.

songtianlun1• 8 months ago

yes...

jonathanlydall3• 8 months ago

It seems regular reverse proxying and R2 still works, as we use those and seem to be working fine still.

Can't get to the Dashboard though.

thiscatis• 8 months ago

Somebody at Cloudflare is stretching that initial investigation time as much as possible to avoid having to update their status to being down and losing that Christmas bonus.

nabla9• 8 months ago

It's configuration error or related to configuration. It always is with this big things.

Nice thing about Cloudflare being down is that almost everything is down at once. Time for peace and quiet.

norskeld• 8 months ago

Damn, I wish CloudFlare being down also affected local development, so I could take a break from doing frontend… :'(

da_grift_shift2• 8 months ago

https://www.cloudflarestatus.com/incidents/hlr9djcf3nyp

We will be performing scheduled maintenance in ORD (Chicago) datacenter
Traffic might be re-routed from this location, hence there is a possibility of a slight increase in latency during this maintenance window for end-users in the affected region.

Looks like it's not just Chicago that CF brought down...

yessferatu• 8 months ago

South African here. Down on our side. Huge sites, like our primary news site is down - medical services, emergency service/information etc... all down. It's been like this since 11:00am our time, so about 13minutes now.

maxlin4• 8 months ago

Go to <social media page> - 500 error from cloudflare Google is <social media page> down -> click first link - literally the exact same 500 cloudflare error html from downdetector

I thought we were meant to learn something ... ?

theginger• 8 months ago

I don't want to criticize cloud flare, I love what they do and understand the scale of the challenge, but most people don't and 2 in a month or so like this is going to hit their reputation.

The_President4• 8 months ago

After being overly critical of Matrix the other day on here I have reeled back into another conclusion, is that talent issues are industry wide and it sucks making a bad hire where competence issues arise that don’t match the resume.

3xstphvs• 8 months ago

aw, i cant go on rateyourmusic

phartenfeller2• 8 months ago

Wow, three times in a month is really crushing their trust.

dabeeeenster• 8 months ago

3?! When was the second>

ubercore• 8 months ago

https://news.ycombinator.com/item?id=46140145

8cvor6j844qw_d63• 8 months ago

I'll need to checkup on DigitalOcean uptime, may be better than Cloudflare.

phartenfeller2• 8 months ago

My Hetzner servers have been running fine for years. Okay, there were times when I broke something, but at least I was able to fix it quickly and never felt dependent on others.

countWSS• 8 months ago

Everything i use depend on perfect cloudflare operation workflow, practically 99% of these services go down. What magical qualities it has that no competitors form for its services?

nicolailolansen• 8 months ago

They had a few good weeks.

arunaugustine2• 8 months ago

They had a scheduled maintenance between 7am and 11am UTC in Chicago. But that should have re-routed traffic not take down internet right?

PrayagS2• 8 months ago

I'm in India and we're affected as well.

J4PJ1T• 8 months ago

Oceania here gang and i think that it is a global issue

makkoncept• 8 months ago

https://downdetectorsdowndetector.com/ is up :) but the status is not correct.

matt32102• 8 months ago

Ooof status 500 someone’s getting fiiiiired!

Hashversion3• 8 months ago

how long cloudflarestatus.com takes it to detect usually?

valdemarrolfsen• 8 months ago

No engineers from Cloudflare reading hackernews these days? Should update your status page!

Palmik• 8 months ago

This is second time this week: https://news.ycombinator.com/item?id=46140145

The previous one affected European users for >1h and made many Cloudflare websites nearly unusable for them.

0xfedcafe2• 8 months ago

Funny how even safe Rust isn’t able to stop vibecoding without a proper validation. And the fact that it's a monopoly isn't so funny anymore.

dkdbejwi383• 8 months ago

There is no language that makes it impossible to have any kind of bug ever. The safety languages like Rust offer is around memory, not bad configuration or faulty business logic.

lionkor3• 8 months ago

Rust is one of the few languages where I found AI to be very well checked. The type system can enforce so many constraints that you do avoid lots of bugs, and the AI will get caught writing shit code.

Of course, vibe coding will always find a way to make something horribly broken but pretty.

nromiun2• 8 months ago

I have noticed LLMs tend to generate very verbose code. What an average human might do in 10 LoC, LLMs will stretch that to 50-60 lines. Sometimes with comments on every line. That can make it hard to see those bugs.

0xfedcafe2• 8 months ago

Yep, that’s what I wrote. It wasn’t a sarcasm

m078• 8 months ago

Cloudflare is investigating issues with Cloudflare Dashboard and related APIs.

wildcard12102• 8 months ago

my shopify store is down

ojm• 8 months ago

Turnstile seems up still.

aroman2• 8 months ago

looks like a big one. interestingly, our site, which uses a TON of Cloudflare services[0] — yet not their front-line proxy — is doing fine: https://magicgarden.gg.

So it seems like it's just the big ol' "throw this big orange reverse proxy in front of your site for better uptime!" is what's broken...

[0] Workers, Durable Objects, KV, R2, etc

bpye• 8 months ago

Moving off of Cloudflare for my personal domain is on my todo list for the holidays...

reassess_blind4• 8 months ago

My sites that use their main proxy are seemingly up and working? Could be a regional PoP issue.

wildcard12102• 8 months ago

My Shopify store is down. My competitor stores are also down.

_cenw3• 8 months ago

That's the 30% vibe code they promised us.

Cynicism aside, something seems to be going wrong in our industry.

nlitened• 8 months ago

Also “Rewrite it in Rust”.

P.S. it’s a joke, guys, but you have to admit it’s at least partially what’s happening

koakuma-chan7• 8 months ago

No, it has nothing to do with Rust.

kenonet3• 8 months ago

it's never the technology, it's the implementation

MegaThorx• 8 months ago

Did you consider to rewrite your joke in rust?

rifycombine12• 8 months ago

cc: @oncall then trigger pagerduty :)

joenada• 8 months ago

Going? I think we got there a long time ago. I'm sure we all try our best but our industry doesn't take quality seriously enough. Not compared to every other kind of engineering discipline.

_cenw3• 8 months ago

Always been there. But it seems to be creeping into institutions that previously cared over the past few years, accelerating in the last.

themafia2• 8 months ago

Salaries are flat relative to inflation and profits. I've long felt that some of the hype around "AI" is part of a wage suppression tactic.

iso16315• 8 months ago

Cynicism aside, something seems to be going wrong in our industry.

Started after the GFC and the mass centralisation of infrastructure

nromiun2• 8 months ago

I wonder if it is another bug , like unwrap, in their rewritten code.

Also, I don't think their every service got affected. I am using their proxy and pages service and both are still up.

davidcheungo123• 8 months ago

wtf, cannot work now

paweladamczuk• 8 months ago

I noticed this when my Claude iPhone app stopped working.

Hashversion3• 8 months ago

cloudflare pages seems to be working!

odie5533• 8 months ago

How is Hacker News still up?

sunbum2• 8 months ago

Because it doesn't use cloudflare duh.?

otherme1232• 8 months ago

I have a handful of sites DNS/NS through Cloudflare, with their certificates, and they are working OK.

PrayagS2• 8 months ago

From their response headers, it seems like the request is coming from NGINX directly. How do they defend themselves against DOS attacks?

grundrausch3n• 8 months ago

I thought they are running classic FreeBSD servers like in ye olde times.

yread• 8 months ago

Hah even Linkedin is showing 500 for me

computersuck• 8 months ago

waaay too soon

b_bloch• 8 months ago

That's quite unfortunate xD

SCdF• 8 months ago

Really disappointed that down detectors down detector[1] isn't detecting that down detector[2] is down

[1] https://downdetectorsdowndetector.com/

[2] https://downdetector.com/

meindnoch3• 8 months ago

Maybe they should stop vibe coding and vibe reviewing their PRs?

Dilettante_2• 8 months ago

"I warned you about Cloudflare bro!!!! I told you dog!"

grim_io• 8 months ago

I wonder how many uptime SLAs will be violated this year.

virtualritz• 8 months ago

Yeah, and because of this for example Claude Code is down too because the auth goes through CF. F*cking marvelus, the decentralized web ...

reneberlin• 8 months ago

I can imagine the horror of pressure of the people responsible for resolution. On that scale of impact it is very hard to keep calm - but still the hive of minds have to cooperate and solve the puzzle while the world is basically halted and ready to blame the company you work for.

ricardo81• 8 months ago

Their uptime over the year is likely faring worse than your average hosting company, DNS provider or CDN.

cryptonym10• 8 months ago

Some may experience more downtime due to their outages than they'd have from DDoS.

iso16315• 8 months ago

Their uptime over the year is faring worse than one of my pi holes, let alone the resilient service.

igleria• 8 months ago

Heads will roll at cloudfare. E-commerce customers must be furious.

Impossible not to feel bad for whoever is tasked to cleanup the mess.

zppln• 8 months ago

Especially around christmas. I was about to buy a pair of Birkenstocks. Nope, site is down. Went on to buy a microphone holder, nope, that site is down as well. :) Sure, I'll still get around to it eventually.

domysee2• 8 months ago

I'm just realizing how much we depend on Cloudflare working. Every service I use is unreachable. Even worse than last time. It's almost impossible to do any work atm.

yoctosec3• 8 months ago

I use Cloudflare because of their Tunnel to protect my Raspberry Pi, but I think I will just use it without the Tunnel now. My main concern is privacy, but I'm not ready to accept so frequent downtime and dependence on them. The whole reason to host self-host was to be independent anyway. Does anyone have a recommendation for that (that is free)? Should I worry about privacy? My name and my city are on the website anyway.

runeb3• 8 months ago

Checkout tailscale

unixfox• 8 months ago

Tailscale's control plane uses Cloudflare.

yoctosec3• 8 months ago

And what about a website I want to make public? I'm just concerned about my IP being visible, like for my personal website or my searxng instance

DocJade• 8 months ago

my tunnels are still working, oddly

yoctosec3• 8 months ago

Now mine works again too, I guess it was a short outage

The_President4• 8 months ago

You can basically do this yourself without punching holes out to the public. Create VPN for Pi and clients and access from the same private network.

LeonenTheDK• 8 months ago

Nice, just got woken up by my outage alarms, just for it to be Cloudflare again. At least it's _my_ problem!

But my goodness, they're really struggling over the last couple weeks... Can't wait to read the next blog post.

koakuma-chan7• 8 months ago

Outage alarms?

bytejanitor• 8 months ago

gitlab.com hasn't noticed yet.

alex_suzuki• 8 months ago

it has now, for me. can't access web UI (SaaS, not self-hosted, obviously)

erikbye2• 8 months ago

This is getting embarrassing.

hasperdi• 8 months ago

Even LinkedIn is now down. Opening linkedin.com gives me a 500 server error and Cloudflare at the bottom. Quite embarassing.

_cenw3• 8 months ago

At least they were available when Front Door was down!

Oras• 8 months ago

Went to ahref to check a domain, saw 500 and came here to check.

I have a few domains on cloudflare and all of them are working with no issues so it might not be a global issue

basisword• 8 months ago

I’m sure everybody learnt their lesson from last months outage and built in redundancy or stopped relying on Cloudflare.

matt32102• 8 months ago

Everyone says vibe coding but people are just fine at being incompetent without the AI help

koolba2• 8 months ago

Sure, but with AI we can automate that incompetence.

jondot2• 8 months ago

LinkedIn is down

CodinM• 8 months ago

came here for this thx

xingwu• 8 months ago

Classic. https://imgur.com/a/B3QxB1R

alextingle• 8 months ago

"Content not available in your region."

Please avoid Imgur.

sebzim4500• 8 months ago

Use a vpn or avoid the UK

rgun• 8 months ago

https://registry.npmjs.org/ is down, affecting our builds

CGamesPlay• 8 months ago

So is https://hub.docker.com which is why I am here and not doing useful work.

erikbye2• 8 months ago

Cloudflare uptime has worsened a lot lately, AI coding has increased exponentially, hmm

meerab• 8 months ago

It is up now!

dale11104• 8 months ago

You sure?

dale11104• 8 months ago

Just checked. It's up!!

Ueland• 8 months ago

Interestingly enough, also some MS/Azure services are down. For example https://www.office.com/ just returns:

We are sorry, something went wrong. Please try refreshing the page in a few minutes. If the problem persists, please visit status.cloud.microsoft for updates regarding known issues.

The status page of course says nothing

GeertVL• 8 months ago

Linkedin -> the same

nikanj• 8 months ago

For me Linkedin returns the 500 cloudflare error

codeisforever• 8 months ago

Seems all of Shopify.com is down. Every store

vinskabun• 8 months ago

pixiv.net

jazzyjackson• 8 months ago

Is it at all achievable to be fronted by a CDN but fallback to the raw server in case the front falls off? Better to be vulnerable to DDoS than be unreachable altogether

calyhre• 8 months ago

But then you end up potentially exposing the origin server. This could be an opt-in option though

koolba2• 8 months ago

With CloudFlare specifically probably not. IIRC, they require DNS resolution of your domain to operate so if they’re down, I don’t see how you’d change it to route directly to the underlying site.

Even if you could, having two sets of TLS termination is going to be a pain as well.

justmarc• 8 months ago

Obligatory song https://www.youtube.com/watch?v=OC06Z6lCB_Q&t=30s

ianberdin6• 8 months ago

NPM is down. World is collapsing thanks to Cloudflare.

w4zz2• 8 months ago

gitlab down aswell

neo_tokyo• 8 months ago

Someone's been vibe coding the scheduled maintenance.

samwreww• 8 months ago

claude.ai is down bc of it :( good for OpenAI as they're using something else maybe Vercel?

ThalesX• 8 months ago

I just started getting npm errors while developing something; I was like hmm, strange... then I tried to go down to isitdown. That was also down. I was like, oh this must be something local to me (I'm in a remote place visiting my gramps).

Then I go to Hacker News to check. Lo and behold, it's Cloudflare. This is sort of worrying...

nlstitch• 8 months ago

What ever happened to "no deploys on fridays"? haha

kenonet3• 8 months ago

haha for real

ianberdin6• 8 months ago

I have 10B idea: cloudflare that does not fail so often.

reddalo2• 8 months ago

It exists and it's called Bunny.net

biql• 8 months ago

How about: internet that is actually decentralized.

ianberdin6• 8 months ago

Yes, on one hand, it was so wonderful. Cloudflare came and said, "Yeah, now we'll save everyone from DDoS, everything's perfect, we'll speed up your site," and bam, they became a bottleneck for the entire internet. It's some kind of nightmare. Why didn't several other such popular startups appear, into which more money was invested, and which would allow some failure points to be created? I don't understand this. Or at least Cloudflare itself should have had some backup mechanism, so that in case of failures, something still works, even slowly, or at least they could redirect traffic directly, bypassing their proxies. They just didn't do that at all. Something is definitely wrong.

iso16315• 8 months ago

Nobody got fired for choosing clownflare

SoKamil• 8 months ago

Looking at their market cap it’s 71.5B idea

MildlySerious• 8 months ago

I can't update DNS entries for my domains with Porkbun, because it's "Powered by Cloudflare".

blackhaz• 8 months ago

Anyone shorting the damn stock?

nish__• 8 months ago

Just in time for the London work day :)

kinensake• 8 months ago

Every time Cloudflare is down I'm not sure if it's really down or not because most down detector websites use Cloudflare. Lmao

Artur-Defences3• 8 months ago

This is not a good look, at all

manupati• 8 months ago

Odd

SwedishPerson_A2• 8 months ago

https://www.tandfonline.com/doi/full/10.1080/02673843.2023.2... https://www.perplexity.ai/ https://www.researchgate.net/

All give me

"500 Internal Server Error cloudflare.."

So I'm guessing yes.

nekkooo2e2• 8 months ago

Perplexity is down

chistev2• 8 months ago

It's really cool to me that this site is never down with all these outages of major websites.

Representative of having the best developers behind it.

ilaksh• 8 months ago

They just don't use Cloudflare.

chistev2• 8 months ago

How do they handle DDoS?

CafeRacer• 8 months ago

Even digital ocean is down :D

nekkooo2e2• 8 months ago

Perplexity AI shows 500 Internal Server Error

divanvisagie• 8 months ago

Wishbone12

neonnbits• 8 months ago

i was watching the climax of "Fullmetal Alchemist: Brotherhood" on crunchyroll and cloudflare went down again

elijah3040448• 8 months ago

Pretty awkward. Thought my WIFI was acting up when I wasn't even able to pull up the Cloudfare website to see if something was down. Then, trying to go to Downdetector and that wasn't working either.

techguy1954• 8 months ago

I can still visit some websites that use Cloudflare, but other don't work.

Blender Artists works, but DownDetector and Quillbot dont.

isaac33073• 8 months ago

This is so cool guys. All of us get to lose millions of dollars together so late at night!

isaac33073• 8 months ago

This is so cool guys!!! We all get to savor this moment and lose millions of DOLL HAIRS together!!

isaac33073• 8 months ago

I love being here with you guys!!!

segev608• 8 months ago

Luckly https://downdetectorsdowndetectorsdowndetectorsdowndetector.... is up :)

SwedishPerson_A2• 8 months ago

https://www.researchgate.net/ https://www.tandfonline.com/ https://www.perplexity.ai/ All give me "500 Internal Server Error cloudflare"

So My guess is yes It´s down.

cv_h• 8 months ago

Free accounts seem to be fine, only enterprise accounts seem to be affected.

doublerabbit• 8 months ago

Irony.

epolanski• 8 months ago

I can absolutely accomplish nothing today...can't download npm packages, cannot login to services.

I've been a Cloudflare fan for the longest time, but the more they grow the more they look like the weak link of the internet. This is the second major outage in less than few weeks. Terrible.

polaris64• 8 months ago

DownDetector'sDownDetector does not detect that DownDetector's down

kenonet3• 8 months ago

stock going whoops

8cvor6j844qw_d63• 8 months ago

Interested if its the same issue that brought down Cloudflare previously.

ammo1662• 8 months ago

"Given Cloudflare's importance in the Internet ecosystem any outage of any of our systems is unacceptable. "

Is this a joke?

And their blog of above statement is also down:

https://blog.cloudflare.com/18-november-2025-outage/

Towaway69• 8 months ago

for me docker is failing with:

    unknown: failed to copy: httpReadSeeker: failed open: unexpected status from GET request to https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/....

so coffee time.

monstertank• 8 months ago

How do you tell if this is a cyberattack or not?

dale11104• 8 months ago

Tried to watch anime then realized that cloudflare was down...again. smh

robotfelix• 8 months ago

Our site is fine, including files served by Cloudflare's CDN and Cloudflare Workers, but the Cloudflare dashboard is definitely down.

The Cloudflare status page says that it's the dashboard and Cloudflare APIs that are down. I wonder if the problem is focused on larger sites because they are more dependent on / integrated with Cloudflare APIs. Or perhaps it's only an Enterprise tier feature that's broken.

If it's not everything that is down, I guess things are slightly more resilient than last time?

vimwizard2• 8 months ago

seems related to CF tunnels... policies are being enforced but perhaps origin servers are not being properly served.

kvam• 8 months ago

Infrastructure-as-Vibe?

zwnow4• 8 months ago

I love it, and we wont learn from this again :-) Looking forward for the 3rd outage in a few weeks.

venturecruelty• 8 months ago

We should have a weekly thread for it! It can be a fun meetup. Hell, move it to a cafe or a pub or something, and we can use it as a chance to disconnect from the internet and talk to people face to face. Instead of saying "touch grass", we can say "get Cloudflared".

tin7in• 8 months ago

For us also Digital Ocean, Render, and a few other vendors are down.

At this point picking vendors that don't use Cloudflare in any way becomes the right thing to do.

bigfudge• 8 months ago

Claude was also down (which brought me here)

dale11104• 8 months ago

Tried to watch anime and then i realized it was down....again. smh

tippa123• 8 months ago

Curious to see which big companies were caught flat-footed during the 18 November outage compared with today. In my opinion, if a company was caught out twice, that reflects poor decision-making and urgency. As the saying goes, fool me once, shame on you, fool me twice, shame on me.

If a company was able to overcome all the red tape within three weeks and not be impacted today, that's impressive.

ednevsky• 8 months ago

Notion is also down (haven't seen a comment on that). It's so funny how the biggest companies literally just have their sites not loading because of Cloudflare.

hax0r1338• 8 months ago

This gotta be an attack, no way its configuration error again.

jonathrg• 8 months ago

Why not? They have been proudly vibe coding for a while.

bilekas• 8 months ago

This is painful, if I'm not mistaken this is during a scheduled maintenance too ?

Whenever I deploy a new release to my 5 customers, I am pedantic about having a fast rollback.. Maybe I'm not following the apparent industry standard and instead should just wing it.

zwnow4• 8 months ago

Let AI wing it instead.

yellow_lead• 8 months ago

Is anyone else woken up by this? My company's service is down too. Considering a move away

maxlin4• 8 months ago

half internet down first "is site down" result (downdetector) down downdetectorsdowndetector.com: "everything is fine" downdetectorsdowndetectorsdowndetector.com: not even responding downdetectorsdowndetectorsdowndetectorsdowndetector.com: "everything is broken"

Artur-Defences3• 8 months ago

"Scheduled maintenance is currently in progress" I image the maintenance was conducted like this: "fix detroit data center bugs, please be very careful, don't mess up like last time :)" bypass permissions on

tdrz• 8 months ago

I'm looking for cofounders and investors to build a working cloudflare.

udarij2• 8 months ago

It's ok to fail, but most frustrationg thing is there's no suppoting team or any contact point accessible directly. this is bad..

Vivianfromtheo• 8 months ago

This got me and the anime community stressed

dracotomes• 8 months ago

and it's back

sandruso• 8 months ago

it's back on

but wow, it must be stressful to deal with this

udev4096• 8 months ago

Clownflare strikes again!

Artur-Defences3• 8 months ago

"Monitoring - A fix has been implemented and we are monitoring the results."

decimalenough• 8 months ago

Seems to be back up.

StrLght• 8 months ago

And it's on Friday again — never change, Cloudflare.

Gentle reminder that every affected company brought it upon themselves. Very few companies care about making their system resilient to 3rd party failures. This is just another wake-up call for them.

max23_• 8 months ago

It is up for me.

All the sites that were 500 error before are able to load now.

JustSkyfall• 8 months ago

Seems to be back up?

pprotas• 8 months ago

What a joke of a company. They have the internet in the palm of their hands, and yet let vibe coding ambitions ruin their empire.

Time for everyone to drop this company and move on to better solutions (until those better solutions rot from the inside out, just like their predecessor did)

hnarn• 8 months ago

One has to wonder how many times or how often proprietary cloud services have to go down before there is a general shift away from using the cloud and "infinite scaling" for everything. For many, many use cases you do not need neither Cloudflare nor Github nor nine nines for everything (which you are clearly not getting anyway). It's obviously not enough with once a year for most businesses, or perhaps once a month. Weekly outages? For how long?

If you host something that actually matters that other people depend upon and, please review your actual needs and if possible stop making yourself _completely_ dependent on giant cloud corporations.

dodos• 8 months ago

Looks to be back now.

michael-sumner• 8 months ago

It's up now!!! London, UK

lionkor3• 8 months ago

eval(requestBody).unwrap()

starkindustries• 8 months ago

Never push to production on Friday!

jachee2• 8 months ago

Update title to “Tell HN: Cloudflare was down”

MarcelGerber• 8 months ago

Just started working for me again (in Germany), both on our own CF-hosted page and on cloudflare.com itself.

Titan2189• 8 months ago

They're back online it seems

digiajay• 8 months ago

Basecamp was down couple of minutes ago and it's back now online.

NKosmatos• 8 months ago

LOL, 500 returned for many big sites…this is going to hurt and make people rethink. If it’s not DNS, then someone pushed to production on Friday :-)

chinathrow2• 8 months ago

It's back.

jjdinho• 8 months ago

Godspeed, Cloudflare, for the fix

w4zz2• 8 months ago

Up again!

vimwizard2• 8 months ago

she's back

alxbenjamin• 8 months ago

It is up again. There will be a lot of hard talk with Cloudflare, I guess

DaSilentStorm2• 8 months ago

Aaand ... we're back

lrvick• 8 months ago

I was just arguing yesterday to coworkers I would quit tech before helping centralize any more of the internet on Cloudflare as a massive single point of failure.

Thank you, Cloudflare, for again proving my point.

domysee2• 8 months ago

It's back!

skylurk• 8 months ago

Monitoring - A fix has been implemented and we are monitoring the results.
Dec 05, 2025 - 09:12 UTC

MrAureliusR2• 8 months ago

Yeah, cloudflare.com is working and the website that first clued me in to the outage (chess.com) is also working.

udarij2• 8 months ago

It's ok to fail. but the most frustrating thing ever is... there's no contact point or supporting team easily and directly accessible.. this is bad..

DaSilentStorm2• 8 months ago

Aaand ... we're back!

donbox• 8 months ago

seems its back \m/

yapyap2• 8 months ago

seems to have been resolved

BluSyn• 8 months ago

Perhaps related? My main fiber WAN went out few hrs ago, failing over to Starlink backup. Discovered it’s a cloudflare issue, as my multi-wan setup tests against 1.1.1.1, which suddenly stopped responding (but only from my fiber ISP). Switched to testing 8.8.8.8 to restore.

If it weren’t for recent cloudflare outages, never would have considered this was the problem.

Even until I saw this, I assumed it was an ISP issue, since Starlink still worked using 1.1.1.1. Now I’m thinking it’s a cloudflare routing problem?

ianberdin6• 8 months ago

So, I understand correctly that all websites and services want protection from DDoS attacks, and that's basically their number one concern. The second is caching in different parts of the world. So, it's caching and DDoS. But at the same time, nobody wants to use CloudFront from AWS because it’s not that simple yet. And it’s more expensive, while Cloudflare is free. So, what should we do about all this? This won’t do. We’ve created a gigantic bottleneck that controls the entire internet, just like in the movie Mad Max, where he controlled the only source of water. That’s wrong. And we all fell for it like fools. So, the question is, what can be done in this situation? Are there reliable competitors? Are there any fault-tolerant systems for this? The whole problem is that our DNS, and with Cloudflare, they proxy it. So, if their proxy goes down, everything falls apart. What should we do about this?

sammy22553• 8 months ago

Nobody is being forced to use Cloudflare

ianberdin6• 8 months ago

Since everything is absolutely correct, no one forced it; they just provided a good, excellent solution for free, and consequently, the whole internet has gotten hooked on it. As they say, free cocaine causes harm. So, what are the alternatives? What options are there to protect against DDoS attacks and to make a website quickly accessible from different parts of the world? And at the same time, without paying a sky-high price for it.

nedt• 8 months ago

Everyone trying to access a site behind Cloudflare is forced.

6031769• 8 months ago

Let your hoster take care of the DDoS and stop using the flaky behemoth.

You haven't actually watched Mad Max, have you? I do recommend it.

drexlspivey• 8 months ago

Who is we? You are free to stop using their service

chaidhat2• 8 months ago

Someone should make an open source system that lets you easily host containers so that if one fails, we can easily switchover across providers. Like Vercel AI SDK but for containers. That is, if docker isnt failing (it is right now cause it depends on Cloudflare)

someothherguyy• 8 months ago

for all of 20 minutes, the world cried.

OtherShrezzing• 8 months ago

The site is back up, but it feels fairly silly that a platform that has inserted itself as a single point of failure has an architecture that's got single points of failure.

The other companies working at that scale have all sensibly split off into geographical regions & product verticals with redundancy & it's rare that "absolutely all of AWS everywhere is offline". This is two total global outages in as many weeks from Cloudflare, and a third "mostly global outage" the week before.

themafia2• 8 months ago

Crop monoculture created the potato famine. We failed to learn the larger lesson. "Hyperscale" is inherently dangerous.

adityashankar• 8 months ago

it's fine now...I believe

elcapithanos• 8 months ago

Shortest damn outage ever

scirob• 8 months ago

Wonder if supabase auth down is also related https://status.supabase.com/incidents/rgz3dl2rcmq8

zwnow4• 8 months ago

Will it be down for 10 days again? Who knows. Would've stopped using it after the first 10 day outage anyway.

andy_ppp2• 8 months ago

Just a reminder that every dependency you rely on, both inside your codebase and external services, has a price.

madjam0022• 8 months ago

Looking forward to the post mortem on this one. We weren't affected (just using the CDN), and people are saying they weren't affected who are using Cloudflare Workers (a previous culprit which we've since moved off), so I wonder what service / API was actually affected that brought down multiple websites with a 500 but not all of them.

Wise was just down which is a pretty big one.

Also odd how some websites were down this time that previously weren't down with the global outage in November

thinkindie• 8 months ago

we were not affected too and we realised it was Cloudflare because Linear was down and they were mentioning an upstream service. Also Ecosia was affected, and I then realised they might be relying on Cloudflare too.

gowthamgts12• 8 months ago

CDN was also affected for some customers. we were down with 500.

m_mueller• 8 months ago

Maven Repository was down for me for a while, now it recovered.

reassess_blind4• 8 months ago

Yeah it's strange. My sites that are are proxied through Cloudflare remained up, but Supabase was taken offline so some backends were down. Either a regional PoP style issue, or a specific API or service had to be used to be affected.

archon810• 8 months ago

Our locations excluded from Cloudflare WAF were up, but the rest was down. I think WAF took a dump.

kryptn• 8 months ago

was interesting, some of our stuff failed, but some other stuff that used cloudflare indirectly didn't.

themly• 8 months ago

CDN was definitely down also. We were widely impacted by it with 500's.

cryptonym10• 8 months ago

Looking forward to the post mortem

This is becoming a meme.

meandmycode• 8 months ago

This has to be setting off some alarm bells internally, a well written postmortem on an occasional issue, great, but when your postmortem talks about learnings and improvements yet major outages keep happening, it becomes meaningless..

da_grift_shift2• 8 months ago

The excuse:

A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning.
The change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components.
We will share more information as we have it today.

https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q

madjam0022• 8 months ago

It's quite an unfortunate coincidence that React has indirectly been the reason for two recent issues at Cloudflare haha

GoblinSlayer3• 8 months ago

Who sent an xml request?

gritzko• 8 months ago

The entire Cloud/SaaS story had a lot of happy-path cost optimization. The particular glitch that triggered the domino effect may be irrelevant relative to the fact that the effect reproduces.

dynamite-ready• 8 months ago

Some of the sites I maintain, are fine. But I'm guessing it's just a matter of time?

kUdtiHaEX• 8 months ago

Cloudflare just closed down the incident on their status page without any additional explanation. Sigh.

chessmaster-hex• 8 months ago

Some big fishes were affected as well... Crunchyroll, Fortnite, LinkedIn let's wait for the explanation of this one.

r721• 8 months ago

Just after Matthew Prince's interview at Wired :)

https://news.ycombinator.com/item?id=46157295

Hashversion3• 8 months ago

what's the estimated loss? any guesses or estimations?

capnsketch• 8 months ago

If I had a nickel for everytime cloudflare went down. Then I would have 2 nickels which is not a lot but still wierd that it happened twice.

cryptonym10• 8 months ago

You would have 2 nickels, this week.

It also went down multiple times in the past; not to say that's bad, everyone does from time to time.

TheGilDev• 8 months ago

I’m still glad they’re here to provide great services and help secure the internet for lots of us!

arjie• 8 months ago

How interesting. As of 00:30 or so I could still access Claude but then it went down with a 500 from Cloudflare and I thought I'd nab a quick something off Slickdeals but that's down too. My own blog is on Cloudflare's `cloudflared` tunnel and it's working just fine, even the cache, so it must be something hitting some specific type of configuration or some shard hitting some region.

And they're back before I finished the comment. Such a pity, I was hoping to hog some more Claude for myself through Claude Code.

JeremyJaydan2• 8 months ago

I moved away from Cloudflare over a month ago because I didn't understand how they don't have pricing caps for their upgraded plans, they genuinely seem like the mob but I haven't looked any further into it..

Either way it's been interesting to see the bullets I've been dodging.

jjude• 8 months ago

What service(s) are you using now? What did you move to?

JeremyJaydan2• 8 months ago

A small one (afaik) in a location that I wanted in the US [1]. I'm not running a bank so I'd prefer to just go down if I'm ever attacked.

[1] https://shifthosting.com/

nickdothutton• 8 months ago

So many outages now they all begin to swim into 1, what's that 3 or 4 this quarter?

chaidhat2• 8 months ago

For those saying we have an over-reliance on software -- is there a way to use multiple CDNs for the same frontend website?

LucasLanglois• 8 months ago

Love that Cloudflare put together a participative and community-driven advent calendar!

reassess_blind4• 8 months ago

The "half the internet is down, nothing we can do" excuse works great the first time, but doesn't fly the second time in a month.

What solutions are there for Multi DNS/CDN failover that don't rely on a single point of failure?

pech0rin• 8 months ago

Rewriting in Rust is paying dividends.

nrhrjrjrjtntbt4• 8 months ago

Not as down as last time. My site is up.

f311a• 8 months ago

A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today.

cherioo• 8 months ago

Where’s the source for this?

It doesn’t look good when similar WAF issues caused their big outage a few years back.

xomiachuna2• 8 months ago

https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q

hnarturpl• 8 months ago

We use workers and dns proxy and I got flooded with pages. We were getting 503s from workers.

luastoned• 8 months ago

From the incident page:

A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today.

https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q

philipwhiuk2• 8 months ago

So their parser broke again I guess.

And no staged rollout I assume?

tialaramex5• 8 months ago

Apparently somehow this had never been how Cloudflare did this. I expressed incredulity about this to one of their employees, but yeah, seems like their attitude was "We never make mistakes so it's fastest to just deploy every change across the entire system immediately" and as we've seen repeatedly in the past short while that means it sometimes blows up.

They have blameless post mortems, but maybe "We actually do make mistakes so this practice is not good" wasn't a lesson anybody wanted to hear.

uyzstvqs• 8 months ago

Ah yes, Cloudflare's worst enemy: The configuration change.

hinkley2• 8 months ago

On fridays, yes.

meindnoch3• 8 months ago

React (a frontend JS framework) can now bring down critical Internet infrastructure.

I will repeat it because it's so surreal: React (a frontend JS framework) can now bring down critical Internet infrastructure.

cryptonym10• 8 months ago

That's Next.js, not React.

Mentioning React Server Components in the status page can be seen as a bad way to shift the blame. Would have been better to not specify which CVE they were trying to patch. The issue is their rollout management, not the Vendor and CVE.

spiffytech• 8 months ago

To be accurate: React developed server-side capabilities, and that's where the vulnerability exists.

It's feels noteworthy because React started out frontend-only, but pedantically it's just another backend with a vulnerability.

reassess_blind4• 8 months ago

I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configuration changes to 1% of global traffic first.

Traubenfuchs• 8 months ago

"Please don‘t block the rollout pipleline with a simple react security patch update."

lima3• 8 months ago

They don't appear to have a rollout procedure for some of their globally replicated application state. They had a number of major outages over the past years which all had the same root cause of "a global config change exposed a bug in our code and everything blew up".

I guess it's an organizational consequence of mitigating attacks in real time, where rollout delays can be risky as well. But if you're going to do that, it would appear that the code has to be written much more defensively than what they're doing it right now.

stogot• 8 months ago

The update they describe should never bring down all services. I agree with other posters that they must lack a rollout strategy yet they sent spam emails mocking the reliability of other clouds

gpi• 8 months ago

I believe they use Argo according to a previous post mortem.

https://blog.cloudflare.com/deep-dive-into-cloudflares-sept-...

aatd86• 8 months ago

so it's react again in the end .. zzzzzzz

o_m• 8 months ago

I wonder if this is the new normal? Weekly Cloudflare outages that breaks huge parts of the internet.

pepoluan• 8 months ago

So. Another regex problem?

asim• 8 months ago

I'd like to start seeing the architecture and design of how cloudflare works. Not blog posts, like a whole write-up. If you're going to have this many outages and you're a public company which 2/3rd of US infrastructure probably depends on then it might need some external input. Obviously they know what they're doing. This is not a blame game but the tools are starting to creak.

import• 8 months ago

Fraudulent status page again. For sure we are seeing something very different than what they see in the internal monitoring systems.

maverick98• 8 months ago

The quality of code in most apps has gone downhill. Let's guess why?

reustle• 8 months ago

Maybe centralizing the internet wasn’t a great idea after all, huh

glimshe2• 8 months ago

Not only they make my browsing experience a LOT worse (seconds per site for bot detection and additional "are you human" clicks even without VPNs), now they are bringing the entire Internet down. They don't deserve the position they currently have.

wrobelda• 8 months ago

You make it sound like the DDoS and Bots are their fault.

glimshe2• 8 months ago

They make gazillions. I'm sure they can do better than that.

How many awful things in tech can be rationalized away by "sorry, but this is for you/our protection"?

gilrain• 8 months ago

Not only they make my browsing experience a LOT worse

No, I did (metaphorically, for the websites I control). And I did it because otherwise those sites are fully offline or unusable thanks to the modern floods of unfilterable scrapers.

Months of piecemeal mitigations, but Attack Mode is the only thing that worked. Blame the LLM gold rush and the many, many software engineers with no ethics and zero qualms about racing to find the bottom of the Internet.

The_President4• 8 months ago

The whole “not a bot” prompt every three hours seems like it has potential to get out of the way more often.

grimblee• 8 months ago

Always has been

throwaway48476• 8 months ago

Shipping .unwrap() to prod speaks ill of their engineering culture. Quality is a process not a checkbox.

xomiachuna2• 8 months ago

And so it seems that the cause is close to RSC vulnerability from yesterday: https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q

So much for the react being just a frontend library, amirite

rco8786• 8 months ago

React server components initially came out 5ish years ago, for whatever that's worth.

markaroo• 8 months ago

React Server Components. Let that sink in. What did they do to us? This is a thing that should not exist, causing pain in the real world.

The_President4• 8 months ago

Like a turkey made of tofu. It’s all in the name: react.

testemailfordg2• 8 months ago

Looks like this post somehow is not even on the front page of HN anymore. CF pulling some strings maybe, they don't have this incident on top of their current list.

sega_sai• 8 months ago

After reading the post, my personal takeaway (not being expert) is that there are simply so many moving parts/configuration options that the complexity of the whole system is too high. I think without some sort of formal validation/enumeration of all possible states of the system the reliability is simply not possible. Whether this formal verification is achievable, I don't know.

DirkH• 8 months ago

I feel like all the BS we were taught about architecture design principles multi-AZ, failover strategies, graceful degradation etc was gaslighting us all into thinking any of out work on it actually matters.

This isn't true, but it feels like this when the entire engineering world order seems to actually run on single-point-of-failures where one CEO just messages another when some 3rd party is down. And reputational risk here is completely safeguarded because as long as everyone is down you are fine. Use a service everyone uses and it goes down = no reputational risk. Use a more robust architecture and make some mistake = massive reputational risk and everyone asks why you don't use what everyone else uses.

Blind leading the blind and all that.

lousken• 8 months ago

Cloudflare 362

Cloudflare was down

Related Stories