Cloudflare outage on November 18, 2025 post mortem

https://blog.cloudflare.com/18-november-2025-outage/

by eastdakota • 8 months ago

1465 893 8 months ago

blog.cloudflare.com

Related: Cloudflare Global Network experiencing issues - https://news.ycombinator.com/item?id=45963780 - Nov 2025 (1580 comments)

nawgz• 8 months ago

a change to one of our database systems' permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system ... to keep [that] system up to date with ever changing threats
The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail

A configuration error can cause internet-scale outages. What an era we live in

Edit: also, after finishing my reading, I have to express some surprise that this type of error wasn't caught in a staging environment. If the entire error is that "during migration of ClickHouse nodes, the migration -> query -> configuration file pipeline caused configuration files to become illegally large", it seems intuitive to me that doing this same migration in staging would have identified this exact error, no?

I'm not big on distributed systems by any means, so maybe I'm overly naive, but frankly posting a faulty Rust code snippet that was unwrapping an error value without checking for the error didn't inspire confidence for me!

jmclnx• 8 months ago

I have to wonder if AI was involved with the change.

norskeld2• 8 months ago

I don't think this is the case with CloudFlare, but for every recent GitHub outage or performance issue... oh boy, I blame the clankers!

mewpmewp25• 8 months ago

It would have been caught only in stage if there was similar amount of data in the database. If stage has 2x less data it would have never occurred there. Not super clear how easy it would have been to keep stage database exactly as production database in terms of quantity and similarity of data etc.

I think it's quite rare for any company to have exact similar scale and size of storage in stage as in prod.

Aeolun5• 8 months ago

I think it's quite rare for any company to have exact similar scale and size of storage in stage as in prod.

We’re like a millionth the size of cloudflare and we have automated tests for all (sort of) queries to see what would happen with 20x more data.

Mostly to catch performance regressions, but it would work to catch these issues too.

I guess that doesn’t say anything about how rare it is, because this is also the first company at which I get the time to go to such lengths.

norskeld2• 8 months ago

This wild `unwrap()` kinda took me aback as well. Someone really believed in themselves writing this. :)

Jach• 8 months ago

They only recently rewrote their core in Rust (https://blog.cloudflare.com/20-percent-internet-upgrade/) -- given the newness of the system and things like "Over 100 engineers have worked on FL2, and we have over 130 modules" I won't be surprised for further similar incidents.

shoo• 8 months ago

The speed and transparency of Cloudflare publishing this port mortem is excellent.

I also found the "remediation and follow up" section a bit lacking, not mentioning how, in general, regressions in query results caused by DB changes could be caught in future before they get widely rolled out.

Even if a staging env didn't have a production-like volume of data to trigger the same failure mode of a bot management system crash, there's also an opportunity to detect that something has gone awry if there were tests that the queries were returning functionally equivalent results after the proposed permission change. A dummy dataset containing a single http_requests_features column would suffice to trigger the dupe results behaviour.

In theory there's a few general ways this kind of issue could be detected, e.g. someone or something doing a before/after comparison to test that the DB permission change did not regress query results for common DB queries, for changes that are expected to not cause functional changes in behaviour.

Maybe it could have been detected with an automated test suite of the form "spin up a new DB, populate it with some curated toy dataset, then run a suite of important queries we must support and check the results are still equivalent (after normalising row order etc) to known good golden outputs". This style of regression testing is brittle, burdensome to maintain and error prone when you need to make functional changes and update what then "golden" outputs are - but it can give a pretty high probability of detecting that a DB change has caused unplanned functional regressions in query output, and you can find out about this in a dev environment or CI before a proposed DB change goes anywhere near production.

binarymax• 8 months ago

28M 500 errors/sec for several hours from a single provider. Must be a new record.

No other time in history has one single company been responsible for so much commerce and traffic. I wonder what some outage analogs to the pre-internet ages would be.

captainkrtek2• 8 months ago

Something like a major telco going out, for example the AT&T 1990 outage of long distance calling:

The standard procedures the managers tried first failed to bring the network back up to speed and for nine hours, while engineers raced to stabilize the network, almost 50% of the calls placed through AT&T failed to go through.
Until 11:30pm, when network loads were low enough to allow the system to stabilize, AT&T alone lost more than $60 million in unconnected calls.
Still unknown is the amount of business lost by airline reservations systems, hotels, rental car agencies and other businesses that relied on the telephone network.

https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collap...

adventured• 8 months ago

No other time in history has one single company been responsible for so much commerce and traffic.

AWS very likely has Cloudflare beat in commerce responsibility. Amazon is equal to ~2.3% of US GDP by itself.

manquer• 8 months ago

Absolute volume maybe[1], as relative % of global digital communication traffic, the era of early telegraph probably has it beat.

In the pre digital era, East India Company dwarfs every other company in any metric like commerce controlled, global shipping, communication traffic, private army size, %GDP , % of workforce employed by considerable margins.

The default was large consolidated organization throughout history, like say Bell Labs, or Standard Oil before that and so on, only for a brief periods we have enjoyed benefits of true capitalism.

[1] Although I suspect either AWS or MS/Azure recent down-times in the last couple of years are likely higher

nullbyte8082• 8 months ago

Yes, all(most) eggs should not be in one basket. Perfect opportunity to setup a service that checks cloudflare then switches a site's DNS to akami as a backup.

alhirzel2• 8 months ago

I wonder what some outage analogs to the pre-internet ages would be.

Lots of things have the sky in common. Maybe comet-induced ice ages...

0xbadcafebee2• 8 months ago

So, to recap:

  - Their database permissions changed unexpectedly (??)
  - This caused a 'feature file' to be changed in an unusual way (?!)
     - Their SQL query made assumptions about the database; their permissions change thus resulted in queries getting additional results, permitted by the query
  - Changes were propagated to production servers which then crashed those servers (meaning they weren't tested correctly)
     - They hit an internal application memory limit and that just... crashed the app
  - The crashing did not result in an automatic backout of the change, meaning their deployments aren't blue/green or progressive
  - After fixing it, they were vulnerable to a thundering herd problem
  - Customers who were not using bot rules were not affected; CloudFlare's bot-scorer generated a constant bot score of 0, meaning all traffic is bots

In terms of preventing this from a software engineering perspective, they made assumptions about how their database queries work (and didn't validate the results), and they ignored their own application limits and didn't program in either a test for whether an input would hit a limit, or some kind of alarm to notify the engineers of the source of the problem.

From an operations perspective, it would appear they didn't test this on a non-production system mimicing production; they then didn't have a progressive deployment; and they didn't have a circuit breaker to stop the deployment or roll-back when a newly deployed app started crashing.

tptacek22• 8 months ago

People jump to say things like "where's the rollback" and, like, probably yeah, but keep in mind that speculative rollback features (that is: rollbacks built before you've experienced the real error modes of the system) are themselves sources of sometimes-metastable distributed system failures. None of this is easy.

0xbadcafebee2• 8 months ago

How about where's the most basic test to check if your config file will actually run at all in your application? It was a hard-coded memory limit; a git-hook test suite run a MacBook would have caught this. But nooo, let's not run the app for 0.01 seconds with this config before sending it out to determine the fate of the internet?

This is literally the CrowdStrike bug, in a CDN. This is the most basic, elementary, day 0 test you could possibly invent. Forget the other things they fucked up. Their app just crashes with a config file, and nobody evaluates it?! Not every bug is preventable, but an egregious lack of testing is preventable.

This is what a software building code (like the electrical code's UL listings that prevent your house from burning down from untested electrical components) is intended to prevent. No critical infrastructure should be legal without testing, period.

paulddraper2• 8 months ago

Looks like you have the perfect window to disrupt them with a superior product.

mercnz2• 8 months ago

just before this outage i was exploring bunnycdn as the idea of cloudflare taking over dns still irks me slightly. there are competitors. but there's a certain amount of scale that cloudflare offers which i think can help performance in general. that said in the past i found cloudflare performance terrible when i was doings lots of testing. they are predominantly a pull based system not a push, so if content isn't current the cache miss performance can be kind of blah. i think their general backhaul paths have improved, but at least from new zealand they used to seem to do worse than hitting a los angeles proxy that then hits origin. (although google was in a similar position before, where both 8.8.8.8 and www.google.co.nz/.com were both faster via los angeles than via normal paths - i think google were doing asia parent, like if testing 8.8.8.8 misses it was super far away). i think now that we have http/3 etc though that performance is a bit simpler to achieve, and that ddos, bot protection is kind of the differentiator, and i think that cloudflare's bot protection may work reasonably well in general?

rawgabbit2• 8 months ago

     > The change explained above resulted in all users accessing accurate metadata about tables they have access to. Unfortunately, there were assumptions made in the past, that the list of columns returned by a query like this would only include the “default” database:

  SELECT
  name,
  type
  FROM system.columns
  WHERE
  table =        'http_requests_features'
  order by name;

    Note how the query does not filter for the database name. With us gradually rolling out the explicit grants to users of a given ClickHouse cluster, after the change at 11:05 the query above started returning “duplicates” of columns because those were for underlying tables stored in the r0 database.

rawgabbit2• 8 months ago

Here is a bit more context in addition to the quote above. A ClickHouse permissions change made a metadata query start returning duplicate column metadata from an extra schema, which more than doubled the size and feature count of a Bot Management configuration file. When this oversized feature file was deployed to edge proxies, it exceeded a 200-feature limit in the bot module, causing that module to panic and the core proxy to return 5xx errors globally

zzzeek• 8 months ago

Instead, it was triggered by a change to one of our database systems' permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system.

And here is the query they used ** (OK, so it's not exactly):

     SELECT * from feature JOIN permissions on feature.feature_type_id = permissions.feature_type_id

someone added a new row to permissions and the JOIN started returning two dupe feature rows for each distinct feature.

** "here is the query" is used for dramatic effect. I have no knowledge of what kind of database they are even using much less queries (but i do have an idea).

more edits: OK apparently it's described later in the post as a query against clickhouse's table metadata table, and because users were granted access to an additional database that was actually the backing store to the one they normally worked with, some row level security type of thing doubled up the rows. Not sure why querying system.columns is part of a production level query though, seems overly dynamic.

captainkrtek2• 8 months ago

I believe they mentioned ClickHouse

SerCe• 8 months ago

As always, kudos for releasing a post mortem in less than 24 hours after the outage, very few tech organisations are capable of doing this.

bayesnet• 8 months ago

And a well-written one at that. Compared to the AWS port-mortem this could be literature.

philipwhiuk5• 8 months ago

Except it fails to document anything about the actions they made to Warp in London during the resolution.

yen223• 8 months ago

I'm curious about how their internal policies work such that they are allowed to publish a post mortem this quickly, and with this much transparency.

Any other large-ish company, there would be layers of "stakeholders" that will slow this process down. They will almost always never allow code to be published.

thesh4d0w• 8 months ago

The person who posted both this blog article and the hacker news post, is Matthew Prince, one of highly technical billionaire founders of cloudflare. I'm sure if he wants something to happen, it happens.

tom1337• 8 months ago

I mean the CEO posted the post-mortem so there aren't that many layers of stakeholders above. For other post-mortems by engineers, Matthew once said that the engineering team is running the blog and that he wouldn't event know how to veto even if he wanted [0]

[0] https://news.ycombinator.com/item?id=45588305

madeofpalk• 8 months ago

From what I've observed, it depends on whether you're an "engineering company" or not.

eastdakota11• 8 months ago

Well… we have a culture of transparency we take seriously. I spent 3 years in law school that many times over my career have seemed like wastes but days like today prove useful. I was in the triage video bridge call nearly the whole time. Spent some time after we got things under control talking to customers. Then went home. I’m currently in Lisbon at our EUHQ. I texted John Graham-Cumming, our former CTO and current Board member whose clarity of writing I’ve always admired. He came over. Brought his son (“to show that work isn’t always fun”). Our Chief Legal Officer (Doug) happened to be in town. He came over too. The team had put together a technical doc with all the details. A tick-tock of what had happened and when. I locked myself on a balcony and started writing the intro and conclusion in my trusty BBEdit text editor. John started working on the technical middle. Doug provided edits here and there on places we weren’t clear. At some point John ordered sushi but from a place with limited delivery selection options, and I’m allergic to shellfish, so I ordered a burrito. The team continued to flesh out what happened. As we’d write we’d discover questions: how could a database permission change impact query results? Why were we making a permission change in the first place? We asked in the Google Doc. Answers came back. A few hours ago we declared it done. I read it top-to-bottom out loud for Doug, John, and John’s son. None of us were happy — we were embarrassed by what had happened — but we declared it true and accurate. I sent a draft to Michelle, who’s in SF. The technical teams gave it a once over. Our social media team staged it to our blog. I texted John to see if he wanted to post it to HN. He didn’t reply after a few minutes so I did. That was the process.

BrtByte5• 8 months ago

Cloudflare seems to have baked this level of transparency into their culture and incident response process

andrewinardeer• 8 months ago

Plenty are capable. Most don't bother.

BrtByte5• 8 months ago

It's not just a PR-friendly summary either... they included real technical detail, timestamps, even code snippets

eastdakota11• 8 months ago

* published less than 12 hours from when the incident began. Proud of the team for pulling together everything so quickly and clearly.

ignoramous2• 8 months ago

That's all well & good, but I'm curious...

Spent some time after we got things under control talking to customers. Then went home.

What did sama / Fidji say? ;) Turnstile couldn't have been worth that.

gucci-on-fleek2• 8 months ago

This showed up to Internet users trying to access our customers' sites as an error page indicating a failure within Cloudflare's network.

As a visitor to random web pages, I definitely appreciated this—much better than their completely false “checking the security of your connection” message.

The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was triggered by a change to one of our database systems' permissions

Also appreciate the honesty here.

On 18 November 2025 at 11:20 UTC (all times in this blog are UTC), Cloudflare's network began experiencing significant failures to deliver core network traffic. […]
Core traffic was largely flowing as normal by 14:30. We worked over the next few hours to mitigate increased load on various parts of our network as traffic rushed back online. As of 17:06 all systems at Cloudflare were functioning as normal.

Why did this take so long to resolve? I read through the entire article, and I understand why the outage happened, but when most of the network goes down, why wasn't the first step to revert any recent configuration changes, even ones that seem unrelated to the outage? (Or did I just misread something and this was explained somewhere?)

Of course, the correct solution is always obvious in retrospect, and it's impressive that it only took 7 minutes between the start of the outage and the incident being investigated, but it taking a further 4 hours to resolve the problem and 8 hours total for everything to be back to normal isn't great.

eastdakota11• 8 months ago

Because we initially thought it was an attack. And then when we figured it out we didn’t have a way to insert a good file into the queue. And then we needed to reboot processes on (a lot) of machines worldwide to get them to flush their bad files.

tptacek22• 8 months ago

Richard Cook #18 (and #10) strikes again!

https://how.complexsystems.fail/#18

It'd be fun to read more about how you all procedurally respond to this (but maybe this is just a fixation of mine lately). Like are you tabletopping this scenario, are teams building out runbooks for how to quickly resolve this, what's the balancing test for "this needs a functional change to how our distributed systems work" vs. "instead of layering additional complexity on, we should just have a process for quickly and maybe even speculatively restoring this part of the system to a known good state in an outage".

tetec1• 8 months ago

Yeah, I can imagine that this insertion was some high-pressure job.

gucci-on-fleek2• 8 months ago

Thanks for the explanation! This definitely reminds me of CrowdStrike outages last year:

- A product depends on frequent configuration updates to defend against attackers.

- A bad data file is pushed into production.

- The system is unable to easily/automatically recover from bad data files.

(The CrowdStrike outages were quite a bit worse though, since it took down the entire computer and remediation required manual intervention on thousands of desktops, whereas parts of Cloudflare were still usable throughout the outage and the issue was 100% resolved in a few hours)

dbetteridge• 8 months ago

Question from a casual bystander, why not have a virtual/staging mini node that receives these feature file changes first and catches errors to veto full production push?

Or you do have something like this but the specific db permission change in this context only failed in production

hbbio• 8 months ago

Thx for the explanation!

Side thought as we're working on 100% onchain systems (for digital assets security, different goals):

Public chains (e.g. EVMs) can be a tamper‑evident gate that only promotes a new config artifact if (a) a delay or multi‑sig review has elapsed, and (b) a succinct proof shows the artifact satisfies safety invariants like ≤200 features, deduped, schema X, etc.

That could have blocked propagation of the oversized file long before it reached the edge :)

philipwhiuk5• 8 months ago

Why was Warp in London disabled temporarily. No mention of that change was discussed in the RCA despite it being called out in an update.

For London customers this made the impact more severe temporarily.

prawn• 8 months ago

Just asking out of curiosity, but roughly how many staff would've been involved in some way in sorting out the issue? Either outside regular hours or redirected from their planned work?

mindentropy• 8 months ago

Is there some way to check the sanity of the configuration change, monitor it and then revert back to an earlier working configuration if things don't work out?

chrismorgan• 8 months ago

much better than their completely false “checking the security of your connection” message

The exact wording (which I can easily find, because a good chunk of the internet gives it to me, because I’m on Indian broadband):

example.com needs to review the security of your connection before proceeding.

It bothers me how this bald-faced lie of a wording has persisted.

(The “Verify you are human by completing the action below.” / “Verify you are human” checkbox is also pretty false, as ticking the box in no way verifies you are human, but that feels slightly less disingenuous.)

eastdakota11• 8 months ago

Next time open your dev console in your window and look at how much is going on in the background.

jpadkins• 8 months ago

checking the checkbox does verify you are human, for the most part.

EvanAnderson4• 8 months ago

It reads a lot like the Crowdstrike SNAFU. Machine-generated configuration file b0rks-up the software that consumes it.

The "...was then propagated to all the machines that make up our network..." followed by "....caused the software to fail." screams for a phased rollout / rollback methodology. I get that "...it’s critical that it is rolled out frequently and rapidly as bad actors change their tactics quickly" but today's outage highlights that rapid deployment isn't all upside.

The remediation section doesn't give me any sense that phased deployment, acceptance testing, and rapid rollback are part of the planned remediation strategy.

tptacek22• 8 months ago

I don't think this system is best thought of as "deployment" in the sense of CI/CD; it's a control channel for a distributed bot detection system that (apparently) happens to be actuated by published config files (it has a consul-template vibe to it, though I don't know if that's what it is).

EvanAnderson4• 8 months ago

That's why I likened it Crowdstrike. It's a signature database that blew up the consumer of said database. (You probably caught my post mid-edit, too. You may be replying to the snarky paragraph I felt better of and removed.)

Edit: Similar to Crowdstrike, the bot detector should have fallen-back to its last-known-good signature database after panicking, instead of just continuing to panic.

eastdakota11• 8 months ago

That’s correct.

JB_Dev2• 8 months ago

Code and Config should be treated similarly. If you would use a ring based rollout, canaries, etc for safely changing your code, then any config that can have the same impact must also use safe rollout techniques.

Aeolun5• 8 months ago

I’m fairly certain it will be after they read this thread. It doesn’t feel like they don’t want, or are incapable of improving?

navigate8310• 8 months ago

I'm amazed that they are not using any simulator of some sort and pushing changes directly to production.

perlgeek• 8 months ago

At my employer, we have a small script that automatically checks such generated config files. It does a diff between the old and the new version, and if the diff size exceeds a threshold (either total or relative to the size of the old file), it refuses to do the update, and opens a ticket for a human to look over it.

It has somewhat regularly saved us from disaster in the past.

tristan-morris3• 8 months ago

Why call .unwrap() in a function which returns Result<_,_>?

For something so critical, why aren't you using lints to identify and ideally deny panic inducing code. This is one of the biggest strengths of using Rust in the first place for this problem domain.

sayrer• 8 months ago

Yes, can't have .unwrap() in production code (it's ok in tests)

orphea2• 8 months ago

Like goto, unwrap is just a tool that has its use cases. No need to make a boogeyman out of it.

keyle• 8 months ago

unwrap itself isn't the problem...

tptacek22• 8 months ago

Probably because this case was something more akin to an assert than an error check.

stefan_• 8 months ago

You are saying this would not have happened in a C release build where asserts define to nothing?

Wonder why these old grey beards chose to go with that.

tristan-morris3• 8 months ago

Oh absolutely, that's how it would have been treated.

Surely a unwrap_or_default() would have been a much better fit--if fetching features fails, continue processing with an empty set of rules vs stop world.

marcusb• 8 months ago

Rust has debug asserts for that. Using expect with a comment about why the condition should not/can't ever happen is idiomatic for cases where you never expect an Err.

This reads to me more like the error type returned by append with names is not (ErrorFlags, i32) and wasn't trivially convertible into that type so someone left an unwrap in place on an "I'll fix it later" basis, but who knows.

thundergolfer2• 8 months ago

Fly writes a lot of Rust, do you allow `unwrap()` in your production environment? At Modal we only allow `expect("...")` and the message should follow the recommended message style[1].

I'm pretty surprised that Cloudflare let an unwrap into prod that caused their worst outage in 6 years.

1. https://doc.rust-lang.org/std/option/enum.Option.html#recomm...

piperswe• 8 months ago

In that case, it should probably be expect rather than unwrap, to document why the assertion should never fail.

koakuma-chan• 8 months ago

Why is there a 200 limit on appending names?

nickmonad• 8 months ago

Limits in systems like these are generally good. They mention the reasoning around it explicitly. It just seems like the handling of that limit is what failed and was missed in review.

zmj• 8 months ago

Everything has a limit. You can define it, or be surprised when you find out what it is.

otterley13• 8 months ago

work has already begun on how we will harden them against failures like this in the future. In particular we are:
Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input
Enabling more global kill switches for features
Eliminating the ability for core dumps or other error reports to overwhelm system resources
Reviewing failure modes for error conditions across all core proxy modules

Absent from this list are canary deployments and incremental or wave-based deployment of configuration files (which are often as dangerous as code changes) across fault isolation boundaries -- assuming CloudFlare has such boundaries at all. How are they going to contain the blast radius in the future?

This is something the industry was supposed to learn from the CrowdStrike incident last year, but it's clear that we still have a long way to go.

Also, enabling global anything (i.e., "enabling global kill switches for features") sounds like an incredibly risky idea. One can imagine a bug in a global switch that transforms disabling a feature into disabling an entire system.

nikcub• 8 months ago

They require the bot management config to update and propagate quickly in order to respond to attacks - but this seems like a case where updating a since instance first would have seen the panic and stopped the deploy.

I wonder why clickhouse is used to store the feature flags here, as it has it's own duplication footguns[0] which could have also easily lead to a query blowing up 2/3x in size. oltp/sqlite seems more suited, but i'm sure they have their reasons

[0] https://clickhouse.com/docs/guides/developer/deduplication

HumanOstrich5• 8 months ago

I don't think sqlite would come close to their requirements for permissions or resilience, to name a couple. It's not the solution for every database issue.

Also, the link you provided is for eventual deduplication at the storage layer, not deduplication at query time.

mewpmewp25• 8 months ago

It seems they had this continous rollout for the config service, but the services consuming this were affected even by small percentage of these config providers being faulty, since they were auto updating every few minutes their configs. And it seems there is a reason for these updating so fast, presumably having to react to threat actors quickly.

otterley13• 8 months ago

It's in everyone's interest to mitigate threats as quickly as possible. But it's of even greater interest that a core global network infrastructure service provider not DOS a significant proportion of the Internet by propagating a bad configuration too quickly. The key here is to balance responsiveness against safety, and I'm not sure they struck the right balance here. I'm just glad that the impact wasn't as long and as severe as it could have been.

Scaevolus• 8 months ago

Global configuration is useful for low response times to attacks, but you need to have very good ways to know when a global config push is bad and to be able to rollback quickly.

In this case, the older proxy's "fail-closed" categorization of bot activity was obviously better than the "fail-crash", but every global change needs to be carefully validated to have good characteristics here.

Having a mapping of which services are downstream of which other service configs and versions would make detecting global incidents much easier too, by making the causative threads of changes more apparent to the investigators.

ants_everywhere• 8 months ago

it's always a config push. people rollout code slowly but don't have the same mechanisms for configs. But configs are code, and this is a blind spot that causes an outsized percentage of these big outages.

Buttons8402• 8 months ago

When a failsafe system fails, it fails by failing to fail safely.

Yokohiii6• 8 months ago

I think global kill switches are just an last resort machanism, to bypass identified faulty subsystems. Even if there is a risk with it, in this instance the risk was zero, because CF was dead already. This wont change the blast radius, but it's duration and proliferation.

In reference to fault isolation boundaries: I am not familiar with their CI/CD, in theory the error could have been caught/prevented there, but that comes with a lot of depends or it's tricky. But it looks like they didn't go the extra mile to care about safety sensitive areas. So euphemistic speaking, they are now recalibrating balance of safety measures.

lukan• 8 months ago

"Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare. While it turned out to be a coincidence, it led some of the team diagnosing the issue to believe that an attacker may be targeting both our systems as well as our status page."

Unfortunately they do not share, what caused the status page to went down as well. (Does this happen often? Otherwise a big coincidence it seems)

Aeolun5• 8 months ago

I mean, that would require a postmortem from statuspage.io right? Is that a service operated by cloudflare?

edoceo• 8 months ago

Atlasaian

eastdakota11• 8 months ago

We don’t know. Suspect it may just have been a big uptick in load and a failure of its underlying infrastructure to scale up.

dnw• 8 months ago

Yes, probably a bunch of automated bots decided to check the status page when they saw failures in production.

reassess_blind2• 8 months ago

The status page is hosted on AWS Cloudfront, right? It sure looks like Cloudfront was overwhelmed by the traffic spike, which is a bit concerning. Hope we'll see a post from their side.

jbmsf• 8 months ago

It looks a lot like a CloudFront error we randomly saw today from one of our engineers in South America. I suspect there was a small outage in AWS but can't prove it.

francisofascii• 8 months ago

This situation reminds me of risk assessment, where you sometimes assume two rare events are independent, but later learn they are actually highly correlated.

notatoad2• 8 months ago

it seems like a good chance that despite thinking their status page was completely independent of cloudfront, enough of the internet is dependent on cloudfront now that they're simply wrong about the status page's independence.

verletzen• 8 months ago

i think you've got cloudflare and cloudfront mixed up.

paulddraper2• 8 months ago

Quite possibly it was due to high traffic.

IDK Atlassian Statuspage clientele, but it's possible Cloudflare is much larger than usual.

vsgherzi2• 8 months ago

Why does cloudflare allow unwraps in their code? I would've assumed they'd have clippy lints stopping that sort of thing. Why not just match with { ok(value) => {}, Err(error) => {} } the function already has a Result type.

At the bare minimum they could've used an expect("this should never happen, if it does database schema is incorrect").

The whole point of errors as values is preventing this kind of thing.... It wouldn't have stopped the outage but it would've made it easy to diagnose.

If anyone at cloudflare is here please let me in that codebase :)

waterTanuki• 8 months ago

Not a cloudflare employee but I do write a lot of Rust. The amount of things that can go wrong with any code that needs to make a network call is staggeringly high. unwrap() is normal during development phase but there are a number of times I leave an expect() for production because sometimes there's no way to move forward.

vsgherzi2• 8 months ago

I'm in a similar boat, at the very leas an expect can give hits to what happened. However this can also be problematic if your a library developer. Sometimes rust is expected to never panic especially in situations like WASM. This is a major problem for companies like Amazon Prime Video since they run in a WASM context for their TV APP. Any panic crashes everything. Personally I usually just either create a custom error type (preferred) or erase it away with Dyn Box Error (no other option). Random unwraps and expects haunt my dreams.

SchemaLoad4• 8 months ago

Yeah it seems likely that even if there wasn't an unwrap, there would have been some error handling that wouldn't have panicked the process, but would have still left it inoperable if every request was instead going through an error path.

frumplestlatz19• 8 months ago

At risk of sounding harsh, that’s a huge failure in your modeling of invariants that should not be permitted in development.

Permitting it in development is why one ends up in the position of having to use an `expect()` in production code, because your API surfaces are wrong and can’t model your actual invariants.

ozgrakkurt3• 8 months ago

And the error magically disappears when the function returns it?

csomar3• 8 months ago

It doesn’t disappear, it forces you to handle it.

pornel2• 8 months ago

unwrap() is only the most superficial part of the problem. Merely replacing `unwrap()` with `return Err(code)` wouldn't have changed the behavior. Instead of "error 500 due to panic" the proxy would fail with "error 500 due to $code".

Unwrap gives you a stack trace, while retuned Err doesn't, so simply using a Result for that line of code could have been even harder to diagnose.

`unwrap_or_default()` or other ways of silently eating the error would be less catastrophic immediately, but could still end up breaking the system down the line, and likely make it harder to trace the problem to the root cause.

The problem is deeper than an unwrap(), related to handling rollouts of invalid configurations, but that's not a 1-line change.

frumplestlatz19• 8 months ago

We don't know what the surrounding code looks like, but I'd expect it handles the error case that's expressed in the type signature (unless they `.unwrap()` there too).

The problem is that they didn't surface a failure case, which means they couldn't handle rollouts of invalid configurations correctly.

The use of `.unwrap()` isn't superficial at all -- it hid an invariant that should have been handled above this code. The failure to correctly account for and handle those true invariants is exactly what caused this failure mode.

ed_mercer• 8 months ago

Wow. 26M/s 5xx error HTTP status codes over a span of roughly two hours. That's roughly 187 billion HTTP errors that interrupted people (and systems)!

watchful_moose4• 8 months ago

Some of these would be retries that wouldn't have happened if not for earlier errors.

moralestapia• 8 months ago

No publicity is bad publicity.

Best post mortem I've read in a while, this thing will be studied for years.

A bit ironic that their internal FL2 tool is supposed to make Cloudflare "faster and more secure" but brought a lot of things down. And yeah, as other have already pointed out, that's a very unsafe use of Rust, should've never made it to production.

sigmar• 8 months ago

Wow. What a post mortem. Rather than Monday morning quarterbacking how many ways this could have been prevented, I'd love to hear people sound-off on things that unexpectedly broke. I, for one, did not realize logging in to porkbun to edit DNS settings would become impossible with a cloudflare meltdown

brandon272• 8 months ago

That's unfortunate. I'll need to investigate whether Porkbun plans on decoupling its auth from being reliant on CloudFlare, otherwise I will need to migrate a few domains off of that registrar.

ojosilva• 8 months ago

This is the multi-million dollar .unwrap() story. In a critical path of infrastructure serving a significant chunk of the internet, calling .unwrap() on a Result means you're saying "this can never fail, and if it does, crash the thread immediately."The Rust compiler forced them to acknowledge this could fail (that's what Result is for), but they explicitly chose to panic instead of handle it gracefully. This is textbook "parse, don't validate" anti-pattern.

I know, this is "Monday morning quarterbacking", but that's what you get for an outage this big that had me tied up for half a day.

wrs10• 8 months ago

It seems people have a blind spot for unwrap, perhaps because it's so often used in example code. In production code an unwrap or expect should be reviewed exactly like a panic.

It's not necessarily invalid to use unwrap in production code if you would just call panic anyway. But just like every unsafe block needs a SAFETY comment, every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used can enforce this.

dist1ll5• 8 months ago

every unwrap in production code needs an INFALLIBILITY comment. clippy::unwrap_used can enforce this.

How about indexing into a slice/map/vec? Should every `foo[i]` have an infallibility comment? Because they're essentially `get(i).unwrap()`.

dehrmann3• 8 months ago

It's the same blind spot people have to Java's checked exceptions. People commonly resort to Pokemon exception handling and either blindly ignoring or rethrowing as a runtime exception. When Rust got popular, I was a bit confused by people talking about how great Result it's essentially a checked exception without a stack trace.

speed_spread• 8 months ago

Pet peeve: unwrap() should be deprecated and renamed or_panic(). More consistent with the rest of stdlib methods and appropriately scarier.

_zagj• 8 months ago

It seems people have a blind spot for unwrap

Not unlike people having a blind spot for Rust in general, no?

littlestymaar3• 8 months ago

In production code an unwrap or expect should be reviewed exactly like a panic.

An unwrap should never make it to production IMHO. It's fine while prototyping, but once the project gets closer to production it's necessary to just grep `uncheck` in your code and replace those that can happen with a proper error management and replace those that cannot happen with `expect`, with a clear justification of why they cannot happen unless there's a bug somewhere else.

bombela5• 8 months ago

This thread warms my heart. Rust has set a new baseline that many and myself now take for granted.

We are now discussing what can be done to improve code correctness beyond memory and thread safety. I am excited for what is to come.

brabel• 8 months ago

Yes, I always thought it was wrong to use unwrap in examples. I know, people want to keep examples simple, but it trains developers to use unwrap() as they see that everywhere. Yes, there are places where it's ok as that blog post explains so well: https://burntsushi.net/unwrap/ But most devs IMHO don't have the time to make the call correctly most of the time... so it's just better to do something better, like handle the error and try to recover, or if impossible, at least do `expect("damn it, how did this happen")`.

quotemstr5• 8 months ago

people have a blind spot for unwrap

It's not about whether you should ban unwrap() in production. You shouldn't. Some errors are logic bugs beyond which a program can't reasonably continue. The problem is that the language makes it too easy for junior developers (and AI!) to ignore non-logic-bug problems with unwrap().

Programmers early in their careers will do practically anything to avoid having to think about errors and they get angry when you tell them about it.

mamcx• 8 months ago

The problem here is that for a service "fail fast" is mostly the best default.

arccy• 8 months ago

if you make it easy to be lazy and panic vs properly handling the error, you've designed a poor language

otterley13• 8 months ago

https://en.wikipedia.org/wiki/Crash-only_software

yoyohello137• 8 months ago

So… basically every language ever?

Except maybe Haskell.

SchwKatze2• 8 months ago

Unwrap isn't a synonym for laziness, it's just like an assertion, when you do unwrap() you're saying the Result should NEVER fail, and if does, it should abort the whole process. What was wrong was the developer assumption, not the use of unwrap.

JuniperMesos8• 8 months ago

It's a little subtler than this. You want it to be easy to not handle an error while developing, so you can focus on getting the core logic correct before error-handling; but you want it to be hard to deploy or release the software without fully handling these checks. Some kind of debug vs release mode with different lints seems like a reasonable approach.

leshenka• 8 months ago

All languages with few exceptions have these kinds of escape hatches like unwrap

nine_k3• 8 months ago

At Facebook they name certain "escape hatch" functions in a way that inescapably make them look like a GIANT EYESORE. Stuff like DANGEROUSLY_CAST_THIS_TO_THAT, or INVOKE_SUPER_EXPENSIVE_ACTION_SEE_YOU_ON_CODE_REVIEW. This really drives home the point that such things must not be used except in rare extraordinary cases.

If unwrap() were named UNWRAP_OR_PANIC(), it would be used much less glibly. Even more, I wish there existed a super strict mode when all places that can panic are treated as compile-time errors, except those specifically wrapped in some may_panic_intentionally!() or similar.

kibwen5• 8 months ago

In Rust, `.unwrap()` is nine characters, whereas propagating the Result via `?` is one.

selfmodruntime6• 8 months ago

This is untrue. A `?` operator would have done just fine here. I agree with you though that it should be possible to explicitely forbid unwraps.

cvhc3• 8 months ago

Some languages and style guides simply forbid throwing exceptions without catching / proper recovery. Google C++ bans exceptions and the main mechanism for propogating errors is `absl::Status` which the caller has to check. Not familiar with Rust but it seems unwrap is such a thing that would be banned.

pdimitar6• 8 months ago

There are even lints for this but people get impatient and just override them or fight for them to no longer be the default.

As usual: people problem, not a tech problem. In the last years a lot of strides have been made. But people will be people.

gpm2• 8 months ago

Unwrap is used in places where in C++ you would just have undefined behavior. It wouldn't make any more sense to blanket ban it than it would ban ever dereferencing a pointer just in case its null - even if you just checked that it wasn't null.

kibwen5• 8 months ago

Not familiar with Rust but it seems unwrap is such a thing that would be banned.

Panics aren't exceptions, any "panic" in Rust can be thought of as an abort of the process (Rust binaries have the explicit option to implement panics as aborts). Companies like Dropbox do exactly this in their similar Rust-based systems, so it wouldn't surprise me if Cloudflare does the same.

"Banning exceptions" wouldn't have done anything here, what you're looking for is "banning partial functions" (in the Haskell sense).

vlovich1235• 8 months ago

To be fair, this failed in the non-rust path too because the bot management returned that all traffic was a bot. But yes, FL2 needs to catch panics from individual components but I’m not sure if failing open is necessarily that much better (it was in this case but the next incident could easily be the result of failing open).

But more generally you could catch the panic at the FL2 layer to make that decision intentional - missing logic at that layer IMHO.

hedora5• 8 months ago

Catching panic probably isn’t a great idea if there’s any unsafe code in the system. (Do the unsafe blocks really maintain heap invariants if across panics?)

ajross• 8 months ago

I'm not completely sure I agree. I mean, I do agree about the .unwrap() culture being a bug trap. But I don't think this example qualifies.

The root cause here was that a file was mildly corrupt (with duplicate entries, I guess). And there was a validation check elsewhere that said "THIS FILE IS TOO BIG".

But if that's a validation failure, well, failing is correct? What wasn't correct was that the failure reached production. What should have happened is that the validation should have been a unified thing and whatever generated the file should have flagged it before it entered production.

And that's not an issue with function return value API management. The software that should have bailed was somewhere else entirely, and even there an unwrap explosion (in a smoke test or pre-release pass or whatever) would have been fine.

crote• 8 months ago

It sounds to me like there was validation, but the system wasn't designed for the validation to ever fail - at which point crashing is the only remaining option. You've essentially turned it into an assertion error rather than a parsing/validation error.

Ideally every validation should have a well-defined failure path. In the case of a config file rotation, validation failure of the new config could mean keeping the old config and logging a high-priority error message. In the case of malformed user-provided data, it might mean dropping the request and maybe logging it for security analysis reasons. In the case of "pi suddenly equals 4" checks the most logical approach might be to intentionally crash, as there's obviously something seriously wrong and application state has corrupted in such a way that any attempt to continue is only going to make things worse.

But in all cases there's a reason behind the post-validation-failure behavior. At a certain point leaving it up to "whatever happens on .unwrap() failure" isn't good enough anymore.

ChrisMarshallNY5• 8 months ago

Swift has implicit unwrap (!), and explicit unwrap (?).

I don't like to use implicit unwrap. Even things that are guaranteed to be there, I treat as explicit (For example, (self.view?.isEnabled ?? false), in a view controller, instead of self.view.isEnabled).

I always redefine @IBOutlets from:

    @IBOutlet weak var someView!

to:

    @IBOutlet weak var someView?

I'm kind of a "belt & suspenders" type of guy.

monocularvision3• 8 months ago

So what happens if it ends up being nil? How does your app react?

In this particular case, I would rather crash. It’s easier to spot in a crash report and you get a nice stack trace.

Silent failure is ultimately terrible for users.

Note: for the things I control I try to very explicitly model state in such a way as I never need to force unwrap at all. But for things beyond my control like this situation, I would rather end the program than continue with a state of the world I don’t understand.

groundzeros20157• 8 months ago

This is terrible. The whole reason they introduced this is because IBOutlets would get silently disconnected and then in the field a user would complain that a feature stopped working.

Crash early, crash often. Find the bugs and bad assumptions.

shadowgovt2• 8 months ago

In addition, it looks like this system wasn't on any kind of 1%/10%/50%/100% rollout gating. Such a rollout would trivially have shown the poison input killing tasks.

penteract• 8 months ago

To me it reads like there was a gradual rollout of the faulty software responsible for generating the config files, but those files are generated on approximately one machine, then propogated across the whole network every 5 minutes.

Bad data was only generated if the query ran on a part of the cluster which had been updated. As a result, every five minutes there was a chance of either a good or a bad set of configuration files being generated and rapidly propagated across the network.

helloericsf• 8 months ago

Not a DBA, how do you do DB permission rollout gating?

smj-edison• 8 months ago

Isn't the point of this article that pieces of infrastructure don't go down to root causes, but due to bad combinations of components that are correct individually? After reading "engineering a safer world", I find root cause analysis rather reductionistic, because it wasn't just an unwrap, it was that the payload was larger than normal, because of a query that didn't select by database, because a clickhouse made more databases visible. Hard to say "it was just due to an unwrap" imo. Especially in terms of how to fix an issue going forwards. I think the article lists a lot of good ideas, that aren't just "don't unwrap", like enabling more global kill switches for features, or eliminating the ability for core dumps or other error reports to overwhelm system resources.

brianpan• 8 months ago

You're right. A good postmortem/root cause analysis would START from "unwrap" and continue from there.

You might start with a basic timeline of what happened, then you'd start exploring: why did this change affect so many customers (this would be a line of questioning to find a potential root cause), why did it take so long to discover or recover (this might be multiple lines of questioning), etc.

AgentME• 8 months ago

This is assuming that the process could have done anything sensible while it had the malformed feature file. It might be in this case that this was one configuration file of several and maybe the program could have been built to run with some defaults when it finds this specific configuration invalid, but in the general case, if a program expects a configuration file and can't do anything without it, panicking is a normal thing to do. There's no graceful handling (beyond a nice error message) a program like Nginx could do on a syntax error in its config.

The real issue is further up the chain where the malformed feature file got created and deployed without better checks.

aloha2436• 8 months ago

panicking is a normal thing to do

I do not think that if the bot detection model inside your big web proxy has a configuration error it should panic and kill the entire proxy and take 20% of the internet with it. This is a system that should fail gracefully and it didn't.

The real issue

Are there single "real issues" with systems this large? There are issues being created constantly (say, unwraps where there shouldn't be, assumptions about the consumers of the database schema) that only become apparent when they line up.

JeremyNT• 8 months ago

Exactly! Sometimes exploding is simply the least bad option, and is an entirely sensible approach.

WD-42• 8 months ago

Yea, Rust is safe but it’s not magic. However Nginx doesn’t panic on malformed config. It exits with hopefully a helpful error code and message. The question is then could the cloudflare code have exited cleanly in a way that made recovery easier instead of just straight panicking.

kondro• 8 months ago

One feature failing like this should probably log the error and fail closed. It shouldn't take down everything else in your big proxy that sits in front of your entire business.

bobbylarrybobby• 8 months ago

I don't know too much about how the feature file distribution works but in the event of failure to read a new file, wouldn't logging the failure and sticking with the previous version of the file be preferable?

diath2• 8 months ago

Falling back to a generic base configuration in the presence of an incoming invalid config file would probably be a sensible thing to do.

butvacuum• 8 months ago

It rang more as "A/B deployments are pointless if you can't tell if a downstream failure is related." To me.

nrhrjrjrjtntbt2• 8 months ago

I wonder what happens if they handle it gracefully? sounds like performance degradation (better than reliability degradation!).

Also wonder with a sharded system why are they not slow rolling out changes and monitoring?

ironman1478• 8 months ago

I'm not a fan of rust, but I don't think that is the only takeaway. All systems have assumptions about their input and if the assumption is violated, it has to be caught somewhere. It seems like it was caught too deep in the system.

Maybe the validation code should've handled the larger size, but also the db query produced something invalid. That shouldn't have ever happened in the first place.

asa4003• 8 months ago

It seems like it was caught too deep in the system.

Agreed, that's also my takeaway.

I don't see the problem being "lazy programmers shouldn't have called .unwrap()". That's reductive. This is a complex system and complex system failures aren't monocausal.

The function in question could have returned a smarter error rather than panicking, but what then? An invariant was violated, and maybe this system, at this layer, isn't equipped to take any reasonable action in response to that invariant violation and dying _is_ the correct thing to do.

But maybe it could take smarter action. Maybe it could be restarted into a known good state. Maybe this service could be supervised by another system that would have propagated its failure back to the source of the problem, alerting operators that a file was being generated in such a way that violated consumer invariants. Basically, I'm describing a more Erlang model of failure.

Regardless, a system like this should be able to tolerate (or at least correctly propagate) a panic in response to an invariant violation.

9rx2• 8 months ago

The takeaway here isn’t about Rust itself, but that the Rust marketing crew’s claims that we constantly read on HN and elsewhere about the Result type magically saving you from making mistakes is not a good message to send.

guluarte• 8 months ago

it's usually because of fail fast and fail hard, in theory critical bugs will be caught in dev/test

jcalvinowens2• 8 months ago

This is the multi-million dollar .unwrap() story.

That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

chrisweekly• 8 months ago

semantic? or pedantic?

navidmafi• 8 months ago

And dare I say, an exhibition of hindsight bias.

AdieuToLogic2• 8 months ago

> This is the multi-million dollar .unwrap() story.
That's too semantic IMHO. The failure mode was "enforced invariant stopped being true". If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

Problem is, the enclosing function (`fetch_features`) returns a `Result`, so the `unwrap` on line #82 only serves as a shortcut a developer took due to assuming `features.append_with_names` would never fail. Instead, the routine likely should have worked within `Result`.

sznio• 8 months ago

If they'd written explicit code to fail the request when that happened, the end result would have been exactly the same.

If the `.unwrap()` was replaced with `.expect("Feature config is too large!")` it would certainly make the outage shorter.

antonvs• 8 months ago

This is textbook "parse, don't validate" anti-pattern.

How so? “Parse, don’t validate” implies converting input into typed values that prevent representation of invalid state. But the parsing still needs to be done correctly. An unchecked unwrap really has nothing to do with this.

kccqzy• 8 months ago

GP completely misunderstands “parse, don’t validate” and also calls it an anti-pattern. GP clearly has no idea what this is.

rafaelmn• 8 months ago

That's such a bad take after reading the article. If you're going to write a system that preallocates and is based on hard assumptions about max size - the panic/unwrap approach is reasonable.

The config bug reaching prod without this being caught and pinpointed immediately is the strange part.

kevin_thibedeau2• 8 months ago

It's reasonable when testing protocols exercise the panic scenario. This is the problem with punting on error recovery. Nobody checks faults that propagate across domains of responsibility.

AtNightWeCode2• 8 months ago

Exactly. The newbie mistake in SQL is also way worse than this. But the whole design is also bad. Clearly implementing things at the wrong place.

And, it took like over an hour between the problem started til my sites went down. That is just crazy.

thatoneengineer2• 8 months ago

I agree there's no way to soft-error this, though "truncate and raise an alert" is arguably the better pattern.

slanterns2• 8 months ago

Today, many friends pinged me saying Cloudflare was down. As a core developer of the first generation of Cloudflare FL, I'd like to share some thoughts.
This wasn't an attack, but a classic chain reaction triggered by “hidden assumptions + configuration chains” — permission changes exposed underlying tables, doubling the number of lines in the generated feature file. This exceeded FL2's memory preset, ultimately pushing the core proxy into panic.
Rust mitigates certain errors, but the complexity in boundary layers, data flows, and configuration pipelines remains beyond the language's scope. The real challenge lies in designing robust system contracts, isolation layers, and fail-safe mechanisms.
Hats off to Cloudflare's engineers—those on the front lines putting out fires bear the brunt of such incidents.
Technical details: Even handling the unwrap correctly, an OOM would still occur. The primary issue was the lack of contract validation in feature ingest. The configuration system requires “bad → reject, keep last-known-good” logic.
Why did it persist so long? The global kill switch was inadequate, preventing rapid circuit-breaking. Early suspicion of an attack also caused delays.
Why not roll back software versions or restart?
Rollback isn't feasible because this isn't a code issue—it's a continuously propagating bad configuration. Without version control or a kill switch, restarting would only cause all nodes to load the bad config faster and accelerate crashes.
Why not roll back the configuration?
Configuration lacks versioning and functions more like a continuously updated feed. As long as the ClickHouse pipeline remains active, manually rolling back would result in new corrupted files being regenerated within minutes, overwriting any fixes.

https://x.com/guanlandai/status/1990967570011468071

anonymous9082132• 8 months ago

This tweet thread invokes genuine despair in me. Do we really have to outsource even our tweets to LLMs? Really? I mean, I get spambots and the like tweeting mass-produced slop. But what compels a former engineer of the company in question to offer LLM-generated "insight" to the outage? Why? For what purpose?

* For clarity, I am aware that the original tweets are written in Chinese, and they still have the stench of LLM writing all over them; it's not just the translation provided in the above comment.

abalone4• 8 months ago

I’ve led multiple incident responses at a FAANG, here’s my take. The fundamental problem here is not Rust or the coding error. The problem is:

1. Their bot management system is designed to push a configuration out to their entire network rapidly. This is necessary so they can rapidly respond to attacks, but it creates risk as compared to systems that roll out changes gradually.

2. Despite the elevated risk of system wide rapid config propagation, it took them 2 hours to identify the config as the proximate cause, and another hour to roll it back.

SOP for stuff breaking is you roll back to a known good state. If you roll out gradually and your canaries break, you have a clear signal to roll back. Here was a special case where they needed their system to rapidly propagate changes everywhere, which is a huge risk, but didn’t quite have the visibility and rapid rollback capability in place to match that risk.

While it’s certainly useful to examine the root cause in the code, you’re never going to have defect free code. Reliability isn’t just about avoiding bugs. It’s about understanding how to give yourself clear visibility into the relationship between changes and behavior and the rollback capability to quickly revert to a known good state.

Cloudflare has done an amazing job with availability for many years and their Rust code now powers 20% of internet traffic. Truly a great team.

ignoramous2• 8 months ago

Their bot management system is designed to push a configuration out to their entire network rapidly.

Once every 5m is not "rapidly". It isn't uncommon for configuration systems to do it every few seconds [0].

While it’s certainly useful to examine the root cause in the code.

Believe the issue is as much an output from a periodic run (clickhouse query) caused by (on the surface, an unrelated change) causing this failure. That is, the system that validated the configuration (FL2) was different to the one that generated it (ML Bot Management DB).

Ideally, it is the system that vends a complex configuration that also vends & tests the library to consume it, or the system that consumes it, does so as if it was "tasting" the configuration first before devouring it unconditionally [1].

Of course, as with all distributed system failures, this is all easier said and done in hindsight.

[0] Avoiding overload in distributed systems by putting the smaller service in control (pg 4), https://d1.awsstatic.com/builderslibrary/pdfs/Avoiding%20ove...

[1] Lessons from CloudFront (2016), https://youtube.com/watch?v=n8qQGLJeUYA&t=1050

jbs789• 8 months ago

Thanks for this assessment.

In a productive way, this view also shifts the focus to improving the system (visibility etc), empowering the team, rather than focusing on the code which broke (probably strikes fear in the individuals, to do anything!)

JB_Dev2• 8 months ago

Does their ring based rollout really truly have to be 0->100% in a few seconds?

I don’t really buy this requirement. At least make it configurable with a more reasonable default for “routine” changes. E.g. ramping to 100% over 1 hour.

As long as that ramp rate is configurable, you can retain the ability to respond fast to attacks by setting the ramp time to a few seconds if you truly think it’s needed in that moment.

pas• 8 months ago

Rolling out new code should be done differently than rolling out new data to fight bots.

If every time there's a new bot someone needs to write code that can blow up their whole service, maybe they need to iterate a bit on this design?

dev_l1x_be3• 8 months ago

Exactly the right take. Even when you want to have rapid changes on your infra, do it at least by region. You can start with the region where the least amount of users are impacted and if everything is fine, there is no elevated number of crashes for example, you can move forward. It was a standard practice at $RANDOM_FAANG when we had such deployments.

nrhrjrjrjtntbt2• 8 months ago

This guy SREs

tormeh2• 8 months ago

Partial disagree. There should be lints against 'unwrap's. An 'expect' at least forces you to write down why you are so certain it can't fail. An unwrap is not just hubris, it's also laziness, and has no place in sensitive code.

And yes, there is a lint you can use against slicing ('indexing_slicing') and it's absolutely wild that it's not on by default in clippy.

HelloNurse• 8 months ago

The "coding error" is a somewhat deliberate choice to fail eagerly that is usually safe but doesn't align with the need to do something (propagation of the configuration file) without failing.

I'm sure that there are misapplied guidelines to do that instead of being nice to incoming bot management configuration files, and someone might have been scolded (or worse) for proposing or attempting to handle them more safely.

zelphirkalt2• 8 months ago

It is just 2 different layers. Of course the code is also a problem, if it is in fact as the GP describes it. You are taking the higher level view, which is the second layer of dealing with not only this specific mistake, but also other mistakes, that can be related to arbitrary code paths.

Both are important, and I am pretty sure, that someone is gonna fix that line of code pretty soon.

polack3• 8 months ago

They failed on so many levels here.

How can you write the proxy without handling the config containing more than the maximum features limit you set yourself?

How can the database export query not have a limit set if there is a hard limit on number of features?

Why do they do non-critical changes in production before testing in a stage environment?

Why did they think this was a cyberattack and only after two hours realize it was the config file?

Why are they that afraid of a botnet? Does not leave me confident that they will handle the next Aisuru attack.

I'm migrating my customers off Cloudflare. I don't think they can swallow the next botnet attacks and everyone on Cloudflare go down with the ship, so it will be safer to not be behind Cloudflare when it hits.

matternot2• 8 months ago

I don't understand why they didn't validate and sanitize the new config file revision. If bad(whatever that reason is) throw an error and revert back to previous version. You don't need to take down the whole internet for that.

rkachowski• 8 months ago

Classic @devops_borat

"To make error is human. To propagate error to all server in automatic way is #devops"

BrtByte5• 8 months ago

You can write the safest code in the world, but if you're shipping config changes globally every few minutes without a robust rollback plan or telemetry that pinpoints when things go sideways, you're flying blind

mlrtime• 8 months ago

I've also led a team of Incident Commanders at a FAANG.

If this was a routine config change, I could see how it could take 2 hours to start the mediation plan. However they should have dashboards that correlate config setting changes with 500 errors (or equivalent). It gets difficult when you have many of of these going out at the same time and they are slowly rolled out.

The root cause document is mostly for high level and the public. The details on this specific outage will be in a internal document with many action items, some of them maybe quarter long projects including fixing this specific bug and maybe some linter/monitor to prevent it from happening again.

antihero4• 8 months ago

I would say that whilst this is a good top down view, that `.unwrap()` should have been caught at code-review and not allowed. Clippy rule could have saved a lot of money.

That and why the hell wasn't their alerting showing up colossal amount of panics in their bot manager thing?

seethishat• 8 months ago

The bot is efficient. This is by design. It will push out mistakes just as efficiently as it pushes out good changes. Good or bad... the plane of control is unchanged.

This is the danger of automated control systems. If they get hacked or somehow push out bad things (CloudStrike), they will have complete control and be very efficient.

hoppp• 8 months ago

You write so much rust you causally apply unwrap now to everything?

Rust compiler is a god of sorts, or at least a law of nature haha

Way to comment and go instantly off topic

throwaway38294• 8 months ago

This is a bummer. The unwrap()'ing function already returned a result and should have just propagated the error. Presumably the caller could have handled more sensibly than just panic'ing.

ozgrakkurt3• 8 months ago

Not panicking code is tedious to write. It is not realistic to expect everything to be non panic. There is a reason that panicking exists in the first place.

Them calling unwrap on a limit check is the real issue imo. Everything that takes in external input should assume it is bad input and should be fuzz tested imo.

In the end, what is the point of having a limit check if you are just unwrapping on it

cube00• 8 months ago

Not panicking code is tedious to write.

Using the question mark operator [1] and even adding in some anyhow::context goes a long way to being able to fail fast and return an Err rather then panicking.

Sure you need to handle Results all the way up the stack but it forces you to think about how those nested parts of your app will fail as you travel back up the stack.

[1] https://doc.rust-lang.org/rust-by-example/std/result/questio...

pjmlp• 8 months ago

Which is something I will bookmark for the usual Rust doesn't do exceptions discussions, except it kind of does even if called differently.

karel-3d3• 8 months ago

As a gopher I never understand why is there so many unwraps in an average rust code.

Average Go code has much less panics than Rust has unwraps, which are functionally equivalent.

richardwhiuk2• 8 months ago

Because Go silently gives you zero/null instead

speedgoose3• 8 months ago

The average golang code segfaults by design.

selfmodruntime6• 8 months ago

I love Go and write a ton of it. I've had real segfaults quite a lot.

branko_d4• 8 months ago

Safe things should be easy, dangerous things should be hard.

This .unwrap() sounds too easy for what it does, certainly much easier than having an entire try..catch block with an explicit panic. Full disclosure: I don't actually know Rust.

kettlecorn• 8 months ago

I don't think 'unwrap' is inherently the problem.

Any project has to reason about what sort of errors can be tolerated gracefully and which cannot. Unwrap is reasonable in scenarios you expect to never be reached, because otherwise your code will be full of all sorts of possible permutations and paths that are harder to reason about and may cascade into extremely nuanced or subtle errors.

Rust also has a version of unwrap called "expect" where you provide a string that logs why the unwrap occurred. It's similar, but for pieces of code that are crucial it could be a good idea to require all 'unwraps' to instead be 'expects' so that people at least are forced to write down a reason why they believe the unwrap can never be reached.

__bax• 8 months ago

git blame on .unwrap() line

selfmodruntime6• 8 months ago

While this is true, I wish that Rust had more of a first-class support for `no_panic`. Every solution we do have is hacky. I wish that I could guarantee that there were no panic calls anywhere in a code path.

gwd5• 8 months ago

This is the multi-million dollar .unwrap() story.

While there are certainly many things to admire about Rust, this is why I prefer Golang's "noisy" error handling. In golang that would be either:

    feature_values, err := features.append_with_names(...)

And the compiler would have complained that this value of `err` was unused; or you'd write:

    feature_values, _ := features.append_with_names(...)

And it would be far more obvious that an error message is being ignored.

(Renaming `unwrap` to `unwrapOrPanic` would probably help too.)

mamp• 8 months ago

I haven't been writing Rust for that long (about 2 years) but every time I see .unwrap() I read it as 'panic in production'. Clippy needs to have harder checks on unwrap.

jbreckmckye2• 8 months ago

But I could screw it up in Go, if I made the same assumptions

    fvs, err := features.AppendWithNames(..)
    if err != nil {
        // this will NEVER break
        panic(err)
    }

Ultimately I don't think language design can be the sole line of defence against system failures; it can only guide developers to think about error cases

tuetuopay5• 8 months ago

That may be me, but `.unwrap()` is much more obvious than `_`:

- it's literally written out that you're assuming it to be Ok

- there are no indications that the `_` is an error: it could very well be some other return value from the function. in your example, it could be the number of appended features, etc

That's why Go's error handling is indeed noisy: it's noise and you reduce noise by not handling errors. Rust's is terse yet verbose: if you add stuff it's because you're doing something wrong. You explicitly spelled out the error is being ignored.

Ukv• 8 months ago

And it would be far more obvious that an error message is being ignored.

Haven't used Go so maybe I'm missing some consideration, but I don't see how ", _" is more obvious than ".unwrap()". If anything it seems less clear, since you need to check/know the function's signature to see that it's an error being ignored (wouldn't be the case for a function like https://pkg.go.dev/math#Modf).

twhitmore2• 8 months ago

Interesting to see Rust error handling flunk out in practice.

It may be that forcing handling at every call tends to makes code verbose, and devs insensitized to bad practice. And the diagnostic Rust provided seems pretty garbage.

There is bad practice here too -- config failure manifesting as request failure, lack of failing to safe, unsafe rollout, lack of observability.

Back to language design & error handling. My informed view is that robustness is best when only major reliability boundaries need to be coded.

This the "throw, don't catch" principle with the addition of catches on key reliability boundaries -- typically high-level interactions where you can meaningfully answer a failure.

For example, this system could have a total of three catch clauses "Error Loading Config" which fails to safe, "Error Handling Request" which answers 5xx, and "Socket Error" which closes the HTTP connection.

Ciantic• 8 months ago

It may be that forcing handling at every call tends to makes code verbose

Rust has a lot of helpers to make it less verbose, even that error they demonstrate could've been written in some form `...code()?` with `?` helper that would have propagated the error forwards.

However I do acknowledge that writing Error types is boring sometimes so people don't bother to change their error types and just unwrap. But even my dinghy little apps for my personal use I do simple serach `unwrap` and make sure I have as few as possible.

hypeatei• 8 months ago

I don't understand how your takeaway is that this is a language flaw other than to assume that you have some underlying disdain for Rust. That's fine, but state it clearly please.

The end result would've been the exact same if they "handled" the error: a bunch of 500s. The language being used doesn't matter if an invariant in your system is broken.

andy_ppp• 8 months ago

This is why the Erlang/Elixir methodology of having supervision and letting things crash gracefully is so useful. You can either handle every single error gracefully or handle crashing gracefully - it's much easier and more realistic in large codebases to do the later.

tuetuopay5• 8 months ago

This would not have helped: the code would crash before doing anything useful at all.

If anything, the "crash early" mentality may even be nefarious: instead of handling the error and keeping the old config, you would spin on trying to load a broken config on startup.

NoboruWataya• 8 months ago

They should link this article in the docs for `unwrap()`.

sphericalkat• 8 months ago

Handling the error still would've returned a 5xx in this case, since the config file was still over the limit of features the service could handle.

BrtByte5• 8 months ago

Feels like a case where safety guarantees of Rust lulled them into thinking the edge cases were covered

peanut-walrus• 8 months ago

I wonder if similar to infrastructure resilience, code resilience is also required for critical services that can never go down? Instead of relying on a single implementation for a critical service, have multiple independent implementations in different languages. Back when I was running my own DNS servers, I did always ensure that primary and secondary were running on different platforms and different software.

meltyness• 8 months ago

tokio default behavior within a task is to ignore panics, such as an Err/None unwrap, and only crash that task, so it's impact limited so that's nice, maybe that's where the snowblindness came from.

it'd be kinda hard to amend the clippy lints to ignore coroutine unwraps but still pipe up on system ones. i guess.

edit: i think they'd have to be "solely-task-color-flavored" so definitely probably not trivial to infer

quotemstr5• 8 months ago

If the error had been an exception instead of a result, could have bubbled up

I have been saying for years that Rust botched error handling in unfixable ways. I will go to the grave believing Rust fumbled.

The design of the Rust language encourages people to use unwrap() to turn foreseeable runtime problems into fatal errors. It's the path of least resistance, so people will take it.

Rust encourages developers to consider only the happy path. No wonder it's popular among people who've never had to deal with failure.

All of the concomitant complexity--- Result, ?, the test thing, anyhow, the inability for stdlib to report allocation failure --- is downstream of a fashion statement against exceptions Rust cargo-culted from Go.

The funniest part is that Rust does have exceptions. It just calls them panics. So Rust code has to deal with the ergonomic footgun of Result but pays anyway for the possibility of exceptions. (Sure, you can compile with panic=abort. You can't count on it.)

I could not be more certain that Rust should have been a language with exceptions, not Result, and that error objects are a gross antipattern we'll regret for decades.

Veliladon• 8 months ago

Errors work just like exceptions especially if you use the ? operator and let the error bubble up the chain. This is the Rust equivalent of an unhandled exception and the ripcord being pulled.

otabdeveloper42• 8 months ago

Oh come on, stop spreading FUD. Rust programs are 100% immune to crashes and bugs, they have memory safety (c).

Also, exception handling is hard and lame. We don't need exceptions, just add a "match" block after every line in your program.

JuniperMesos8• 8 months ago

What's the point of this sarcastic comment? Do you think that some people claim that Rust's memory safety guarantees mean that a Rust program is incapable of crashing or having a bug? This is a dumb thing to claim certainly, but I'm not aware of anyone actually making this claim.

I'm also not sure what you're getting at with the comment about exception handling being lame. I think the ML/Haskell inspired model that Rust uses of having a parameterized Result type for fallible operations is generally better than exceptions for a variety of reasons (although maybe better Exception semantics could help with some of this), but what does this have to do with match blocks?

echelon17• 8 months ago

This is the multi-million dollar .unwrap() story.

First multi-million dollar .unwrap() story.

torginus2• 8 months ago

Say what you want exception haters, but at least in exceptions-as-default languages, the decision of a particular issues is fatal to the whole program can be decided centrally at a high level, and not every choice is forced to be up to individual discretion.

underdeserver5• 8 months ago

But you can do the same thing with Rust, by piping up Results.

torginus2• 8 months ago

By the way - does this discussion matter and were they wrong to use unwrap()?

The way they wrote the code means that having more than 200 features is a hard non-transient error - even if they recovered from it, it meant they'd have had the same error when the code got to the same place.

I'm sure when the process crashed, k8s restarted the pod or something - then it reran the same piece of code and crashed in the same place.

While I don't necessarily agree with crashing as business strategy, I don't think that doing anything other than either dropping the extra rules or allocating more memory - neither of which the original code was built to do (probably by design).

The code made the local hard assumption that there won't ever be more than 200 rules and its okay to crash if that count is exceeded.

If you design your code around an invariant never being violated (which is fine), you have to make it clear on a higher level that they did.

This isn't a Rust problem (though Rust does make it easy to do the wrong thing here imo)

grogers2• 8 months ago

Instead of crashing when applying the new config, it's more common to simply ignore the new config if it cannot be applied. You keep running in the last known good state. Operators then get alerts about the failures and can diagnose and resolve the underlying issue.

That's not always foolproof, e.g. a freshly (re)started process doesn't have any prior state it can fall back to, so it just hard crashes. But restarts are going to be rate limited anyways, so even then there is time to mitigate the issue before it becomes a large scale outage

trengrj• 8 months ago

Classic combination of errors:

Having the feature table pivoted (with 200 feature1, feature2, etc columns) meant they had to do meta queries to system.columns to get all the feature columns which made the query sensitive to permissioning changes (especially duplicate databases).

A Crowdstrike style config update that affects all nodes but obviously isn't tested in any QA or staged rollout strategy beforehand (the application panicking straight away with this new file basically proves this).

Finally an error with bot management config files should probably disable bot management vs crash the core proxy.

I'm interested here why they even decided to name Clickhouse as this error could have been caused by any other database. I can see though the replicas updating causing flip / flopping of results would have been really frustrating for incident responders.

tptacek22• 8 months ago

Right but also this is a pretty common pattern in distributed systems that publish from databases (really any large central source of truth); it might be like the problem in systems like this. When you're lucky the corner cases are obvious; in the big one we experienced last year, a new row in our database tripped an if-let/mutex deadlock, which our system dutifully (and very quickly) propagated across our entire network.

The solution to that problem wasn't better testing of database permutations or a better staging environment (though in time we did do those things). It was (1) a watchdog system in our proxies to catch arbitrary deadlocks (which caught other stuff later), (2) segmenting our global broadcast domain for changes into regional broadcast domains so prod rollouts are implicitly staged, and (3) a process for operators to quickly restore that system to a known good state in the early stages of an outage.

(Cloudflare's responses will be different than ours, really I'm just sticking up for the idea that the changes you need don't follow obviously from the immediate facts of an outage.)

nullbyte8082• 8 months ago

I thought it was an internal mess-up. I thought an employee screwed a file up. Old methods are sometimes better than new. AI fails us again!

ksajadi• 8 months ago

May I just say that Matthew Prince is the CEO of Cloudflare and a lawyer by training (and a very nice guy overall). The quality of this postmortem is great but the fact that it is from him makes one respect the company even more.

dzonga2• 8 months ago

thread fl2_worker_thread panicked: called Result::unwrap() on an Err value

I don't use Rust, but a lot of Rust people say if it compiles it runs.

Well Rust won't save you from the usual programming mistake. Not blaming anyone at cloudflare here. I love Cloudflare and the awesome tools they put out.

end of day - let's pick languages | tech because of what we love to do. if you love Rust - pick it all day. I actually wanna try it for industrial robot stuff or small controllers etc.

there's no bad language - just occassional hiccups from us users who use those tools.

dzonga2• 8 months ago

other people might say - why use unsafe rust - but we don't know the conditions of what the original code shipped under. why the pr was approved.

could have been tight deadline, managerial pressure or just the occasional slip up.

tptacek22• 8 months ago

What people are saying is that idiomatic prod rust doesn't use unwrap/expect (both of which panic on the "exceptional" arm of the value) --- instead you "match" on the value and kick the can up a layer on the call chain.

olivia-banks• 8 months ago

What happens to it up the callstack? Say they propagated it up the stack with `?`. It has to get handled somewhere. If you don't introduce any logic to handle the duplicate databases, what else are you going to do when the types don't match up besides `unwrap`ing, or maybe emitting a slightly better error message? You could maybe ignore that module's error for that request, but if it was a service more critical than bot mitigation you'd still have the same symptom of getting 500'd.

jryio2• 8 months ago

You misunderstand what Rust’s guarantees are. Rust has never promised to solve or protect programmers from logical or poor programming. In fact, no such language can do that, not even Haskell.

Unwrapping is a very powerful and important assertion to make in Rust whereby the programmer explicitly states that the value within will not be an error, otherwise panic. This is a contract between the author and the runtime. As you mentioned, this is a human failure, not a language failure.

Pause for a moment and think about what a C++ implementation of a globally distributed network ingress proxy service would look like - and how many memory vulnerabilities there would be… I shudder at the thought… (n.b. nginx)

This is the classic example of when something fails, the failure cause over indexes on - while under indexing on the quadrillions of memory accesses that went off without a single hitch thanks to the borrow checker.

I postulate that whatever the cost in millions or hundreds of millions of dollars by this Cloudflare outage, it has paid for more than by the savings of safe memory access.

See: https://en.wikipedia.org/wiki/Survivorship_bias

samdoesnothing5• 8 months ago

Pause for a moment and think about what a C++ implementation of a globally distributed network ingress proxy service would look like - and how many memory vulnerabilities there would be… I shudder at the thought

I mean thats an unfalsifiable statement, not really fair. C is used to successfully launch spaceships.

Whereas we have a real Rust bug that crashed a good portion of the internet for a significant amount of time. If this was a C++ service everyone would be blaming the language, but somehow Rust evangelicals are quick to blame it on "unidiomatic Rust code".

A language that lets this easily happen is a poorly designed language. Saying you need to ban a commonly used method in all production code is broken.

diath2• 8 months ago

You misunderstand what Rust’s guarantees are.

Well, no, most Rust programmers misunderstand what the guarantees are because they keep parroting this quote. Obviously the language does not protect you from logic errors, so saying "if it compiles, it works" is disingenuous, when really what they mean is "if it compiles, it's probably free of memory errors".

metaltyphoon2• 8 months ago

Well Rust won't save you from the usual programming mistake

This is not a Rust problem. Someone consciously chose to NOT handle an error, possibly thinking "this will never happen". Then someone else conconciouly reviewed (I hope so) a PR with an unwrap() and let it slide.

Ekaros2• 8 months ago

And people doing testing failed to ignore their excuse of this never happening and still testing it. With this kind of systems you need the separate group that just ignores any "this will never happen" and still checks what happens if it does.

Now it might be that it was tested, but then ignored or deprioritised by management...

lmm3• 8 months ago

Rust won't save you from the usual programming mistake.

Disagree. Rust is at least giving you an "are you sure?" moment here. Calling unwrap() should be a red flag, something that a code reviewer asks you to explain; you can have a linter forbid it entirely if you like.

No language will prevent you from writing broken code if you're determined to do so, and no language is impossible to write correct code in if you make a superhuman effort. But most of life happens in the middle, and tools like Rust make a huge difference to how often a small mistake snowballs into a big one.

SchemaLoad4• 8 months ago

Yep, unwrap() and unsafe are escape hatches that need very good justifications. It's fine for casual scripts where you don't care if it crashes. For serious production software they should be either banned, or require immense scrutiny.

tick_tock_tick2• 8 months ago

Disagree. Rust is at least giving you an "are you sure?" moment here. Calling unwrap() should be a red flag, something that a code reviewer asks you to explain; you can have a linter forbid it entirely if you like.

No one treats it like that and nearly every Rust project is filled with unwraps all over the place even in production system like Cloudflare's.

nedt• 8 months ago

you can have a linter forbid it entirely if you like.

It would be better if that would be the other way round "linter forbids it unless you ask it not to". Never wrong to allow users to shoot themself in the foot, but it should be explicit.

Klonoar2• 8 months ago

I don't use Rust, but a lot of Rust people say if it compiles it runs.

Do you grok what the issue was with the unwrap, though...?

Idiomatic Rust code does not use that. The fact that it's allowed in a codebase says more about the engineering practices of that particular project/module/whatever. Whoever put the `unwrap` call there had to contend with the notion that it could panic and they still chose to do it.

It's a programmer error, but Rust at least forces you to recognize "okay, I'm going to be an idiot here". There is real value in that.

uecker3• 8 months ago

While I agree that Rust got it right by being more explicit, a lot of bugs in C/C++ can also easily avoided with good engineering practices. The Rust argument that it is mainly the fault of the programming language with C/C++ was always a huge and unfair exaggeration. Now with this entirely predictable ".unwrap" desaster (in general, not necessarily this exact scenarious), the "no true Rustacean would have put unwrap in production" fallacy is sad and funny at the same time.

the84722• 8 months ago

"if it compiles it runs" - this is indeed an inaccurate marketing slogan. A more precise formulation would be "if it compiles then the static type system, pattern matching, explicit errors, Send bounds, etc. will have caught a lot of bugs that in other languages would have manifested as runtime errors".

Anecdotally I can write code for several hours, deploy it to a test sandbox without review or running tests and it will run well enough to use it, without silly errors like null pointer exceptions, type mismatches, OOBs etc. That doesn't mean it's bug-free. But it doesn't immediately crash and burn either. Recently I even introduced a bug that I didn't immediately notice because careful error handling in another place recovered from it.

rvz2• 8 months ago

Great write up.

This is the first significant outage that has involved Rust code, and as you can see the .unwrap is known to carry the risk of a panic and should never be used on production code.

chatmasta• 8 months ago

Wow, crazy disproportional drop in the stock price… good buying opportunity for $NET.

rvz2• 8 months ago

Agree.

Cloudflare is very cheap at these prices.

nanankcornering• 8 months ago

Matt, Looking forward in regaining Elon's and his team trust to use CF again.

wilg• 8 months ago

I wish Elon would regain my trust!

RagingCactus• 8 months ago

Lots of people here are (perhaps rightfully) pointing to the unwrap() call being an issue. That might be true, but to me the fact that a reasonably "clean" panic at a defined line of code was not quickly picked up in any error monitoring system sounds just as important to investigate.

Assuming something similar to Sentry would be in use, it should clearly pick up the many process crashes that start occurring right as the downtime starts. And the well defined clean crashes should in theory also stand out against all the random errors that start occuring all over the system as it begins to go down, precisely because it's always failing at the exact same point.

rixed3• 8 months ago

Exactly! You could have `rand() > 0.5 && panic!()` in the code of your bot module, and that should not put the internet on fire.

The issue here is about the system as a whole not any line of code.

frumplestlatz19• 8 months ago

The issue here is about the system as a whole not any line of code.

Unsoundness in the type system that leads to a systemic failure is about the system as a whole.

Not everything can be recovered from restarting a process, and process correctness and recovery is something that also derives from your type system.

slyall• 8 months ago

Ironically just now I got a Cloudflare "Error code 524" page because blog.cloudflare.com was down

jijji• 8 months ago

this is where change management really shines because in a change management environment this would have been prevented by a backout procedure and it would never have been rolled out to production before going into QA, with peer review happening before that... I don't know if they lack change management but it's definitely something to think about

mercnz2• 8 months ago

i think that is data rather than code which is where it falls short, in a way you need stringent code and more safeguarded code; it's like if everyone sends you 64k posts as that's all your proxy layer lets in, someone checked sending 128kb and it gave an error before reaching your app - and then someone sends 128kb and the proxy layer has changed - and your app crashes as it was more than 64kb and your app had an assert against that. to actually track issues with erraneous data that overflows well and stuff isn't so much code test but more like fuzz testing, brute force testing etc. which i think people should do; but that's more like we need strong test networks, and also those test networks may need to be more internet like to reflect real issues too, so the whole testing infrastructure in itself becomes difficult to get right - like they have their own tunneling system etc, they could segregate some of their servers and make a test system with better error diagnosis etc potentially. but to my mind, if they had better error propogation back that really identified what was happening and where then that would be a lot better in general. sure, start doing that on a test network. this is something i've beeen tihnking about in general - i made a simple rpc system for being able to send real time rust tracing logs (it allows to just use the normal tracing framework and use a thin rpc layer) back from multiple end servers but that's mostly for granular debugging. i've never quite understood why systems like systemd-journald aren't more network centric when they're going to be big and complex kitchensink approaches - apparently there's dbus support, but to my mind something inbetween debugging level of code and warning/info. like even if it's doing things like 1/20 of log info it's too much volume if things like large files getting close to limits is increasing etc and we can see this as things run, and can see if it's localised or common etc it'd help have more resilient systems. something may already exist in this line but i didn't come across anything in a reasonably passive way - i mean there's debugging tools like dtrace etc that have been around for ages.

NetMageSCW7• 8 months ago

It is a bit tough to do all that in five minutes.

yoyohello137• 8 months ago

People really like to hate on Rust for some reason. This wasn’t a Rust problem, no language would have saved them from this kind of issue. In fact, the compiler would have warned that this was a possible issue.

I get it, don’t pick languages just because they are trendy, but if any company’s use case is a perfect fit for Rust it’s cloudflare.

SchemaLoad4• 8 months ago

Yeah even if you handled this situation without unwrap() if you just went down an error path that didn't panic, the service would likely still be inoperable if every single request went down the error path.

HL33tibCe72• 8 months ago

Okay, but if you returned a wrapped error it’d at least be easier to debug.

samdoesnothing5• 8 months ago

The reason why people are criticizing is because Rust evangelicals say stuff like "if it compiles it works" or talk about how Rust's type system is so much better than other languages that it catches logic errors like this. You won't see Go or Java developers making such strong claims about their preferred languages.

Patryk27• 8 months ago

[...] it catches logic errors like this

but Rust's type system did catch this error - and then author decided it's fine to panic if this error happens

You won't see Go or Java developers making such strong claims about their preferred languages.

yess no Java developer ever said that OOP will solve world hunger

JamesJGoodwin• 8 months ago

Currently that limit is set to 200, well above our current use of ~60 features. Again, the limit exists because for performance reasons we preallocate memory for the features.

So they basically hardcoded something, didn't bother to cover the overflow case with unit tests, didn't have basic error catching that would fallback and send logs/alerts to their internal monitoring system and this is why half of the internet went down?

testemailfordg2• 8 months ago

"Customers on our old proxy engine, known as FL, did not see errors, but bot scores were not generated correctly, resulting in all traffic receiving a bot score of zero."

This simply means, the exception handling quality of your new FL2 is non-existent and is not at par / code logic wise similar to FL.

I hope it was not because of AI driven efficiency gains.

lmm3• 8 months ago

In most domains, silently returning 0 in a case where your logic didn't actually calculate the thing you were trying to calculate is far worse than giving a clear error.

habibur• 8 months ago

    On 18 November 2025 at 11:20 UTC (all times in this blog are UTC), Cloudflare's network began experiencing significant failures
    As of 17:06 all systems at Cloudflare were functioning as normal

6 hours / 5 years gives ~99.98% uptime.

TehShrike• 8 months ago

I'm feeling generous tonight, I'm willing to consider 0.99986 to round to 99.99%

arjie• 8 months ago

Great post-mortem. Very clear. Surprised that num(panicking threads) didn't show up somewhere in telemetry.

avereveard• 8 months ago

Question: customer having issues also couldn't switch their dns to bypass the service, why is the control plane updated along the data plane here it seem a lot of use could save business continuity if they could change their dns entry temporarily

wildmXranat3• 8 months ago

Hold up ,- when I used a C or similar language for accessing a database and wanted to clamp down on memory usage to deterministically control how much I want to allocated, I would explicitly limit the number of rows in the query.

There never was an unbound "select all rows from some table" without a "fetch first N rows only" or "limit N"

If you knew that this design is rigid, why not leverage the query to actually do it ?

What am I missing ?

JuniperMesos8• 8 months ago

Because nothing forced them to and they didn't think of it. Maybe the people writing the code that did the query knew that the tables they were working with never had more than 60 rows and figured "that's small" so they didn't bother with a limit. Maybe the people who wrote the file size limit thought "60 rows isn't that much data" and made a very small file size limit and didn't coordinate with the first people.

Anyway regardless of which language you use to construct a SQL query, you're not obligated to put in a max rows

wildmXranat3• 8 months ago

I imagine there's numerous ways to protect against it and protection should've been added by whoever decided on this optimization. In data layer, create some kind of view which never returns more than 200 rows from base table(s). In code, use some kind of iterator. I'm not a Rust guy, just a C defensive practices type of dude, but maybe they just missed a biggie during a code review.

sema4hacker• 8 months ago

If you deploy a change to your system, and things start to go wrong that same day, the prime suspect (no matter how unlikely it might seem) should be the change you made.

wildmXranat3• 8 months ago

My first question when faced with an unknown error is "What was the last change and when was it promoted?"

1970-01-012• 8 months ago

This is an area where they are allowed to think yet another record setting DDoS attack first and bad config second.

1970-01-012• 8 months ago

I would have been a bit cheeky and opened with 'It wasn't DNS.'

cvhc3• 8 months ago

I don't get why that SQL query was even used in the first place. It seems it fetches feature names at runtime instead of using a static hardcoded schema. Considering this decides the schema of a global config, I don't think the dynamicity is a good idea.

back_to_basics• 8 months ago

While it's certainly worthwhile to discuss the Technical and Procedural elements that contributed to this Service Outage, the far more important (and mutually-exclusive aspect) to discuss should be:

Why have we built / permitted the building of / Subscribed to such a Failure-intolerant "Network"?

JuniperMesos8• 8 months ago

Who's "we"? This is not a trick question, what specific people do you think acted wrongly here? I don't use Cloudflare personally. I don't run any of the sites that do use it. The people who did make the decision to put thier websites behind Cloudflare could stop, and maybe some will, but presumably they're paying for it because they think, perhaps accurately, that they get value out of it. Should some power compel them not to use Cloudflare?

issafram• 8 months ago

I give them a pass on lots of things, but this is inexcusable

alhirzel2• 8 months ago

I worry this is the big botnet flexing.

Even worse - the small botnet that controls everything.

ademarre2• 8 months ago

I integrated Turnstile with a fail-open strategy that proved itself today. Basically, if the Turnstile JS fails to load in the browser (or in a few specific frontend error conditions), we allow the user to submit the web form with a dummy challenge token. On the backend, we process the dummy token like normal, and if there is an error or timeout checking Turnstile's siteverify endpoint, we fail open.

Of course, some users were still blocked, because the Turnstile JS failed to load in their browser but the subsequent siteverify check succeeded on the backend. But overall the fail-open implementation lessened impact to our customers nonetheless.

Fail-open with Turnstile works for us because we have other bot mitigations that are sufficient to fall back on in the event of a Cloudflare outage.

cj• 8 months ago

So to bypass captcha all a user has to do is block the script from loading? I can see that working but only for attacks that aren’t targeted?

ademarre2• 8 months ago

Only if they are able to block the siteverify check performed by our backend server. That's not the kind of attack we are trying to mitigate with Turnstile.

pdimitar6• 8 months ago

While I heavily frown upon using `unwrap` and `expect` in Rust code and make sure to have Clippy tell me about every single usage of them, I also understand that without them Rust might have been seen as an academic curiosity language.

They are escape hatches. Without those your language would never take off.

But here's the thing. Escape hatches are like emergency exits. They are not to be used by your team to go to lunch in a nearby restaurant.

---

Cloudflare should likely invest in better linting and CI/CD alerts. Not to mention isolated testing i.e. deploy this change only to a small subset and monitor, and only then do a wider deployment.

Hindsight is 20/20 and we can all be smartasses after the fact of course. But I am really surprised because lately I am only using Rust for hobby projects and even I know I should not use `unwrap` and `expect` beyond the first iteration phases.

---

I have advocated for this before but IMO Rust at this point will benefit greatly from disallowing those unsafe APIs by default in release mode. Though I understand why they don't want to do it -- likely millions of CI/CD pipelines will break overnight. But in the interim, maybe a rustc flag we can put in our `Cargo.toml` that enables such a stricter mode? Or have that flag just remove all the panicky API _at compile time_ though I believe this might be a Gargantuan effort and is likely never happening (sadly).

In any case, I would expect many other failures from Cloudflare but not _this_ one in particular.

duped4• 8 months ago

This is not a reasonable take to me. unwrap/expect are the idiomatic way to express code paths returning Option/Result as unreachable.

Bubbling up the error or None does not make the program correct. Panicking may be the only reasonable thing to do.

If panicking is guaranteed because of some input mistake to the system your failure is in testing.

pdimitar6• 8 months ago

I agree the failure is in testing but what you can and should do is raise in alert in your APM system before the runtime panic, in the code path that is deemed impossible to hit.

I am not trashing on them, I've made such mistakes in the past, but I do expect more from them is all.

And you will not believe how many alerts I got for the "impossible" errors.

I do agree there was not too much that could have been done, yes. But they should have invested in more visibility and be more thorough. I mean, hobbyist Rust devs seem to do that better.

It was just a bit disappointing for me. As mentioned above, I'd understand and sympathise with many other mistakes but this one stung a bit.

csomar3• 8 months ago

Say you panic, now you need to have an external system that catches this panic and reports back; and does something meanwhile to recover your system.

If you think about it, it’s not really different from handling the bubbled up error inside of Rust. You don’t (?) your results and your errors go away, they just move up the chain.

ulfw• 8 months ago

The internet hasn't been the internet in years. It was originally built to withstand wars. The whole idea of our IP based internet was to reroute packages should networks go down. Decentralisation was the mantra and how it differed from early centralised systems such as AOL et al.

This is all gone. The internet is a centralised system in the hand of just a few companies. If AWS goes down half the internet does. If Azure, Google Cloud, Oracle Cloud, Tencent Cloud or Alibaba Cloud goes down a large part of the internet does.

Yesterday with Cloudflare down half the sites I tried gave me nothing but errors.

The internet is dead.

rubatuga• 8 months ago

I mean you still depend on authoritative dns servers no?

samdoesnothing5• 8 months ago

It's not that deep, if AWS or Cloudflare suddenly disappeared sites would move to different hosts, it wouldn't mean the internet would die.

lapcat• 8 months ago

It's unbelievable that the end of this postmortem is an advertisement for Cloudflare.

The last thing we need here is for more of the internet to sign up for Cloudflare.

JuniperMesos8• 8 months ago

It's in Cloudflare's interest and they wrote the blog post.

AtNightWeCode2• 8 months ago

So they made a newbie mistake in SQL that would not even pass an AI review. They did not verify the change in a test environment. And I guess the logs are so full of errors it is hard to pinpoint which matters. Yikes.

cvshane• 8 months ago

Would be nice if their Turnstile could be turned off on their login page when something like this happens, so we can attempt to route traffic away from Cloudflare during the outage. Or at least have a simple app where this can be modified from.

snoppy45• 8 months ago

I think you should give me a credit for all the income I lost due to this outage. Who authorized a change to the core infrastructure during the period of the year when your customers make the most income? Seriously, this is a management failure at the highest levels of decision-making. We don't make any changes to our server infrastructure/stack during the busiest time of the year, and neither should you. If there were an alternative to Cloudflare, I'd leave your service and move my systems elsewhere.

sophacles• 8 months ago

I think you should get exactly what the contract you signed said you'd get. Outages happen in all infrasturture. Planned and unplanned ones both. The SLA and SLO are literal acknowledgements of the fact that and part of the contract for that reason.

Its fair to be upset at their decision making - use that to renegotiate your contract.

nwellinghoff• 8 months ago

The real take away is that so much functionality depends on a few players. This is a fundamental flaw in design that is getting worse by the year as the winner takes all winners win. Not saying they didn’t earn their wins. But the fact remains. The system is not robust. Then again, so what. It went down for a while. Maybe we shouldn’t depend on the internet being “up” all the time.

awesome_dude• 8 months ago

But but but muh rust makes EVERYTHING safer!!!!

My dude, everything is a footgun if you hold it wrong enough

homeonthemtn• 8 months ago

Another long day...

weihz5138• 8 months ago

Glad it's fixed, keep going!

bri3d3• 8 months ago

Everyone is hating on unwrap, but to me the odd and more interesting part is that it took 3 hours to figure this out? Even with a DDoS red herring, shouldn’t there have been a crash log or telemetry anomaly correlated? Also, shouldn’t the next steps and resolution focus more on this aspect, since it’s a high leverage tool for identifying any outage caused by a panic rather than just preventing a recurrence of random weird edge case #9999999?

spprashant2• 8 months ago

I have nowhere near the experience managing such complex systems, but I can empathize with this. In a high-pressure situations the most obvious things get missed. If someone is convinced System X is at fault, your mind can make leaps to justify every other degraded system is a downstream effect of that. Cause and effect can get switched.

Sometimes you have smart people in the room who dig deeper and fish it out, but you cannot always rely on that.

bri3d3• 8 months ago

I have plenty of empathy, having been in plenty of similar situations. It's not a matter of "I can't BELIEVE it took that long" (although it is a bit surprising) so much as that I disagree with the key takeaways here in the HN comments section and in the blog itself, which focus strongly on fixing rare edge case issues (the bad ClickHouse query and a bad config file causing a panic via unwrap), rather than reducing MTTR for all issues by improving the debug and monitoring experience.

I'm also suspicious that

Eliminating the ability for core dumps or other error reports to overwhelm system resources

from the blog had a lot more to do with the issue than perhaps the narrative is letting on.

reassess_blind2• 8 months ago

Once they figured it out they didn't have a way to load in a new feature file, had to figure that out, and then restart every machine.

bri3d3• 8 months ago

This took one of the three hours; it seems to have taken from 11:28 to 13:37 to recognize that the configuration file panic was the cause of the issue.

speedgoose3• 8 months ago

Yes this is the weird part for me. With good monitoring, the panic at unwrap should have been detected immediately. I assume they weren't looking at the right place, but still. If you use Sentry for example, a brand new panic should be pretty visible.

discordianfish• 8 months ago

Indeed, nothing about the root issues are particular surprising but why they missed a critical service panicing across their fleet is not bubbling up.

My best guess is too many alerts firing without a clear hierarchy and possibilities to seprate cause from effect. It's a typical challenge but I wish they would shed some light on that. And its a bit concerning that improving observability is not part of their follow up steps.

themark• 8 months ago

“…and the fluctuation stabilized in the failing state.”

Sounds like the ops team had one hell of a day.

abigailphoebe2• 8 months ago

kudos to getting this blog post out so fast, it’s well written and is appreciated.

i’m a little confused on how this was initially confused for an attack though?

is there no internal visibility into where 5xx’s are being thrown? i’m surprised there isn’t some kind of "this request terminated at the <bot checking logic>" error mapping that could have initially pointed you guys towards that over an attack.

also a bit taken aback that .unwrap()’s are ever allowed within such an important context.

would appreciate some insight!

pornel2• 8 months ago

1. Cloudflare is in the business of being a lightning rod for large and targeted DoS attacks. A lot of cases are attacks.

2. Attacks that make it through the usual defences make servers run at rates beyond their breaking point, causing all kinds of novel and unexpected errors.

Additionally, attackers try to hit endpoints/features that amplify severity of their attack by being computationally expensive, holding a lock, or trigger an error path that restarts a service — like this one.

abigailphoebe2• 8 months ago

eh, i'm not convinced.

this was in the middle of a scheduled maintenance, with all requests failing at a singular point - that being a .unwrap().

there should be internal visibility into the fact a large number of requests are failing all at the same LOC - and attention should be focused there instantly imo.

or at the very least, it shouldn't take 4 hours for anyone to even consider it wasn't an attack.

in situations such as this, where your entire infra is fucked, you should have multiple crisis teams working in parallel, under different assumptions.

if even one additional team was created that worked under the assumption it was an infra issue rather than an attack, this situation could have been resolved many hours earlier.

for a product as vital to the internet as cloudflare, it is unacceptable to not have this kind of crisis management.

robofanatic• 8 months ago

hope no one was fired

spprashant2• 8 months ago

A lot of outages off late seem to be related to automated config management.

Companies seem to place a lot of trust is configs being pushed automatically without human review into running systems. Considering how important these configs are, shouldn't they perhaps first be deployed to a staging/isolated network for a monitoring window before pushing to production systems?

Not trying to pontificate here, these systems are more complicated than anything I have maintained. Just trying to think of best practices perhaps everyone can adopt.

jdlyga• 8 months ago

We shouldn't be having critical internet-wide outages on a monthly basis. Something is systematically wrong with the way we're architecting our systems.

smt883• 8 months ago

Cloudflare, Azure, and other single points of failure are solving issues inherent to webhosting, and those problems have become incredibly hard due to the massive scale of bad actors and the massive complexity of managing hardware and software.

What would you propose to fix it? The fixed cost of being DDoS-proof is in the hundreds of millions of dollars.

Pooge• 8 months ago

Costs to architect systems that serve millions of request daily have gone down. Not up.

Hell, I would be very curious to know the costs to keep HackerNews running. They probably serve more users than my current client.

People want to chase the next big thing to write it on their CV, not architect simple systems that scale. (Do they even need to scale?)

vlaaad• 8 months ago

Saying "other single points of failure" makes no sense whatsoever.

keypusher2• 8 months ago

The most surprising thing to me here is that it took 3 hours to root cause, and points to a glaring hole in the platform observability. Even taking into account the fact that the service was failing intermittently at first, it still took 1.5 hours after it started failing consistently to root cause. But the service was crashing on startup. If a core service is throwing a panic at startup like that, it should be raising alerts or at least easily findable via log aggregation. It seems like maybe there was some significant time lost in assuming it was an attack, but it also seems strange to me that nobody was asking "what just changed?", which is usually the first question I ask during an incident.

Fiadliel• 8 months ago

If one actually looks at the current pingora API, it has limited ability to initialize async components at startup - the current pattern seems to be to lazily initialize on first call. An obvious downside of this is that a service can startup in a broken state. e.g. https://github.com/cloudflare/pingora/issues/169

I can imagine that this could easily lead to less visibility into issues.

eastdakota11• 8 months ago

That’s not accurate. As with any incident response there were a number of theories of the cause we were working in parallel. The feature file failure was one identified as potential in the first 30 minutes. However, the theory that seemed the most plausible based on what we were seeing (intermittent, initially concentrated in the UK, spike in errors for certain API endpoints) as well as what else we’d been dealing with (a bot net that had escalated DDoS attacks from 3Tbps to 30Tbps against us and others like Microsoft over the last 3 months). We worked multiple theories in parallel. After an hour we ruled out the DDoS theory. We had other theories also running in parallel, but at that point the dominant theory was that the feature file was somehow corrupt. One thing that made us initially question the theory was nothing in our changelogs seemed like it would have caused the feature file to grow in size. It was only after the incident that we realized the database permissions change had caused it, but that was far from obvious. Even after we identified the problem with the feature file, we did not have an automated process to role the feature file back to a known-safe previous version. So we had to shut down the reissuance and manually insert a file into the queue. Figuring out how to do that took time and waking people up as there are lots of security safeguards in place to prevent an individual from easily doing that. We also needed to double check we wouldn’t make things worse. The propagation then takes some time especially because there are tiers of caching of the file that we had to clear. Finally we chose to restart the FL2 processes on all the machines that make up our fleet to ensure they all loaded the corrected file as quickly as possible. That’s a lot of processes on a lot of machines. So I think best description was it took us an hour for the team to coalesce on the feature file being the cause and then another two to get the fix rolled out.

keypusher2• 8 months ago

Thank you for the clarification and insight, with that context it does make more sense to me. Is there anything you think can be done to improve the ability to identify issues like this more quickly in the future?

QREguy• 8 months ago

Any "limits" on system should be alerted... like at 70% or 80% threshold.. it might be worth it for a SRE to revisit the system limits and ensuring threshold based alerting around it..

Adam2025• 8 months ago

Cloudflare’s write-up is clear and to the point. A small change spread wider than expected, and they explained where the process failed. It’s a good reminder that reliability depends on strong workflows as much as infrastructure.

vasuadari• 8 months ago

Wondering why they didn’t disable the bot management temporarily to recover. Websites could have survived temporarily without it compared to the outage itself.

uecker3• 8 months ago

So an unhandled error condition after an configuration update similar to Crowdstrike - if they had just used a programming language where this can't happen due to the superior type system such as Rust. Oh wait.

nurettin• 8 months ago

I can never get used to the error happening at call site rather than within the function where the early return of Err happened. It is not "much cleaner", you have no idea which line and file caused it at call site. By default Returning should have a way of setting a marker which can then be used to map back to the line() and file(). 10+ years and still no ergonomics.

aetherspawn• 8 months ago

Cloudflare Access is still experiencing weird issues for us (it’s asking users to SSO login to our public website even though our zone rules - set on a completely different zone - haven’t changed).

I don’t think the infrastructure has been as fully recovered as they think yet…

laurentiurad• 8 months ago

Reason for the failure: switched to Chad IDE to ship new features.

sanjitb2• 8 months ago

cloudflare:

Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare.

also cloudflare:

The Cloudflare Dashboard was also impacted due to both Workers KV being used internally and Cloudflare Turnstile being deployed as part of our login flow.

perching_aix3• 8 months ago

I believe you're mistakenly equating Cloudflare's status page with the Cloudflare Dashboard? They're not the same thing.

Cloudflare's status page: https://www.cloudflarestatus.com/

Cloudflare Dashboard: https://dash.cloudflare.com/

sanjitb2• 8 months ago

thanks for clarifying! i guess then they never explained why the status page went down, even though it's supposed to be running on independent infrastructure.

slanterns2• 8 months ago

https://x.com/guanlandai/status/1990967570011468071

wileydragonfly• 8 months ago

Did some $300k chief of IT blame it all on some overworked secretary clicking a link in an email they should have run through a filter? Because that’s the MO.

mos87• 8 months ago

Hopefully that's the case.

aspbee555• 8 months ago

unwraps are so very easy to use and they have bit me so many times because you can nearly never run into a problem and suddenly crashes from an unwrap that almost always was fine

NetMageSCW7• 8 months ago

How was AI involved?

nromiun• 8 months ago

Unbelievable. I guess it's time to grep for every .unwrap in our code.

zgk7iqea• 8 months ago

theres a clippy lint for that

keiywuvfwofw2• 8 months ago

This post was written by chatgpt????

https://blog.cloudflare.com/18-november-2025-outage/#:~:text...

boarush• 8 months ago

Not all em dashes are ChatGPT. Good writers use it wherever required.

keiywuvfwofw2• 8 months ago

This post was written by chatgpt??

https://blog.cloudflare.com/18-november-2025-outage/#:~:text...

oskarkk• 8 months ago

Here's a random post from their blog by the same author from 2017 with an em dash:

As we wrote before, we believe Blackbird Tech's dangerous new model of patent trolling — where they buy patents and then act their own attorneys in cases — may be a violation of the rules of professional ethics.

https://blog.cloudflare.com/patent-troll-battle-update-doubl...

ChatGPT didn't invent the em dash, some people were always using it. But yeah, it's often one of the signs of AI.

NetMageSCW7• 8 months ago

Too many think AI instead of horses.

dilyevsky2• 8 months ago

Long time ago Google had a very similar incident where ddos protection system ingested a bad config and took everything down. Except it was auto resolved in like four minutes by an automatic rollback system before oncall was even able to do anything. Perhaps Cloudflare should invest in a system like that

lofaszvanitt• 8 months ago

It is staggering to see that even large companies like CF have zero monitoring, so they would know what happened in t=0.

zeroq• 8 months ago

But Rust was supposed to cure cancer and solve world hunger. Is this the end of the hello world but in Rust saga?

thatoneengineer2• 8 months ago

The unwrap: not great, but understandable. Better to silently run with a partial config while paging oncall on some other channel, but that's a lot of engineering for a case that apparently is supposed to be "can't happen".

The lack of canary: cause for concern, but I more or less believe Cloudflare when they say this is unavoidable given the use case. Good reason to be extra careful though, which in some ways they weren't.

The slowness to root cause: sheer bad luck, with the status page down and Azure's DDoS yesterday all over the news.

The broken SQL: this is the one that I'd be up in arms about if I worked for Cloudflare. For a system with the power to roll out config to ~all of prod at once while bypassing a lot of the usual change tracking, having this escape testing and review is a major miss.

Xunjin• 8 months ago

Share the same opinion, as others pointed out, the status page down probably caused by bots checking it.

watchful_moose4• 8 months ago

It's probably not ok to silently run with a partial config, which could have undefined semantics. An old but complete config is probably ok (or, the system should be designed to be safe to run in this state).

vbezhenar3• 8 months ago

IMO: there should be explicit error path for invalid configuration, so the program would abort with specific exit code and/or message. And there should be a superviser which would detect this behaviour, rollback old working config and wait for few minutes before trying to apply new config again (of course with corresponding alerts).

So basically bad config should be explicitly processed and handled by rolling back to known working config.

jgilias3• 8 months ago

You don’t even need all the ceremony. If the config gets updated every 5 minutes, it surely is being hot-reloaded. If that’s the case, the old config is already in memory when the new config is being parsed. If that’s the case, parsing shouldn’t have panicked, but logged a warning, and carried on with the old config that must already be in memory.

bungle• 8 months ago

System outputting the configuration file failed (it could check the size and/or content and stop right away), but also a system importing the file failed. These usually sound simple/stupid in a hindsight. I am not a fan of everything centralising to a few hands. As in bad situation, they can also be weaponised or attacked. And in good situation their blast radius is just too big and a bit random, in this case global.

philipwhiuk5• 8 months ago

For unwrap, Cloudflare should consider adding lint tooling that prevents unwrap being added to production code.

groundzeros20157• 8 months ago

It’s a feature, not a bug. Assert assumptions and crash on bad one.

Crashing is not an outage. It’s a restart and a stack trace for you to fix.

twoodfin3• 8 months ago

The query is surely faulty: Even if this wasn’t a huge distributed database with who-knows-what schemas and use cases, looking up a specific table by its unqualified name is sloppy.

But the architectural assumption that the bot file build logic can safely obtain this operationally critical list of features from derivative database metadata vs. a SSOT seems like a bigger problem to me.

nijave3• 8 months ago

Quite surprising a single bad config file brought down their entire global network across multiple products

hbarka• 8 months ago

ClickHouse db was mentioned. Does this incident raise any critiques about it?

mmaunder• 8 months ago

tl;dr A permissions change in a ClickHouse database caused a query to return duplicate rows for a “feature file” used by Cloudflares Bot Management system, which doubled the file size. That oversized file was propagated to their core proxy machines, triggered an unhandled error in the proxy’s bot-module (it exceeded its pre-allocated limit), and as a result the network started returning 5xx errors. The issue wasn’t a cyber-attack — it was a configuration/automation failure.

jeffrallen2• 8 months ago

This is an excellent lesson learned: Harden loading of internally generated config files as though they were untrusted content.

Gonna use that one at $WORK.

kylegalbraith• 8 months ago

The outage sucked for everyone. The root cause also feels like something they could have caught much earlier in a canary rollout from my reading of this.

All that said, to have an outage reported turned around practically the same day, that is this detailed, is quite impressive. Here's to hoping they make their changes from this learning, and we don't see this exact failure mode again.

agentifysh• 8 months ago

how would you build redundancy around cloudflare failing?

i think this is happening way too frequently

meanwhile VPS, dedicated servers hum along without any issues

i dont want to use kubernetes but if we have to build mission critical systems doesn't seem like building on cloudflare is going to cut it

makach• 8 months ago

Excellent write up. Cybersecurity professionals read the story and learn. It’s textbook lesson in post-mortem incident analysis - a mvp for what is expected from us all in a similar situation.

Reputationally this is extremely embarrassing for Cloudflare, but imo they seem to get their feet back on the ground. I was surprised to see not just one, but two apologies to the internet. This just cements how professional and dedicated the Cloudflare team is to ensure stable resilient internet and how embarrassed they must have been.

A reputational hit for sure, but outcome is lessons learned and hopefully stronger resilience.

Martcpp• 8 months ago

deny (clippy:: unwrap_used)

agonux• 8 months ago

Time to rewrite with golang, explicit error handling ;-)

dev_l1x_be3• 8 months ago

Was it DNS this time?

MagicMoonlight• 8 months ago

Having a system which automatically deploys configuration files across a million servers every 5 minutes without testing it seems stupid to me.

anal_reactor• 8 months ago

Honestly... everyone shit themselves that internet doesn't work, but next week this outage will be forgotten by 99% of population. I was doing something on my PC when I saw clear information that Cloudflare is down, so I decided to just go take a nap, then read a book, then go for a walk. Once I was done, the internet was working again. Panic was not necessary on my side.

What I'm trying to say is that things would be much better if everyone took a chill pill and accepted the possibility that in rare instances, the internet doesn't work and that's fine. You don't need to keep scrolling TikTok 24/7.

but my use case is especially important

Take a chill pill. Probably it isn't.

vultour• 8 months ago

It's funny how everyone seems to be having a meltdown over this. I didn't even notice anything was wrong until I read about it on Reddit 5 hours later, even though I was working all day. Sounds to me like people are too reliant on random websites.

xlii• 8 months ago

Cloudflare rewrites Rust services to <next-cool-language> /joke

...

(I'd pick Haskell, cause I'm having fun with it recently :P)

HL33tibCe72• 8 months ago

An unwrap like that in production code on the critical path is very surprising to me.

I haven’t worked in Rust codebases, but I have never worked in a Go codebase where a `panic` in such a location would make it through code review.

Is this normal in Rust?

xnotcursed• 8 months ago

absolutely not normal, this is why in my opinion it took them so long to understand the core issue. instead of a nice error message and a backtrace saying something like "failed to parse config feature names" they thought they were under attack because the service was just crashing instead.

jokoon• 8 months ago

I don't understand what's the business of cloudflare.

They just sell proxies, to whoever.

Why are they the only company doing ddos protection?

I just don't get it.

BOOSTERHIDROGEN• 8 months ago

Momentum, I guess, pretty much like you wouldn’t get fired for using AWS or IBM (if that’s still the case now).

hnarn2• 8 months ago

That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.
The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.

I'm no FAANG 10x engineer, and I appreciate things can be obvious in hindsight, but I'm somewhat surprised that engineering at the level of Cloudflare does not:

1. Push out files A/B to ensure the old file is not removed.

2. Handle the failure of loading the file (for whatever reason) by automatically reloading the old file instead and logging the error.

This seems like pretty basic SRE stuff.

watchful_moose4• 8 months ago

Yep, a decent canary mechanism should have caught this. There's a trade off between canarying and rollout speed, though. If this was a system for fighting bots, I'd expect it to be optimized for the latter.

hnarn2• 8 months ago

Presumably optimal rollout speed entails something like or as close to ”push it everywhere all at once and activate immediately” that you can get — that’s fine if you want to risk short downtime rather than delays in rollout, what I don’t understand is why the nodes don’t have any independent verification and rollback mechanism. I might be underestimating the complexity but it really doesn’t sound much more involved than a process launching another process, concluding that it crashed and restarting it with different parameters.

kevincox2• 8 months ago

I'm shocked that an automatic canary rollout wasn't an action item. Pushing anything out globally is a guaranteed failure again in the future.

Even if you want this data to be very fresh you can probably afford to do something like:

1. Push out data to a single location or some subset of servers.

2. Confirm that the data is loaded.

3. Wait to observe any issues. (Even a minute is probably enough to catch the most severe issues.)

4. Roll out globally.

0x001D• 8 months ago

Configuration can be validated. https://cuelang.org

lalam3• 8 months ago

Hack Free fire 8000 5487 3565 644664 464664644

464646449

lalam3• 8 months ago

Hack

Free fire

lalam3• 8 months ago

niedbalski• 8 months ago

Sure this has been said but the issue here is not code is the ability to canary and rollback quickly from any arbitrary (config) change.

zyngaro• 8 months ago

Catastrophique failure for failing to read a file bigger that expected? Wow. This is really embarrassing.

sachahjkl• 8 months ago

me af when there's a postmortem rubbing hands, impish smile on my face

gkoz• 8 months ago

Given this was triggered by an old school configuration change across multiple servers, there's too little discussion of that particular process.

It sounds like the change could've been rolled out more slowly, halted when the incident started and perhaps rolled back just in case.

130R• 8 months ago

If the software has a limit on the size of the feature file then the process that propagates the file should probably validate the size before propagating ..

xyst• 8 months ago

A fucking unhandled exception brought down a majority of the internet? Why do we continue to let these clowns run a large portion of the internet?

Big tech is a fucking joke.

NetMageSCW7• 8 months ago

20% sure seems to grow with every commenter piling on. It’s like an Internet game of telephone.

elAhmo• 8 months ago

Timely post mortem. Sucks to have this happened, but at least they are quite transparent and detailed in the writeup.

cowsandmilk3• 8 months ago

Blog post from less than a week ago on how Cloudflare avoids outages on configuration changes: https://blog.cloudflare.com/finding-the-grain-of-sand-in-a-h...

This has to sting a bit after that post.

leonaves• 8 months ago

Why have a limit on the file size if the thing that happens when you hit the limit is the entire network goes down? Surely not having a limit can't be worse?

__alexs• 8 months ago

Is dual sourcing CDNs feasible these days? Seems like having the capability to swap between CDN providers is good both from a negotiating perspective and a resiliency one.

arkanovicz• 8 months ago

Interesting technical insight, but I would be curious to hear firsthand accounts from the teams on the ground, particularly regarding how the engineers felt the increasing pressure, frantically refreshing their dashboards, searching for phantom DDoS, scrolling codes updates...

l___l• 8 months ago

The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.

What could have prevented this failure?

Cloudflare's software could have included a check that refused to generate the feature file if it's size was higher than the limit.

A testcase could have caught this.

markhandoff• 8 months ago

Dear Matthew Prince, don't you think we (the ones affected by your staff's mistake) should get some sort of compensation??? Yours truly, a Cloudflare client who lost money during the November 18th outage.

NetMageSCW7• 8 months ago

What does your contract with Cloudflare say?

darksideofthem• 8 months ago

Speaking of resiliency, the entire Bot Management module doesn't seems to be a critical part of the system, so for example, what happens if that module goes down for an hour? the other parts of the system should work. So I would rank every module and it's role in the system, and would design it in a way that when a non-critical module fails, other parts still can function.

kjgkjhfkjf• 8 months ago

Seems like a substantial fraction of the web was brought down because of a coding error that should have been caught in CI by a linter.

These folks weren't operating for charity. They were highly paid so-called professionals.

Who will be held accountable for this?

igornadj• 8 months ago

Any feature failing should still allow the traffic to continue. This should be the first bullet in the future actions list.

BrtByte5• 8 months ago

This incident feels like a strong argument for stricter guardrails around internal config propagation

kqr2• 8 months ago

One of the remediations listed is

Eliminating the ability for core dumps or other error reports to overwhelm system resources

but this is not mentioned at all in the timeline above. My best guess would be that the process got stuck in a tight restart loop and filled available disk space with logs, but I'm happy to hear other guesses for people more familiar with Rust.

janpio• 8 months ago

I understood this to be related to this section:

As well as returning HTTP 5xx errors, we observed significant increases in latency of responses from our CDN during the impact period. This was due to large amounts of CPU being consumed by our debugging and observability systems, which automatically enhance uncaught errors with additional debugging information.

(Just above https://blog.cloudflare.com/18-november-2025-outage/#how-clo...)

kqr2• 8 months ago

Ah, yes, of course. I must have missed that. Thanks!

keepamovin• 8 months ago

That's interesting. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network. It's like the issues with HOSTS.TXT needing to be copied among the network of the early internet to allow routing (taking days to download etc) and DNS having to be created to make that propagation less unwieldy.

drc500free• 8 months ago

Makes me wonder which team is responsible for that feature generating query, and if they follow full engineering level QA. It might be deferred to an MLE team that is better than the data scientists but less rigorous than software needs to be.

mmaia• 8 months ago

Exactly. The post screams about all the issues I've seen in multiple companies between DS/MLE and SE/DevOps.

cmilton• 8 months ago

How many changes to production systems does Cloudflare make throughout a day? Are they a part of any change management process? That would be the first place I would check after a random outage, recent changes.

CSMastermind• 8 months ago

I'm honestly curious what culturally is going on inside Cloudflare given they've had a few outages this year.

Forgeties79• 8 months ago

I’ll be honest, I only understand about 30% of what is being said in this thread and that is probably generous. But it is very interesting seeing so many people respond to each other “it’s so simple! what went wrong was…” as they all disagree on what exactly went wrong.

baalimago2• 8 months ago

Interesting. Although principle of least privilege is great, it should not be applied as a feature to filter data.

baalimago2• 8 months ago

Git blame disabled on the line which crashed it - Cowards!

chaos_emergent• 8 months ago

Just a moment to reflect on how much freaking leverage computers give us today - a single permission change took down half the internet. Truly crazy times.

zf00002• 8 months ago

As an IT person, I wonder what it's like to work for a company like this. Where presumably IT stuff has a priority. Unlike the companies I've worked for where IT takes a backseat to everything until something goes wrong. Company I work had a huge new office built, with the plan it would be big enough for future growth, yet despite repeated attempts to reserve a larger space, our server room and infrastructure is actually smaller than our old building and has no room to grow.

rkomorn• 8 months ago

As a former CF employee, I'd say it's a mixed bag.

There are plenty of resources , yet it's somehow never enough. You do tons of pretty amazing things with pretty amazing tools that also have notable shortcomings.

You're surround by smart people who do lots of great work, but you also end up in incident reviews where you find facepalm-y stuff. Sometimes you even find out it was a known corner case that was deemed too unlikely to prioritize.

The last incident for my team that I remember dealing with there ended up with my coworker and I realizing the staging environment we'd taken down hours earlier was actually the source of data for a production dashboard, so we'd lost some visibility and monitoring for a bit.

I've also worked at Facebook (pre-Meta days) and at Datadog, and I'd say it was about the same. Most things are done quite well, but so much stuff is happening that you still end up with occasional incidents that feel like they shouldn't have happened.

jamesblonde• 8 months ago

Cloudflare tried to build their own feature store, and get a grade F.

I wrote a book on feature stores by O'Reilly. The bad query they wrote in Clickhouse could have been caused by another more error - duplicate rows in materialized feature data. For example, in Hopsworks it prevents duplicate rows by building on primary key uniqueness enforcement in Apache Hudi. In contrast, Delta lake and Iceberg do not enforce primary key constraints, and neither does Clickhouse. So they could have the same bug again due to a bug in feature ingestion - and given they hacked together their feature store, it is not beyond the bounds of possibility.

Reference: https://www.oreilly.com/library/view/building-machine-learni...

assbuttbuttass2• 8 months ago

My website was down too, because a tree fell on my power line

zhisme• 8 months ago

Thank you for being honest, all must learn from the mistakes.

Chihuahua0633• 8 months ago

The first automated test detected the issue at 11:31 and manual investigation started at 11:32. The incident call was created at 11:35.

I'm impressed they were able to corral people this quickly.

throw7• 8 months ago

Is this true: from that core proxy diagram, I didn't realize cloudflare sees the full unencrypted packet between you and the server.

If that's true, is there a way to tell (easily) whether a site is using cloudflare or not?

tempest_• 8 months ago

It is pretty easy to see if cloudflare is proxying a site.

Just ping the host and see if the ip belongs to CF.

https://www.cloudflare.com/en-ca/ips/

finally7394• 8 months ago

The NSA has to see the data somehow, right?

jjice• 8 months ago

There's (obviously) a lot of discussion around the use of `unwrap` in production code. I feel like I'm watching comments speak past each other right now.

I'd agree that the use of `unwrap` could possibly make sense in a place where you do want the system to fail hard. There's lot of good reasons to make the system fail hard. I'd lean towards an `expect` here, but whatever.

That said, the function already returns a `Result` and we don't know what the calling code looks like. Maybe it does do an `unwrap` there too, or maybe there is a save way for this to log and continue that we're not aware of because we don't have enough info.

Should a system as critical as the CF proxy fail hard? I don't know. I'd say yes if it was the kind of situation that could revert itself (like an incremental rollout), but this is such an interesting situation since it's a config being rolled out. Hindsight is 20:20 obviously, but it feels like there should've been better logging, deployment, rollback, and parsing/validation capabilities, no matter what the `unwrap`/`Result` option is.

Also, it seems like the initial Clickhouse changes could've been testing much better, but I'm sure the CF team realizes that.

On the bright side, this is a very solid write up so quickly after the outage. Much better than those times we get it two weeks later.

Matthias247• 8 months ago

When I first read about it I assumed it would have been a "poison pill" - a bad config where the ingestion of the config leads the process to crash/restart. And due to that crash on startup, there is no automated possibility to revert to a good config. These things are the worst issues that all global control planes have to deal with.

The report actually seems to confirm this - it was indeed a crash on ingesting the bad config. However I'm actually surprised that the long duration didn't come from "it takes a long time to restart the fleet manually" or "tooling to restart the fleet was bad".

The problem mostly seems to have been "we didn't knew whats going on". Some look into the proxy logs would hopefully have shown the stacktrace/unwrap, and metrics about the incoming requests would hopefully have shown that there's no abnormal amount of requests coming in.

ogurechny• 8 months ago

Given Cloudflare's importance in the Internet ecosystem any outage of any of our systems is unacceptable.

Excuse me, what you've just said? Who decided on “Cloudflare's importance in the Internet ecosystem”? Some see it differently, you know, there's no need for that self-assured arrogance of an inseminating alpha male.

arifiqbal81• 8 months ago

Thanks for the detailed writeup and explaining the root cause in details

However, I have a question from a release deployment process perspective. Why was this issue not detected during internal testing ? I didn't find the RCA analysis covering this aspect. Doesn't cloudflare have an internal test stage as part of its CICD pipeline. Looking the description of the issue, it should have been immediately detected in internal stage test environment.

flerchin• 8 months ago

Based on this writeup it seems that Cloudflare defaults to a 0 score for bot prevention if there's a failure. Could instead it default to a passing score? Default open instead of default closed? This would have been a non-event to a lot of websites if that change was made.

NetMageSCW7• 8 months ago

It feels like their list of after actions is lacking a bit to me.

How about 1. The permissions change project is paused or rolled back until 2. All impacted database interactions (SQL queries) are evaluated for improper assumptions or better 3. Their design that depends on database metainfo and schema is replaced with ones that use specific tables and rows in tables instead of using the meta info as part of their application. 4. All hard coded limits are centralized in a single global module and referenced from their users and then back propagated to any separate generator processes that validate against the limit before pushing generated changes

kwar13• 8 months ago

Perils of a centralized chokepoint. Substantial amounts of the internet shouldn't go down because Cloudflare had a hiccup.

realtyblocks3• 8 months ago

our site realtyblocks.com went down but we redirected the traffic by bypassing the cloudflare DNS routes with few clicks. Thank you for resolving the issue Cloudflare.

realtyblocks3• 8 months ago

our site realtyblocks.com went down but we redirected the traffic by bypassing DNS entries with few clicks. Thank you for resolving the issue Cloudflare.

realtyblocks3• 8 months ago

our site realtyblocks.com went down but we redirected the traffic with few clicks. Thank you for resolving the issue Cloudflare.

Diggsey• 8 months ago

There were two things I think went extremely poorly here:

1) Lack of validation of the configuration file.

Rolling out a config file across the global network every 5 minutes is extremely high risk. Even without hindsight, surely one would see then need for very careful validation of this file before taking on that risk?

There were several things "obviously" wrong with the file that validation should have caught:

- It was much bigger than expected.

- It had duplicate entries.

- Most importantly, when loaded into the FL2 proxy, the proxy would panic on every request. At the very least, part of the validation should involve loading the file into the proxy and serving a request?

2) Very long time to identify and then fix such a critical issue.

I can't understand the complete lack of monitoring or reporting? A panic in Rust code, especially from an unwrap, is the application screaming that there's a logic error! I don't understand how that can be conflated with a DDoS attack. How are your logs not filled with backtraces pointing to the exact "unwrap" in question?

Then, once identified, why was it so hard to revert to a known good version of the configuration file? How did noone foresee the need to roll back this file when designing a feature that deploys a new one globally every 5 minutes?

phyzome• 8 months ago

This is a small issue in the writeup (everything else made sense), but -- why does doubling 60 features exceed the 200 limit? Missing something.

harivyom3• 8 months ago

I am just going by the outage post mortem report. I could not read the article after I read the first few lines - "feature file". I am stuck at consuming the design idea here where you allow multiple inserts for one feature [assuming you have some uniqueness constraint]. Even a simple key-value map per feature should have made the insertions as just a put/replace the value. I think that was not the case here, where cloudflare kept appending to the file for any feature to be added. And I am assuming the features are bot attack patterns as features. Anyway, there is something fundamental here cloudflare should rethink. If someone can educate me on the design, I can continue reading the next lines.

harivyom3• 8 months ago

I am just going by the outage post-mortem report. I could not read the article after I read the first few lines - "feature file" expansion/limits. I am stuck at consuming the design idea here, where you allow multiple inserts for one feature [assuming you have some uniqueness constraint].

Even a simple key-value map per feature should have allowed for insertions as simple as a put/replace of the value and not appending to the file. That was not the case here, where Cloudflare kept appending to the file for any feature to be added. And I am assuming the features are bot attack patterns as features. Anyway, there is something fundamental here that Cloudflare should rethink. If someone can educate me on the design, I can continue reading the next few lines.

mayank94• 8 months ago

Thanks for the detailed RCA. After going through the blog post as well, there’s a curiosity about whether there are opportunities to detect such scenarios earlier in the testing phase. It would be helpful to understand if there are any differences between the test environment and production that might have contributed to this. This could provide insights into strengthening the process going forward.

tete2• 8 months ago

Bit funny. "Memory management" bug in Rust. (writing that as kind of a fan of Rust)

amitkumary96• 8 months ago

As we know that "With great power comes great responsibility" the team should understand this because Cloudflare is used worldwide, and for many countries it was the peak working time when Cloudflare went down so this affected massively. We always want perfect results but it's not possible, I hope the team is not overworking to get the changes on prod.

laotree• 8 months ago

Attempt to reproduce the Cloudflare 2025-11-18 outage.

Cloudflare's incident report is written clearly and explicitly, so based on my own understanding, I’m going to try reproducing this outage. Already completed:

CK cluster Permission change triggering data doubling Cache propagation Unaffected proxy services Proxy services with bot score errors

TODO:

unwrap panic during pre-allocation of cache Full demonstration of the entire outage process

https://github.com/Laotree/reproduce_cf20251118

pkumar000072• 8 months ago

If you knew the expected number of features , any input file with >100 should be discarded as bad input and you failback to the last good feature file received. This would have protected your service even though you are unable to get the newly populated features. I believe these features are not updated that frequently. Even if they were, you would have biased your system towards availability vs 'correctness'.

What were the teams doing between 11 to 1300 hrs , no explanation of what investigations were going on to not being able to figure the root cause.

pkumar000072• 8 months ago

Did your incident response team look at the last few changes that were executed? If they had , they could have just rolleback the change. or just looking at the changes executed, in the vicinity of the start of the outage could have pointed to the problem.

Didn't the services that were crashing due to OOM raise any alerts?

This is shitty at so many levels.

metasnitch02• 8 months ago

It was ASSUREDLY a cyber attack.

Go to jeffblearning on LinkedIn. I took it down with 253 copies of a text file delivered through a vulnerability in Novo’s systems.

I’ve documented all of it.

It’s not done yet…

proverbs53• 8 months ago

What I read is that a non-critical feature (blocking / managing bot - access) was able to impact a critical feature (routing traffic).

Shouldn't the architecture setup in such a way that subcomponents can fail without impacting the critical function of the component?

atari_guy• 8 months ago

It actually wasn't resolved quite that fast. My site continued to have issues for hours afterward before things finally resolved completely.

Cloudflare outage on November 18, 2025 post mortem

Related Stories