Partial Cloudflare outage on October 25, 2022

https://blog.cloudflare.com/partial-cloudflare-outage-on-october-25-2022/

by mfiguiere • 4 years ago

115 54 4 years ago

blog.cloudflare.com

dang• 4 years ago

Cloudflare CDN Partial Outage - https://news.ycombinator.com/item?id=33333332 - Oct 2022 (54 comments)

ec1096852• 4 years ago

Would love to learn more about why a tracing function mutates the request (clears headers). That seems like foot gun that would be impossible for anyone not intimately familiar with the function's implementation to avoid.

jgrahamc7• 4 years ago

Internal retro is going to be digging into that question (amongst others). I imagine we’ll update the post with further details.

Thorrez• 4 years ago

Mutating the headers might make sense if it needs to inject a trace ID into the headers. That doesn't explain why it would clear all headers though.

Cthulhu_• 4 years ago

I suspect it'll be a trivial programming error that passed through tests that didn't test whether it leaves other headers unaffected as well as code review.

jackblemming• 4 years ago

Surprised it both took them so long to decide to rollback and that the rollback lasted so long.

yjftsjthsd-h4• 4 years ago

Yeah,

Once identified we expedited a rollback which completed in 87 minutes.

I appreciate that it's a massive infrastructure spanning the entire globe, but... 87 minutes to revert a change? I wouldn't want the job of fixing it, but that doesn't seem good enough when the impact is that bad.

systemvoltage• 4 years ago

As an outsider without any knowledge of the company infra:

1) I wonder if they so many layers of CI/CD and testing, that the whole pipeline, even rollbacks probably re-run their tests which takes time. Are CI/CD runners the problem or it is something like network/DNS thing?

2) What if someone inside Cloudflare glibly yells "Why can't we just rsync our code to all data centers in less than a minute?" What would be the pushback this person would get? Why is it a stupid suggestion? Answering that would reveal some crazy layers of abstractions that get piled up in large companies.

douglasheriot• 4 years ago

What happens if a rollback somehow makes things worse? Even emergency rollbacks need a gradual rollout so you’ve got a chance to catch any new issues you didn’t expect.

hayst4ck3• 4 years ago

I don't know how cloudflare is configured, but doing anything fast to caches is generally a really bad idea. Caches often protect upstream data stores from overload. If an upstream data store becomes overloaded with too much traffic, then all requests that use that data store will start to slow down. As requests slow down, they will either timeout and fail resulting in major error spikes or they will start to consume all the resources of the service making requests.

So if any caches were cleared in the process of rolling back, a slow rollback makes perfect sense to me.

I would expect a company like cloudflare would have previous revisions of software on their machines that could be reverted to near instantly.

Another thing to consider is what capacity they run at and how long a service takes to initialize. If you can only take 10% of your machines out at any given moment and each one takes a while to initialize, you don't really have much choice in the matter.

Lastly, if you look at the 530 graph, it seems that the drop in 530's was almost instant.

2022-10-25 17:38: An accelerated rollback continues with large data centers acting as Upper tier for many customers.

This leads me to believe they shifted traffic away from the low tiers, which enabled them to roll back as slowly as they like at the cost of a likely minor increase in latency.

dspillett2• 4 years ago

87 minutes to revert a change?

If their processes require a suite of tests be run on staging systems before the revert hits production, that may take considerable time depending on how much is potentially affected.

Also the 87 minutes might be for everything, perhaps it was a rolling update: update some nodes, test, update the next batch, repeat. This might mean that a good chunk of the infrastructure was updated for sooner and traffic could be diverted through those parts (presumably those nodes are over-specified, or just easily scaled, to be able to take a glut of extra load - as they would need to in more normal circumstances in order to contend with waxing & waning traffic patterns).

Perhaps it is even the case that the errant update didn't get out to all areas before the problem became apparent, so they were not rolling back over the whole network. Again this may offer the opportunity to divert load to non-affected areas while the others are rolled back.

So while the full revert took 87 minutes, things may have been up and running properly (except perhaps a little slower than usual) for most users/processes much quicker than that.

mritzmann• 4 years ago

[..] and that the rollback lasted so long

That does not surprise me. There are companies working with CI/CD systems whose tests easily take more than an hour. And until they are completed, they cannot start releasing (enforced via system permissions). This is a good thing, but the possibility of an emergency rollback should be considered beforehand and maybe (it always depends) exceptions made for such purposes.

And sometimes it makes more sense to revert slowly to make sure it doesn't get worse.

stevewatson301• 4 years ago

The incident report is very light on details. For example, it mentions adding instrumentation caused the issue, but no details as to how why they'd need to remove headers or how they'd contain it in the future, except for that they'd need to "fail fast" in this situation.

(You can obviously catch it with a blue-green deployment, which they do mention, but at that point some small portion of traffic has already been affected.)

jgrahamc7• 4 years ago

This incident report was out within hours of the failure. The team is now doing an internal retro to look at decision speed, rollback speed, and the circumstances around that function having a side effect. We'll update the post with more detail.

cesarb• 4 years ago

We'll update the post with more detail.

Just a small suggestion: there's nothing in the post (unless I missed something) that mentions or even implies that this is not the "final report". You could have a small paragraph at the end saying "we might update this post further once we have more details", or even better, have a bullet point list of the things your team is still looking at. That way, readers of the blog post would know it might be worth it to come back later and read the post again.

stall84• 4 years ago

Yes they probably can fill in plenty of detail, but overall I was impressed with the incident report on it's own.. I don't think it's super common for companies' their size to release a timeline/report on the fkup unless they're coerced to.

dspillett2• 4 years ago

For example, it mentions adding instrumentation caused the issue, but no details as to how why they'd need to remove headers

They probably don't need to at all, but I can think or several ways it might unintentionally happen.

Perhaps someone chose to implement the modification as “separate headers & body, modify headers, put the request back together” and something went wrong that made the putting-back-together fail quietly (so the now malformed request passes to the next part without an exception being raised) or not get called at all in some circumstances. Alternately a badly coded routine to remove a header, perhaps the instrumentation information after the point in the processes it is no longer useful, maybe removed more than it should.

How such things get past a code review without someone suggesting a less error-prone method might be a question that gets asked in the internal investigation. I assume somewhere like CF has many checks between initial code modifications and things dropping into production, unless they've really taken “fail fast” to heart!

londons_explore2• 4 years ago

Once identified we expedited a rollback which completed in 87 minutes

Roll forwards and back should usually be done slowly and carefully.

However, during a major outage, there is a good reason to do the rollback very fast. Ie. Within 1 minute. Fast rollbacks end any outage sooner. And if your service is already down, you have nothing to lose and everything to gain.

That provides an interesting infrastructure problem. Fast rollbacks probably involve restarting a large number of servers, having a large number of cold caches, etc. You can find new bugs that only rear their heads during a fast rollback. Things like docker registries getting overloaded, kubernetes control plane being overloaded, other services becoming unhealthy due to a large influx of requests, etc.

The traditional approach is 'dont take risks, don't do a fast rollback'. But that can double or more the length of your downtime. Especially if you roll back the wrong thing first.

I'd therefore encourage preparing ahead of time for fast rollbacks. The dev/test environment should automatically do 'big bang' rollbacks every week. Production rollbacks should even be practiced from time to time, getting faster and faster each time, keeping an eye on metrics to look for instability.

avereveard• 4 years ago

The impact lasted for almost six hours in total.

And faster monitoring, they detected the issue after 4.5 hours and took one to fix

oefrha2• 4 years ago

And if your service is already down, you have nothing to lose and everything to gain.

They had a 5% request failure rate at the peak of this outage. So a botched rollback could definitely do more harm than good if the impact isn’t 100% understood.

jgrahamc7• 4 years ago

Exactly.

londons_explore2• 4 years ago

But there is a good chance that was a 100% request failure rate for a specific service. And when you have 100% of requests failing for a service, you ought to be thinking about doing a fast rollback for that service as long as you have confidence that such a fast rollback won't take other infrastructure down.

quickthrower22• 4 years ago

Another chaos monkey case? Just do a fast rollback every day your D10 rolls a 1.

vitus• 4 years ago

There have been YouTube outages that were exacerbated precisely because the SRE teams explicitly trained for the disaster scenario where everything is hard down and needs to be brought back as quickly as possible.

iancmceachern5• 4 years ago

A bit of an aside, but maybe not.

I live on the same block as the Cloudfare headquarters. They have a big common space that has open windows to the street. They have some big projectors that play what are basically Screensavers on the wall in this common space, visible from the street. For months now one of these big wall sized projections has had in large white letters "this copy of office is not legit, please contact Microsoft to purchase a legit license of windows " (I paraphrase a bit). This seems like something they should address, no?

rat9988• 4 years ago

Have you ever considered contacting them instead of public shaming?

iancmceachern5• 4 years ago

It's not my job. They know this is an issue, it's literally written on their walls. If they wanted to address it they would have.

Edited to add - I did, the night guard didn't seem to care or even know who to bring the issue to.

fogolon• 4 years ago

It doesn't surprise me that they are heavy on the marketing and light on the detail. Also fits with every outage they've had large enough for others to notice, the root cause is always some embarrassingly simple cock-up that only an intern would make.

iancmceachern5• 4 years ago

I'm replying to my own comment to note that I'm getting downvoted for calling out a big company for what seems like software theft. That seems odd as I'm seemingly not the one in the wrong, I'm just sharing a simple observation.

jgrahamc7• 4 years ago

I doubt you're getting downvoted for that. I suspect you're getting downvoted for your comment having nothing to do with the partial outage. And then going on to argue that you're doing the right thing by posting here and spending effort commenting about your comment rather than telling someone at Cloudflare that you spotted something bad.

oefrha2• 4 years ago

You’re free to start a Tell HN or write a blog post and post the link here for people to discuss. But hijacking of unrelated threads is not welcome on HN. Check guidelines.

quickthrower22• 4 years ago

It might be “software theft” but it could be a simple mistake with the license, which can happen if you reinstall or change hardware or something like that, and they need MS support to fix it.

nottorp• 4 years ago

I strongly doubt they don't have enough office licenses. What's probably happening is no one can be bothered to fix the user hostile copy protection.

I bet cracked versions of Office don't display that message, so it's an indication that this one is actually legit :)

daniel_iversen• 4 years ago

Is this why Apple's iMessage was down for a bit - are they maybe using CloudFlare?

DeathArrow• 4 years ago

That's why Whatsapp was down, too?

sammy2255• 4 years ago

hayst4ck3• 4 years ago

87 sounds large

   2022-10-25 17:03: CDN Release is rolled back in Atlanta and root cause is confirmed.
   2022-10-25 17:28: Peak impact with approximately 5% of all HTTP requests resulting in an error with status code 530.
   2022-10-25 17:38: An accelerated rollback continues with large data centers acting as Upper tier for many customers.
   2022-10-25 18:04: Rollback is complete in all Upper Tiers.
   2022-10-25 18:30: Rollback is complete.

But this seems like the 87 minutes referred to.

The 503 graph appears to show a steep drop between 17:38 and and 17:45, which looks like the "rollback" speed to me.

I think they likely drained many of the lower tier caches/pops/edges after which roll back speed doesn't matter very much since the machines being rolled back likely weren't servicing traffic.

large data centers acting as Upper tier for many customers.

To me this sounds like a very awkwardly worded way to say "we shifted traffic".

Clarification on whether traffic was shifted or tiers were drained would be nice.

hayst4ck3• 4 years ago

Additionally, this was a code release based outage(?). It seems like a graph annotated with releases (or a/b testing changes) would have made this outage somewhat trivial to identify.

  2022-10-25 14:39: Multiple teams become involved in the investigation as more customers start reporting increases in errors.
  2022-10-25 17:03: CDN Release is rolled back in Atlanta and root cause is confirmed.

This is what I find worrying.

How does cloudflare track their code releases and does cloudflare annotate their 4 major graphs (errors, rps, utilization, latency) with lines showing when things were pushed to prod?

yjftsjthsd-h4• 4 years ago

I dunno; they say

At first glance, the errors looked like they were caused by a different system that had started a release some time before.

And with somewhat noisy error rates I'm not sure even annotating graphs with releases would have made it unambiguous.

eis• 4 years ago

It seems to me that the rollout process is flawed.

A change should be rolled out to a small fraction of nodes and then monitored for an extended period. 5% of requests returning errors would easily be spotable. Only when a new release has run stable on that portion for some time should it proceed to a bigger subset. You probably want to do this in several steps at the size of CF.

We also learn that it took customer reports to get the investigations rolling but 5xx errors are easily monitored so it points at internal monitoring being lacking even though it's hard for me to believe that they don't have an eye on this already.

It's not the first time that a deploy has brought CloudFlare (partially) down. From the timeline we see there's several hours between incident investigations starting and the rollout being stopped. The rollback should be the first thing considered even before looking at what the actual issue is.

Ideally you have someone sitting next to a red rollback button during a rollout whose only job is it to have an eye on all automatic and customer error reports. :)

ec1096852• 4 years ago

They did a canary deploy in a data center that didn't have hierarchical caching enabled, so they missed this code path.

sideproject• 4 years ago

Unfortunately, my sites are still not ok. I'm getting HTTP 409 errors on some of the domains that CNAME to my Cloudflare domains. I've looked everywhere and I still cannot get this resolved.

jgrahamc7• 4 years ago

Can you email me (jgc) details?

wahnfrieden• 4 years ago

Related: anyone know the easiest free/self hosted alternative to cloudflare tunnels?

yamtaddle• 4 years ago

I'm struggling to understand exactly what it does (via my employer I'm an admin of a CloudFlare account and we use some features outside DNS and proxy-CDN, but far from all of them) but... I wanna say Wireguard?

Of course, anything else is probably gonna have worse UX, because it won't have all the infra pre-built and auto-configured for you.

Also you'll need at least one publicly-accessible server to be the Internet-facing side of it, and that part would necessarily not be free, unless you can get by on some free-tier VM somewhere.

dannyw• 4 years ago

Love Cloudflare or not, the transparency and frankness from the CTO in comments sections like these, is a rare and welcome sight.

teknopaul• 4 years ago

I can imagine why this took a while to fix.

It was not DNS.

Thaxll• 4 years ago

5% is it because you inject traces in 5% of requests?

Partial Cloudflare outage on October 25, 2022

Related Stories