1.1.1.1 lookup failures on October 4th, 2023

https://blog.cloudflare.com/1-1-1-1-lookup-failures-on-october-4th-2023/

by todsacerdoti • 3 years ago

226 65 3 years ago

blog.cloudflare.com

suprjami• 3 years ago

Strangely I noticed this because some parts of eBay stopped loading. I spent a while troubleshooting my privacy/adblock nonsense because surely CloudFlare couldn't be down but that's the only conclusion I could come to.

ryanjshaw• 3 years ago

I thought it was a WSL (Linux on Windows) issue. I did some googling and saw some people discussing DNS issues on WSL, which were to be resolved by setting up a manual DNS config.

They used 8.8.8.8 in their example, which I of course changed to 1.1.1.1!

(Un)surprisingly the issue did not resolve after this, and I went on to do other things, hoping the gremlins would find something better to do in the meantime.

ChrisArchitect• 3 years ago

Earlier discussion while outage was active: https://news.ycombinator.com/item?id=37763143

morugam• 3 years ago

We noticed this through our own, homegrown scripts that check for this, having been screwed by an outage a few years ago. I'm happy they so quickly acknowledge and explain these issues. Good work!

LeoPanthera• 3 years ago

Did 1.0.0.1 also go down? The article doesn't say.

homero3• 3 years ago

Of course it did it's the same service

toast02• 3 years ago

A highly reliable service might run one partition on a completely separate serving stack. It's worth asking.

jacurtis2• 3 years ago

I mean, if its truly the same then it sort of defeats the purpose.

That's my way of saying: it is seperate infrastructure

Remember that Cloudflare is probably the second largest DNS Resolver in the world. They aren't just going to tack two IP's onto the same system. The entire system would be completely independent for reasons exactly like what happened today.

tedunangst3• 3 years ago

Visit 1.1.1.1 from any device to get started with our free app that makes your Internet faster and safer.

Ironic.

denysvitali4• 3 years ago

My only concern:

7:57 UTC: first reports coming in

I noticed this issue quite quickly ("reported" at 7:54 UTC [1]), and I noticed I wasn't alone thanks to Twitter / X. I tried to get in touch with Cloudflare to report this issue - but I haven't found any meaningful contact other than Twitter.

For such an important service, I'm impressed there is no contact email / form where you can get in touch with the engineers responsible for keeping the service up and running.

Other than that, kudos for the well written blog post - as always!

[1] https://nitter.net/DenysVitali/status/1709476961523835246

hackmiester2• 3 years ago

Get what you pay for I guess.

cj• 3 years ago

This sounds so foreign to my experience.

Even when I just had 2 paid domains with Cloudflare, there was real time chat support that seemed to always have an available engineer when something came up.

Maybe they don't provide support like that to free users.

abraham• 3 years ago

You have to pay $200/month for access to live chat support.

https://www.cloudflare.com/plans/

denysvitali4• 3 years ago

I don't even have a Cloudflare account. You don't need a Cloudflare account to use their public DNS servers

lopkeny12ko2• 3 years ago

Do you truly believe that the turnaround time for you contacting support, speaking to a first-tier support agent, having the request escalated to engineering, having a support engineer triaging it to the right team, and finally having the report being gauged so urgent that it pages the oncall for that team, is faster than automated infrastructure monitoring that the teams already have?

For something as core as DNS, I'm sure engineers were aware about the issue within minutes. There's a lot of politics and processes between that and public acknowledgment of an incident.

denysvitali4• 3 years ago

The timeline doesn't mention any internal monitor system but "external reports".

What I want as an engineer is an engineering contact (similar to a NOC) that "normal customers" aren't aware of.

Something as critical as DNS should have that - but maybe I expect too much from a free service.

hackmiester2• 3 years ago

But you are sure of this, even though there were no indications in that direction, and several in the opposite.

sambazi3• 3 years ago

For such an important service, I'm impressed there is no contact email

cloud: you are the product, not the customer

dx0342• 3 years ago

I don't think that statement makes sense here. In what way is the user of 1.1.1.1 the product? And for cloud services in general, they mostly tend to be subscription or usage based. In both cases, the user is clearly the customer.

I get the "you're the product, not the customer" for social networks or ad-based search engines, but certainly not for cloud products.

Wallacy2• 3 years ago

Discord is the place to go. All CF Devs are there.

I got on particular fix one time in just a few minutes, and scalated pretty quickly. I got a recomendation fix on my side (a code change on my deployment that cover the problem) and a permanent fix on CF side 4h latter (time that took to changes propagate to all colos)

araes• 3 years ago

I like how its a 42 joke.

4(0b10) 7:00 ends at 11:02 (4 hr 2 min) on a 4 sum 2x2. And refs to 1.1.1.1 vs 1.0.0.1

robhlt• 3 years ago

The lack of additional alerts in the Remediation section is a little bit concerning. Adding an alert for serving stale root zone data is great, but I think a few more would be very useful too:

- There's a clear uptick in SERVFAIL responses at 7:00 UTC but they don't start their response until an hour later after receiving external reports. This uptick should have automatically triggered an alert. It can't have been within the normal range because they got customer reports about it.

- The resolver failed to load the root zone data on startup and resorted a fallback path. Even if this isn't an error for the resolver it should still be an alert for the static_zone service, because its only client is failing to consume its data.

- The static_zone service should also alert when some percentage of instances fail to parse the root zone data, to get ahead of potential problems before the existing data becomes stale.

RockRobotRock2• 3 years ago

not to be glib but you should consider working for them

throwawaaarrgh• 3 years ago

If you can see shit falling apart and you're not even inside the org, it's probably a tire fire, and working there will just be stressful

xcdzvyn• 3 years ago

Sure. CloudFlare recruiters, DM me :-)

jrockway• 3 years ago

SERVFAIL might not be a good enough signal for alerting. I definitely see third-party DNS providers returning SERVFAIL for their own reasons; if it's a popular host (looking at you, Route 53), then you'll proxy those through and end up alerting on AWS's issue instead of your own.

They might just want a prober; ask every server for cloudflare.com every minute. If that errors, there are big problems. (I remember Google adding itself as malware many many years ago. Nice to have a continuous check for that sort of thing, which I am sure they do now.)

physicles• 3 years ago

They might just want a prober; ask every server for cloudflare.com every minute. If that errors, there are big problems.

Yeah this was my first thought too. Why don't they have such a system in place for as many permutations of their public services as they can think of? We're a small company and we've had this for critical stuff for several years.

lopkeny12ko2• 3 years ago

The alerts you suggested are sufficiently obvious that I'm sure the team has already implemented, or plans to implement, them. The public postmortem report is likely just a small snippet of the more interesting remediation actions.

AndyMcConachie• 3 years ago

The real problem here is that things started failing on September 21, but no one noticed until October 4th. Why was there no logging when resolvers started failing to load the local root zone on September 21?

homero3• 3 years ago

This got me. I spent an hour trying to figure out why my Internet seemingly went down but not fully

ktta• 3 years ago

I'm expecting @tptacek to come in anytime reminding us of his thoughts on DNSSEC

https://sockpuppet.org/blog/2015/01/15/against-dnssec/

dylukes• 3 years ago

People who are actually correct on the basis of significant experience don't tend to feel it necessary to remind others at every possible occasion (unless not doing so would lead to significant harm to others).

They're usually busy doing other things.

sambazi3• 3 years ago

"Grant me the serenity to accept the things I cannot change, the courage to change the things I can and the wisdom to know the difference."

but yea, repeatedly running into a wall of ignorance encourages shifting to "busy doing other things"

tptacek• 3 years ago

unless not doing so would lead to significant harm to others

Ding ding ding.

At any rate: it's very funny that DNSSEC took 1.1.1.1 down, but this bug can't honestly be pinned on DNSSEC itself.

jolux2• 3 years ago

The buried lede here seems to be that this is yet another serious outage (indirectly) caused by using DNSSEC, though I understand why they don’t emphasize this part, given their strong advocacy for DNSSEC adoption.

dc396• 3 years ago

As I read it, DNSSEC signature expiry was what triggered people noticing the root data was stale. That would seem to be a somewhat positive outcome.

ipaddr• 3 years ago

Would you rather an error or stale dns?

jolux2• 3 years ago

DNSSEC signature expiry was what triggered people noticing the root data was stale

If you noticed your brakes had failed because you ended up in a ditch I wouldn't really say that's a positive outcome.

Frankly I can't believe they don't have better monitoring for a system as critical as that.

mcesch2• 3 years ago

I don't know why Cloudflare, like Amazon, often get a free-pass on HN for their DNS implementation bugs. Regardless of DNSSEC's merits or otherwise, this bug isn't inherent to DNSSEC.

icecap12• 3 years ago

These guys always have the best write-ups. I aspire to write postmortems like they do. In fact, maybe my next context for a ChatGPT query will be “you are a member of of CloudFlare’s public relations team…”

All joking aside, it occurred to me that the vast majority of internet and even tech users know very little about DNS. For the longest time, I was in the same boat. After having been in a role where it was necessary to understand the record types and deploy DNS configs, I’m quite thankful I learned. Just remember…it’s always DNS.

edit: typo

jacurtis2• 3 years ago

I aspire to never write a postmortem again.

dx0342• 3 years ago

I love the visualisations. It's really hard to make visualisations that are easy to read, look pretty and are still technically accurate. Cloudflare really excels at that.

indigodaddy• 3 years ago

Would this have affected any authoritative DNS served by them (eg and also taking 1.1.1.1 out of the equation..)

Thinking not correct?

globular-toast• 3 years ago

Some people on my team experienced random breakage around 3rd party 2FA yesterday during this time period. I wouldn't be at all surprised if this was the cause, somehow. I've no idea how, though. It's so easy to become dependent on single points of failure without realising.

minusf• 3 years ago

1.1.1.1 has a WebAssembly app called static_zone running on top of the main DNS logic that serves those new versions when they are available.

webassembly? what is that word even doing in a post mortem about DNSSEC failures?

spoiler• 3 years ago

Wasm runtimes are great for stuff like plugins! Seems like this static_zone thing was something like that (but they call them apps)

Wallacy2• 3 years ago

CF Workers (that runs WebAssembly) are all over place. They may not run the main logic (not the actual Ngix, or DNSEC code) but they are used for several maintaince tasks.

nilslice• 3 years ago

worth a read! https://blog.cloudflare.com/big-pineapple-intro/

vb-8448• 3 years ago

So, another monitoring issue here.

It's incredible how far the SW industry gone in the last decades, but the way monitoring is done is the same as in the 90's.

1.1.1.1 lookup failures on October 4th, 2023

Related Stories