1.1.1.1 outage explanation

https://blog.cloudflare.com/today-we-mitigated-1-1-1-1/

by adamch • 8 years ago

459 84 8 years ago

blog.cloudflare.com

walrus01• 8 years ago

Commendable honesty and level of detail in the public RFO.

Understandable outage. I switched back to cloudflare when it came back up, but this did prompt me to drop 8.8.8.8 in as a 3rd fallback.

gcommer• 8 years ago

This is a great write up. It's also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy rather this sort of smart DDoS mitigation that drops traffic: DNS is so critical that any risk of dropping real traffic is unacceptable. (obviously, such a scale is impractical for 99% of services)

A good takeaway from this outage for the average user would be to make sure that your fallback DNS resolvers are operated by totally separate providers. (eg, configure 1.1.1.1 with 8.8.8.8 as a fallback, rather than 1.1.1.1 and 1.0.0.1) (Edit: fixed cloudflare's secondary address)

Dylan168072• 8 years ago

(the cloudflare secondary is 1.0.0.1)

cmurf• 8 years ago

Cloudfare makes it weirdly difficult to find this. 1.1.1.1 is plastered over many pages but not concomitant with the secondary.

berti• 8 years ago

Or 9.9.9.9 if you're not as comfortable with Google services.

spondyl• 8 years ago

For anyone wondering, 9.9.9.9 is https://www.quad9.net/ who claim to not only resolve requests but also check them against IBM X-Force's threat intelligence database.

joveian2• 8 years ago

At least for personal use, OpenNIC is nice and many of the servers say they do not keep logs. I use the 185.121.177.177 (2a05:dfc7:5::53) anycast server and it works well. They are more likely to disappear randomly than the ones run by large companies.

https://servers.opennic.org/

h1d• 8 years ago

Not if you're not comfortable giving up so many of your internet activity to Google but as others have pointed out there are alternatives to spread.

bluedino• 8 years ago

I’m amazed at how many people abandoned other providers and blindly switched to 1.1.1.1

I can’t even use that address with the ISP Alestra in Mexico

toomuchtodo• 8 years ago

The lack of operational knowledge is real, and there is no easy fix other than resilient defaults by a benevolent dictator (in this case, resolvers across netblock and provider demarcations) or spending the time to educate yourself first on the “why” then the “how”.

crtasm• 8 years ago

I read on the Pi-Hole forums that 'fallback' is a misleading term because clients don't work that way - they will happily spread requests between two functioning DNS servers. Can anyone confirm this or provide further insight?

mjcl• 8 years ago

Different OSes handle it differently. Windows tries the primary, waits 1 second and then starts trying secondaries.

TravelTechGuy• 8 years ago

Can confirm. If I have a secondary DNS specified in my router, other than my pi-hole, it becomes useless. My guess is the router is either measuring response time, and goes with the most efficient, or otherwise round-robining the requests. Either way, requests bypass the pi-hole in such quantities that it became useless.

PS: someone here mentioned that this behavior is OS-dependent - nope, this happens on the router level, and all devices in my apartment suffer.

cookiecaper• 8 years ago

Most systems do not round-robin by default afaik. They just try the listed DNS servers in order (and resolv.conf will only respect the first 3 IIRC).

DNS fallback is often also misunderstood to mean "fall back if the domain is not found", but it really means "fall back if the name server fails to respond". If the domain is not found (i.e. servers returns a valid NXDOMAIN response), most resolvers do not consult any other name servers.

Round-robin, cascading name lookups, and other frequently desired functionality can be obtained through dnsmasq or similar caching/forwarding name servers.

LinuxBender2• 8 years ago

If your recursive DNS servers are Unbound DNS, they will keep note of which infra (upstream forwarder) nodes are responding and will blacklist the ones that do not. You can see this with

    unbound-control dump_infra

After the infra ttl expires, they will probe the down nodes. If they respond, then traffic will be distributed among them again.

iforgotpassword• 8 years ago

Dnsmasq has different modes if you give it multiple upstream resolvers. By default it only queries the fastest one, which it determines by every now and then sending a query to all the servers and picking the one that replies first. You can tell it to always query all servers, or always query them in the order specified.

throwaway2048• 8 years ago

another decent alternative fallback with the same featureset is quad9

https://www.quad9.net/

kondro2• 8 years ago

The core feature of Cloudflare's DNS is getting you closer and faster resolves to their CDN (probably one of the largest at this point, barring the established enterprise encumbants) in a privacy-first way.

Quad9's goal seems to be about threat-detection and prevention.

It would be nice if you could have both simultaneously (and maybe you can), but at the moment both services are actually quite different.

grimgrin• 8 years ago

Maybe check out what NS1 is doing with DNS

https://ns1.com/

bigiain• 8 years ago

9.9.9.10 is "quad nine without the threat detection" from memory.

saghm• 8 years ago

Our FRP framework allows us to express this in clear and readable code. For example, this is part of the code responsible for performing DNS attack mitigation: > def action_gk_dns(...): [...] if port != 53: return None > if whitelisted_ip.get(ip): return None > if ip not in ANYCAST_IPS: return None > [...]

What does this code sample have to do with FRP? This code seems extremely trivial and doesn't give any real indication to me why you'd need a framework of any sort. It seems like they really want to emphasize that they use FRP, but this code just seems completely unrelated.

Chris911• 8 years ago

From one of the linked presentations: https://github.com/cloudflare/gatelogic

dingo_bat• 8 years ago

TL;DR: we should have used an IP that is not traditionally used for testing and internal stuff by everybody including Cisco.

Dylan168072• 8 years ago

Not even close. The system had a glitch because they were doing a major DNS resolver at all. It had nothing to do with the baggage that comes with 1.1.1.1 specifically.

isostatic• 8 years ago

However a few days ago there was a bgp based outage on cloudfare DNs want there?

SmellyGeekBoy• 8 years ago

Sounds like it was truly a "DR" in your case.

sp3322• 8 years ago

They were given the IP block by ARPA specifically with an agreement that they would analyze the junk traffic and report back on it. No one attempting this before has been able to keep up with the amount of noise pointed at that IP. Everyone involved knew exactly what they were getting into.

drawkbox• 8 years ago

Great information on the outage. Looks like another version 2 syndrome side effect [1].

Today, in an effort to reclaim some technical debt, we deployed new code that introduced Gatebot to Provision API.
What we did not account for, and what Provision API didn’t know about, was that 1.1.1.0/24 and 1.0.0.0/24 are special IP ranges. Frankly speaking, almost every IP range is "special" for one reason or another, since our IP configuration is rather complex. But our recursive DNS resolver ranges are even more special: they are relatively new, and we're using them in a very unique way. Our hardcoded list of Cloudflare addresses contained a manual exception specifically for these ranges.
As you might be able to guess by now, we didn't implement this manual exception while we were doing the integration work. Remember, the whole idea of the fix was to remove the hardcoded gotchas!

When porting legacy code it is not only important to understand the edge cases and technical debt built up over time, but to test more heavily in production because you never know if you got them all because some smart guy built them long ago and/or there are unknown hacks that were cornerstones of the system for better or worse.

Phased and alpha/beta rollouts in an almost A/B testing way is good for replacement systems. Version 2 systems can also add new attack vectors or other single points of failure that aren't as know from legacy problems, the Provision API seems like it is a candidate for that.

Over time the Version 2 system will be hardened just before it is EOL and replaced again to fix all the new problems that arise over time. Version 2's do innovate but they also shroud fixing old issues and pain points for new unknown problems.

[1] https://en.wikipedia.org/wiki/Second-system_effect

teddyh• 8 years ago

“Yes, I know, it’s just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I’ll tell you why: those are bug fixes. One of them fixes that bug that Nancy had when she tried to install the thing on a computer that didn’t have Internet Explorer. Another one fixes that bug that occurs in low memory conditions. Another one fixes that bug that occurred when the file is on a floppy disk and the user yanks out the disk in the middle. That LoadLibrary call is ugly but it makes the code work on old versions of Windows 95.

Each of these bugs took weeks of real-world usage before they were found.”

— Things You Should Never Do, Part I, (https://www.joelonsoftware.com/2000/04/06/things-you-should-...)

slededit• 8 years ago

Engineers realized that maintaining all those bug fixes was too expensive. So as an industry we collectively agreed everyone will do constant rewrites.

That win 95 fix? Doesn't matter because win 95 was auto upgraded to be unrecognizable (SaaS is a wonderful thing).

Low memory? Not our problem - go buy another computer. Remember programmer time is more valuable.

You can try and buck the trend but your dependencies wont and the customer doesn't care whose code the bug is in. You won't get any browny points so you might as well just save the effort.

It's a brave new world.

cm21872• 8 years ago

Do they just use python as pseudo code or do they actually run their attack detection in python?

usmannk• 8 years ago

It is python. They linked a presentation[1] and a talk about the bot.

[1] https://speakerdeck.com/majek04/gatelogic-somewhat-functiona...

cm21872• 8 years ago

It's interesting. I would have expected them to use rather something low latency/high performance like c++ or erlang given their scale and performance criticality.

askz• 8 years ago

Any supplementary thoughts on the fact they are running their attack detection in python? What's the point?

Spare_account• 8 years ago

There is a reply in a parallel chain (timestamped after your comment) which expands on the line of thought: https://news.ycombinator.com/item?id=17204069

decker• 8 years ago

I wonder if they are going to factor the availability into all the blog posts about how fast they are.

obeattie• 8 years ago

Major credit to Cloudflare for publishing a clear, honest, and detailed description of what happened. I wish more companies would do this.

One thing I’d be interested to know more about is why it took 17 minutes to fix. While you can and always should strive to make them less likely, outages are inevitable, so how you respond is crucial. Here the outage was very obviously caused by a deployment that I’d assume was supervised by humans – why did it take 17 minutes to roll back?

blackbrokkoli• 8 years ago

I'm not an expert, but is 17 minutes for:

- shit is not working

- is this an attack?

- no it's us

- how?

- that's how

- let's go back

- have to get supervisor

- roll back huge thing

really that long?

dx0344• 8 years ago

With ~150 data centres, roll back alone probably took 5-10 minutes. Don't think 17 minutes is that long.

jacksmith21006• 8 years ago

In 2000 the answer would be no. In 2018 I think it is.

Things change in a time when you would freak getting up in the morning and say google.com did not work.

phyzome• 8 years ago

Yes, this exactly. And to expand on what happens around that:

- monitoring system picks up irregularity (smoothed over some window of time, which delays alerting)

- alert propagates to humans

- humans may take time to notice alert (even a page takes a few seconds to read)

- humans make decisions, may need to talk to other humans (all of what you said above)

- humans evaluate correct procedure, double-check it (you don't want them making the wrong "fix" and making something else worse, do you?)

- humans execute commands

- commands take time to run on large collections of computers (running them completely in parallel can cause thundering herd issues, in some cases)

TomAnthony• 8 years ago

The problem is so clear in their write up that I can understand your thinking. However, in reality as this was going down it was probably not that clear cut.

Especially when you consider that they are getting DoS attacks every 2-3 minutes - so all deploys are going out into a hectic world and the dots maybe aren't that easy to connect under those circumstances.

Robin_Message• 8 years ago

What's interesting here is that the automatic cure (DDoS protection) was worse than the disease (even if there was an attack, blocking all access to the DNS servers is potentially worse than letting them get overloaded).

I wonder if it would be possible to express the idea that if a block being applied drops traffic well below expected levels, it must be a mistake?

ben509• 8 years ago

Is it the block causing the traffic drop, or the DDOS though?

baby• 8 years ago

I switched to 1.1.1.1 when it was released and since I’ve had multiple issues with free wifis where they would fail to hijack my dns requests to allow me to login to their portal. I Assume this is a good thing but can someone explain to me why this is happening and what’s the state on improving these wifi portals?

PS: sorry for hijacking the thread.

syntheticcdo• 8 years ago

Might be unrelated. Always try http://neverssl.com to trigger the captive login on public wifi.

woofcat• 8 years ago

Set your DNS to 1.0.0.1 and it might be a bit better. Still cloudflare's DNS resolver but used by less stupid Wi-Fi portals.

acdha2• 8 years ago

Cisco shipped some hardware which incorrectly used 1.1.1.1 as an internal interface. The long-term fix will be those networks being correctly configured but that may not happen until that hardware is EOLed.

In the meantime, have you tried adding other addresses such as 1.0.0.1, 8.8.8.8, or 9.9.9.9 to see whether the fallback works?

IronWolve• 8 years ago

Doesn't seem fixed, still can't resolve archive.is cloudflare is giving off cloudflare DNS web errors.

https://i.imgur.com/APpQPTJ.png

nestramutat• 8 years ago

I’ve never been able to resolve some of those domains, specifically archive.il. Not a fan, why I switched away.

acdha2• 8 years ago

That's apparently due to the archive.is servers returning different results based on the requesting IP:

https://community.cloudflare.com/t/archive-is-error-1001/182...

zouhair• 8 years ago

Yup now I remember why I took it off.

8_hours_ago• 8 years ago

I wonder if the change was manually reverted after 17 minutes, or if Cloudflare has a system that watches for a spike in failures and automatically reverts the most recent change.

jonnismash• 8 years ago

The next time we mitigate 1.1.1.1 traffic, we will make sure there is a legitimate attack hitting us.

I fucking love these guys

known• 8 years ago

Error 1001 DNS resolution error when trying to access archive.li

But works with 9.9.9.9

xtrimsky1234• 8 years ago

This downtime annoyed me at a critical time. Even though I think it's great they are transparent about it, I don't see myself using their DNS again.

1.1.1.1 outage explanation

Related Stories