Tell HN: 1.1.1.1 appears to be down

https://news.ycombinator.com/item?id=44565836

by Wingy • 1 year ago

135 71 1 year ago

news.ycombinator.com

Cloudflare's DNS server doesn't appear to be working.

    6:03PM storm ~ % ping 1.1.1.1
    PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
    ^C
    --- 1.1.1.1 ping statistics ---
    4 packets transmitted, 0 received, 100% packet loss, time 3103ms

nagisa• 1 year ago

Can confirm its down here too.

1.0.0.1 is also down.

EtienneK• 1 year ago

Yup, same here (Europe). Opened up HN to confirm. Thanks :)

SCHiM• 1 year ago

It's down. Tested from two servers, 8.8.8.8 and others are up.

msvcredist2022• 1 year ago

Confirmed down in the PNW & Virginia (east1) as well.

pablonara• 1 year ago

Down in iowa and montreal too

tom13372• 1 year ago

and here (EU West) I am debugging why my internet is not working and using ping 1.1.1.1 as a check

zubspace• 1 year ago

Same here! Restarted my router and pi hole twice. Now i feel stupid.

PaulHoule• 1 year ago

No shit. My "internet" just went down and I switched over to 8.8.8.8 and got back up.

strongpigeon• 1 year ago

Same. I assumed it was my ISP as it had some hiccups lately, but when I saw that 8.8.8.8 was responding to ICMPs I suspected 1.1.1.1 was down.

explodingwaffle• 1 year ago

I tested with 1.1.1.1 first, didn’t get anything, and gave up for the night. Maybe I should put a different provider as DNS backup? (any DNS gurus to say that that’s a bad idea?)

sashk• 1 year ago

their status page shows there is no problems with it.

thecosas• 1 year ago

Looks like they have it listed as of a few minutes ago: https://www.cloudflarestatus.com/incidents/28r0vbbxsh8f

durakot• 1 year ago

Looks to be down globally... another friendly reminder of our overdependence on a few services (and how many servers are configured to use 1.1.1.1 for DNS queries?)

thekid314• 1 year ago

In NYC it appears down for me too. MacBook-Pro ~ % ping 1.1.1.1 PING 1.1.1.1 (1.1.1.1): 56 data bytes Request timeout for icmp_seq 0

hunkins• 1 year ago

Yep, timeouts on my end.

PING 1.1.1.1 (1.1.1.1): 56 data bytes Request timeout for icmp_seq 0 Request timeout for icmp_seq 1 Request timeout for icmp_seq 2 Request timeout for icmp_seq 3 ^C --- 1.1.1.1 ping statistics --- 5 packets transmitted, 0 packets received, 100.0% packet loss

armitron• 1 year ago

Don't use Cloudflare, they've done enough damage to the Internet with their centralized bs without you needing to further reward them by handing over all your DNS data.

kordlessagain• 1 year ago

I agree. Modern day man-in-the-middle attack via a corporate entity. Rationalized as a protection racket.

chgs2• 1 year ago

HN loves cloudflare. The majority here aren’t of the ethos of the distributed internet of days of old, it’s the “how can I monetise this hustle” ethos. Sad really.

bb88• 1 year ago

Tata Communications in India was the one that apparently caused the outage.

Beaving• 1 year ago

https://x.com/nadeu/status/1944881376366616749

fastily• 1 year ago

the curse of bgp strikes again

no_carrier• 1 year ago

Also can be confirmed by Cloudflare's own route leak detection tool - https://radar.cloudflare.com/routing/anomalies/hijack-107469

tom13372• 1 year ago

https://www.cloudflarestatus.com/incidents/28r0vbbxsh8f

alecsm• 1 year ago

It's down in Spain too.

pwr22• 1 year ago

Down for me from UK

nodesocket• 1 year ago

I just got 45 e-mail notifications from Uptime Kuma and knew something was afoot.

SuperSandro20002• 1 year ago

near total global outage according to https://atlas.ripe.net/measurements/117762218/

gerdesj8• 1 year ago

DNS shouldn't be tested with ICMP. Try dig or nslookup instead. ICMP echo request/reply may help to decide reachability and nothing more.

This is a reasonable test of the DNS service on 1.1.1.1:

  $ dig @1.1.1.1 www.cloudflare.com A

  ; <<>> DiG 9.20.4-3ubuntu1.1-Ubuntu <<>> @1.1.1.1 www.cloudflare.com A
  ; (1 server found)
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34112
  ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

  ;; OPT PSEUDOSECTION:
  ; EDNS: version: 0, flags:; udp: 1232
  ;; QUESTION SECTION:
  ;www.cloudflare.com.            IN      A

  ;; ANSWER SECTION:
  www.cloudflare.com.     36      IN      A       104.16.123.96
  www.cloudflare.com.     36      IN      A       104.16.124.96

  ;; Query time: 39 msec
  ;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
  ;; WHEN: Mon Jul 14 23:32:57 BST 2025
  ;; MSG SIZE  rcvd: 79

[EDIT]:

  $ ping 1.1.1.1
  PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
  From 141.101.70.116 icmp_seq=1 Time to live exceeded
  64 bytes from 1.1.1.1: icmp_seq=2 ttl=50 time=126 ms

So ping fails a bit (and then works - firewall) but DNS works.

The service required is DNS not ping. Test the service.

landofyoshi• 1 year ago

Well, typically 1.1.1.1 responds to pings. So it not responding is an indication that it's no longer working.

captainkrtek• 1 year ago

Or that it started filtering ICMP. If DNS works then it’s doing its job

gerdesj8• 1 year ago

... or a change of policy, funky firewall or whatever.

It does seem to be responding to ping again and since my edit above, the first packet is being responded to so I suspect a NOC is having a fun old time somewhere.

You do need to test the service properly. I do this malarky for a living 8) I'm ever so popular with kiddies and their gaming related fixation with ping times ...

forbiddenlake• 1 year ago

This is all true, but DNS was also down.

Signed, someone who was using 1.1.1.1 as their DNS server and hadn't configured a fallback

gerdesj8• 1 year ago

As a punishment: Compile and install ISC BIND from source and configure it 8)

Many home routers can resolve starting from root or if you must then: 1.1.1.1, 8.8.8.8, 8.8.4.4 will get you started. You might consider 9.9.9.9 and there are quite a few others.

I never, ever, ever, recommend using ISP provided DNS unless you know how they are configured. The anycast jobbies at least publish a policy of some sort.

graton• 1 year ago

You weren't alone. I was actually using 1.1.1.2 and then quickly added 8.8.8.8 once I figured out the issue.

indigodaddy7• 1 year ago

By using ping or MTR, they are testing general connectivity to an endpoint, doesn't matter what service is in play. For example, if you are getting significant packet loss on the endpoint itself in the output of an MTR, then that IS indicative of a network/route/connectivity problem, somewhere along the route (could still be an endpoint issue but definitely not always). The service in question doesn't matter much at that point. Whether the service itself is healthy or not, you are still troubleshooting the overarching issue presented by the bad ping/MTR.

gerdesj8• 1 year ago

ping and mtr only test one thing. I have saws, drills, routers (lol), planes, screwdrivers, hammers and more in my workshop. To be fair a drill driver and a hammer get a lot of jobs done! However, I will get the impact driver out or a fret saw with a very fine blade as the job requires.

The article here is about a loss of DNS service and proves it with ping. That is wrong and you know it. Diagnosing the fault should involve ping but that is not how you conclusively prove DNS is not working.

To be honest you cannot conclusively prove anything in this game but you can at least explore the problem space effectively from your perspective with whatever you have access to. I happen to have a RIPE ATLAS probe at work with a gigantic amount of credit, so I could probably get that system to test Cluodflare DNS from a lot of locations.

If you present to a doctor with some mild but alarming chest pains, I'd hope they wouldn't just look at you and prescribe a course of leeches. A stethoscope is a good start (ICMP) but an ECG (dig) is better. That analogy might need some work 8)

op00to2• 1 year ago

No, ping can be deprioritized while actual interesting packets pass through with much less latency.

Calzifer• 1 year ago

The service required is DNS not ping.

  ping 1.1

is short and easy to remember. Since I'm not using Cloudflare DNS, ping is actually the service I require :D

gerdesj8• 1 year ago

In which case:

  $ ping 127.0.0.1

Provided you have a working IP stack, your ping service requirement is met admirably 8)

I run a lot of pfSense boxes and they (and OPNSense) have a pinger daemon to test connectivity which is really useful for multi-link routing. Bad idea for single links because you add an extra thing to fail. On a router with multiple internet links they are handy. You mostly ping known "reasonably stable" anycast addresses - they are the best option and usually end up being DNS servers - 1.1.1.1, 8.8.8.8, 8.8.4.4 etc are all good candidates.

alexktz• 1 year ago

Praise smokeping

https://imgur.com/a/YGYl0Oy

DoctorOW• 1 year ago

I recently switched from Cloudflare to ControlD and it was perfect timing to miss this!

SuperSandro20002• 1 year ago

hehe https://radar.cloudflare.com/routing/anomalies/hijack-107469

their bgp monitoring found it :)

madisp• 1 year ago

modern state of status pages makes me sad :( You were a good 10 minutes quicker to note the issue than Cloudflare's status page was

esseph5• 1 year ago

Major ones will rarely be automated due to legal liability (among other things).

I agree with you, though.

guluarte• 1 year ago

raise up chads using their own custom DNS resolver with 10+ upstream providers

chgs2• 1 year ago

I have 6 upstream, but that’s for each of two dns serves in home (one on my pi, one on the jellyfin), so I guess that’s 12 upstream together.

bigstrat2003• 1 year ago

Upstream providers? I use root hints, the way God intended.

nh43215rgb• 1 year ago

I wonder how uptime ratio of 1.1.1.1 is against 8.8.8.8

Maybe there is noticeable difference?

esseph5• 1 year ago

Both of these are among the most reliable and resilient internet services in the world.

indigodaddy7• 1 year ago

10-15 minutes ago was getting intermittent TTL exceeded errors when pinging 1.1.1.1. Seems clean now and seem to be resolving ok now

gcau• 1 year ago

The cloudflare status page had nothing reported, so I just assumed its some issue elsewhere (and the HN post didn't exist yet), if it wasn't for HN I'd probably be ordering a new router and ripping apart all my network settings and complaining to my ISP.

g1sm• 1 year ago

This outage made me realize the script I was using to test my internet connectivity was depending 100% on cloudflare: I was both pinging 1.1 AND querying 1.1.1.1 using dig and, if both failed, the script would restart pppd.

Demiurge• 1 year ago

This is it, I've been experiencing issues with DNS for longer than their timeline reports, but I also tracked it down to no response from DNS.

Does anyone have a good backup for CF? I certainly don't want to rely on my ISP, has they've done MITM before.

op00to2• 1 year ago

Nextdns?

evulhotdog• 1 year ago

Quad9, too.

hnarn• 1 year ago

For anyone that has a capable router, an rpi or any kind of home server, I can highly recommend https://github.com/DNSCrypt/dnscrypt-proxy

It lets you send encrypted DNS queries out onto the Internet to any service that supports it (there are many, and you can configure it to use multiple for redundancy), while serving "normal" DNS in your internal network.

It's also trivial to import a blocklist of domains with cron, from hagezi/dns-blocklists for example.

If you have no interest in setting something like this up, at least ensure that you have manually configured or are pushing _multiple_ DNS servers via DHCP. It sucks that 1.1.1.1 went down but it shouldn't matter, there's a reason every operating system supports configuring multiple DNS servers.

For anyone in the EU I can recommend https://www.dns0.eu/ or Mullvad, but at the very least if you're using Cloudflare and don't care about privacy, set 8.8.8.8 as your secondary DNS.

Tell HN: 1.1.1.1 appears to be down

Related Stories