HN user

mnordhoff

199 karma
Posts0
Comments109
View on HN
No posts found.
GCP Outage 1 year ago

Wellp. Incident report: "We posted our first incident report to Cloud Service Health about ~1h after the start of the crashes, due to the Cloud Service Health infrastructure being down due to this outage."

Maybe not the entire region. Amazon was reportedly building a data center complex next to the natural gas Hermiston Generating Plant some distance from the river.

I have bad news for you, Linode's authoritative DNS service also uses Cloudflare DNS Firewall.

  $ dig +short ns1.digitalocean.com aaaa
  2400:cb00:2049:1::adf5:3a33
  $ dig +short ns1.linode.com aaaa
  2400:cb00:2049:1::a29f:1a63

This has a couple bugs:

1.1.1.2 and 1.1.1.3 both return the SVCB records for 1.1.1.1. (I don't know if clients would ignore them, or actually switch to 1.1.1.1.)

Non-SVCB-type queries for _dns.resolver.arpa return NXDOMAIN instead of NOERROR. (This probably doesn't have significant impact, but might break it for some clients, e.g. if they can be tricked into making queries for _dns.resolver.arpa, or if a downstream resolver makes DS queries.)

Resolvers typically cache successful "does not exist" responses for no more than 1-3 hours. (And authoritative servers often have a lower negative TTL.)

(There's a corner case related to DNSSEC that can make it go higher, but that's being worked on, and isn't relevant here.)

In this situation, the nameservers were just down. I haven't done exhaustive research, but the resolvers I'm aware of cache that kind of thing for no more than 15 minutes.

"Because of this Cloudflare’s 1.1.1.1 DNS resolver could no longer respond to queries asking for the IP address of facebook.com or instagram.com."

The instagram.com zone itself uses a third-party DNS service and didn't go down. (But e.g. www.instagram.com is a CNAME to a zone on FB DNS.)

DigitalOcean S-1 5 years ago

DigitalOcean has /32s from ARIN and RIPE.

(And a /48 from APNIC???)

(Edit: And a /36, /40 and /48 from APNIC?)

why the DNS test usualy takes >60 seconds

The server-side part of DNS validation takes about a second. The delay is all about clients waiting for their authoritative DNS servers to update. If you use a fast DNS provider, there's no reason to wait longer than necessary.

If any certs were issued for hijacked domains (which as far as I've ready was only one, not using LetsEncrypt), it's a pretty glaring failure on the issuer, assuming they used "DNS Validation"

It wouldn't be a compliance failure, though. CAs are not required to be invulnerable to BGP hijacking attacks.