HBO Now DNSSEC Misconfiguration Makes Site Unavailable from Comcast Networks

http://www.internetsociety.org/deploy360/blog/2015/03/hbo-now-dnssec-misconfiguration-makes-site-unavailable-from-comcast-networks-fixed-now/

by danyork • 11 years ago

81 76 11 years ago

internetsociety.org

wtbob2• 11 years ago

Also unavailable from Android devices…

danyork19• 11 years ago

Which would make sense because they may be set to use Google's Public DNS Servers (8.8.8.8 and 8.8.4.4 and the IPv6 equivalents).

Google Public DNS has been performing full DNSSEC validation since May 2013: http://www.internetsociety.org/deploy360/blog/2013/05/confir...

yuhong10• 11 years ago

Yea, it is fortunate that people on Twitter quickly figured this out.

danyork19• 11 years ago

If it's currently unavailable on Android, you should be able to start seeing the site again whenever the TTL of the DNS records ages out... this just happened for me on my own home network where I have a DNSSEC-validating DNS resolver on the edge of my network.

However, when I do "dig +dnssec @8.8.8.8 ds hbonow.com" I don't get any DS records back, indicating that the issue should be fixed on Google's PDNS.

Can you tell what DNS servers you are using on your Android device? It may be that your ISP is doing DNSSEC validation and experiencing a similar TTL issue.

wtbob2• 11 years ago

I was being snarky, since HBO Now reportedly supports only Apple clients.

tptacek18• 11 years ago

This shouldn't be possible. In a sane design, a transient policy failure should produce a warning light or a browser popup; something that applications can use to sensibly convey what happened, in some form, to users.

Instead, because DNSSEC makes security policy decisions at a very low layer, in a protocol that virtually all applications expect to work transparently so long as you have connectivity, you get this: instead of a browser popup, a connectivity outage so complete that it is indistinguishable to users from "the site you are trying to get to doesn't exist and never has".

DNSSEC is a terrible, terrible idea. Comcast should stop validating it, immediately. If someone from Comcast's network engineering team is reading this: I was a network engineer, and then worked for 4 years with tier-1 engineering getting Netflow monitoring deployed and scalable. I've been writing DNS security software since I was 19 years old, in the 1990s. You have reasons for whatever configuration you've decided on, but I will take as much time out of my day as is productive for you to convince you to stop working on rolling this out.

What happened yesterday with HBO Now, which had half of Twitter screaming "NET NEUTRALITY OHNOZ!", is going to keep happening over and over again.

Diederich• 11 years ago

Thomas, I'll ask directly even though I believe I know the answer. But this topic is relatively dense so there's a non-trivial chance I've got some wires crossed.

In short: with total objectivity, what are all of the possible, theoretical advantages to the people using order.hbonow.com of HBO using DNSSEC + https + (maybe) HSTS rather than them ONLY using https + HSTS?

I'm posting this (I think) leading question so that it might be possible to cut through VOLUMES of related discussion, and see a relatively simple, straightforward answer.

Thank you!

PS: I have read through your Against DNSSEC post, and the large amounts of related discussion, but I'm still seeking a technical yet succinct response that I can easily 'carry around' with me.

tptacek18• 11 years ago

There is zero reason for HBO to be using DNSSEC today. Browsers don't support it, and email is insecure with or without it, and that describes pretty much their whole attack surface. It was a totally unforced error, which is presumably why HBO's response to this debacle was to eliminate DNSSEC.

danyork19• 11 years ago

tptacek - I was waiting for your reply... almost thought about setting a timer! Glad to see you didn't disappoint.

Toward the end of the article you'll note I point out that this is a great example of a hole in the operational process that has been identified as more and more people have deployed DNSSEC:

http://www.internetsociety.org/deploy360/blog/2015/02/cloudf...

The process of updating (or removing) the DS record NEEDS to be automated. There is a group within the industry working on ideas around this now and I think we'll see that work happen inside the DNSOP Working Group within the IETF.

There is a public mailing list open to anyone interested to join.

Someone1234• 11 years ago

While I agree DNSSEC has its limitations and problems, I haven't heard many alternatives (in general).

DNS has two inherent issues:

- Hijacking traffic via spoofing responses.

- Private information leakage.

With DNSSEC combined with HTTPS it is very reasonable to say that both you're connecting to the party you're expecting to connect to and in addition, no third party listening on the line either knows what host name or page you're visiting (only the IP address of the box).

A lot of people dismiss these limitations or just point wildly at certificate pinning to solve all of our problems (while ignoring that HTTP isn't the only type of traffic the internet was designed for).

Plus with NSA mass surveillance, DNS makes seeing what domains you're visiting and building a "picture" of you as an individual absolutely trivial. The IP addresses still may help them do that to some extent, but it certainly becomes very easy if they can see your DNS packets.

icebraining2• 11 years ago

No, because clients don't validate DNSSEC responses. You'd need to be running a full resolver in your machine, which OSs usually don't.

Also, with SNI¹ they'll know the hostname anyway.

¹ http://en.wikipedia.org/wiki/Server_Name_Indication

danyork19• 11 years ago

Plus with NSA mass surveillance, DNS makes seeing what domains you're visiting and building a "picture" of you as an individual absolutely trivial. The IP addresses still may help them do that to some extent, but it certainly becomes very easy if they can see your DNS packets.

FYI, there is a working group within the IETF working on the issue of providing confidentiality to DNS transactions. It's called DPRIVE and more info can be found here:

https://datatracker.ietf.org/wg/dprive/charter/

IvyMike• 11 years ago

I haven't heard many alternatives (in general).

From tptacek's FAQ ( http://sockpuppet.org/stuff/dnssec-qa.html ):

What’s the alternative to DNSSEC?
Do nothing. The DNS does not urgently need to be secured. All effective security on the Internet assumes that DNS lookups are unsafe. If this bothers people from a design perspective, they should consider all the other protocol interactions in TCP/IP that aren’t secure: BGP4 advertisements, IP source addresses, ARP lookups. Clearly there is some point in the TCP/IP stack where we must draw a line and say “security and privacy are built above this layer”. The argument against DNSSEC simply says the line should be drawn somewhere higher than the DNS.

secalex• 11 years ago

I haven't heard many alternatives (in general).

These problems don't call for a general solution. We should respect the fundamental truth behind the end-to-end principle and solve these problems as close to the application as possible. The likely threats, appropriate default choices and trust model are all best understood by each application developer and building these solutions into Layer-3/4 is bound to cause short-term pain and reduce our long-term flexibility to meet new risks.

For example, EFF's STARTTLS Everywhere project takes into account the fact that a huge percentage of the world's mail moves between a small enough number of providers that human verification of announcements is possible. It also recognizes that the MX configuration for these providers is reasonably static, meaning that changes that propagate in minutes do not need to be accommodated. Small specific solutions like this can be rolled out and provide a real, tangible benefit to users much faster than we can upgrade the entire DNS system to provide a more general solution.

I agree that we need a DNS privacy solution, one that hopefully doesn't eliminate the existence of caching infrastructure.

davidu2• 11 years ago

DNSSEC does nothing to prevent information leakage, the opposite in fact.

People compare it to SSL but it's not like SSL. It does NO encryption of the commuications. It's only validation (argubaly the most broken part of SSL).

A better ananlog to SSL would be DNSCrypt, which we've been using now for years, is slowly gaining traction and adoption by others, and provides real privacy from DNS messages from being intercepted or manipulated.

korethr• 11 years ago

I used to think that DNSSEC was a good thing. Authenticated DNS lookup, what's not to like? Incidents like this make me question that.

But dropping DNSSEC leaves the problem of securing DNS unhandled. What would you propose in its place to address securing DNS?

yuhong10• 11 years ago

DNSCurve, for one.

yuhong10• 11 years ago

I wonder what would be wrong with a new DNSSEC2 that would be designed for online signing of DNS records from the beginning for example (so no NSEC/NSEC3). It would not replace DNSCurve, but would be usable for DANE for example.

hackmiester• 11 years ago

What alternatives to DNSSEC do you advocate?

tptacek18• 11 years ago

Step 1: Scrap DNSSEC

Step 2: Return to drawing board

Step 3: Wait for better solution to emerge

Step 4: Repeat until value of new DNS security system exceeds cost of deployment

Step 5: Profit

I expect never to reach Step 5, but who knows?

danyork19• 11 years ago

In a sane design, a transient policy failure should produce a warning light or a browser popup; something that applications can use to sensibly convey what happened, in some form, to users.

Thomas, I agree that having the DNS resolver only return a regular SERVFAIL without a hint of WHY there was the failure is a challenge. I wasn't around the DNS part of the IETF when this design decision was made. It seems to me that a separate error message would have been preferable... but I don't know the discussions that were had at that time.

There have been several suggestions about ways to include additional diagnostic information with the SERVFAIL so that browsers and applications could take better action. One such proposal was from Evan Hunt at ISC (makers of BIND):

http://tools.ietf.org/html/draft-hunt-dns-server-diagnostics...

The draft expired back in 2014 but he indicated recently on the DNSOP mailing list that he would be open to reviving that draft if people thought it would be useful.

yuhong10• 11 years ago

I think DNS was designed to be a relatively simple protocol and the same for the APIs too.

xorcist4• 11 years ago

Do you hold that reasoning true every protocol, or is DNS somehow special? Should we immediately halt all work on securing BGP, for example, as misconfigurations will make entire ASes unavailable?

tptacek18• 11 years ago

No, it's not true of every protocol. That's my point.

Some protocols are well-situated to make security policy decisions, and some aren't. This is an obvious engineering point: however you secure BGP, you're going to have TCP delivering records. Should we then secure TCP, so that we can ensure availability and prevent attackers from exploiting it to introduce byzantine routing failures? And, if we create TCPSEC, some form of unencrypted IP traffic will be used to deliver those segments to end stations. Should we then secure IP? And then ARP? And then Ethernet headers?

Secure BGP makes a lot of sense. BGP is already more of a policy expression framework than a routing protocol (the routing algorithms used by BGP are basic, and virtually all of the complexity comes from two decades worth of hacks designed to express policy). Fully every single router running defaultless BGP is managed by a team of people engaged intimately with BGP security policy.

DNS, however, runs on every single Internet user's computer, often in different places, and at a layer that isn't fully exposed by APIs, because those APIs were designed with the assumption that DNS fails only due to connectivity failures. DNS is a terrible place to enforce policy.

zrm3• 11 years ago

DNSSEC is a terrible, terrible idea.

DNSSEC is a terrible, terrible implementation. All of your criticisms are valid but none of them are inherent to the idea of authenticating DNS resolution.

That basically means we need to start over from scratch, which is unfortunate, but now tell me if we did we couldn't address your complaints:

1) It isn't necessary. Sure, we can work around the lack of DNS authentication, but then we end up with the horrible CA system. Can we really not do any better than that?

2) It's centralized and government-controlled. So don't sign the root. Instead of hard-coding the root key in the resolvers, hard-code the keys for each TLD. Being able to pick which government can forge your signatures is the best you're going to be able to do; the authority doing the signing is going to exist in somebody's jurisdiction.

3) DNSSEC's cryptography is weak. So use different cryptography. Elliptic curve keys and signatures are an order of magnitude smaller than RSA anyway, which also reduces the DNS DoS amplification potential by that amount.

4) Resolver APIs don't provide good information about why resolution failed. So provide a new API that provides better errors. Put it right in the new RFC. Then you only get validation if you use the new API and nothing changes for existing applications.

5) Deployment is expensive. Why isn't deployment automatic? The default configuration for a DNS server should be to generate a signing key for each domain on first run and then automatically sign all the records with it. If you're paranoid and you want to keep your signing keys offline then you can configure that manually, but nobody has to. And the higher level domain should be able to get the signing key from the lower level domain as soon as you add the NS record, and then confirm with the administrator that it's the right key the same way as with ssh host keys.

6) DNSSEC isn't validated by the endpoints. As far as I can tell there is no actual reason for this even with existing DNSSEC. The client can ask its DNS cache for each of the signing keys up to the root and then check the signatures. A new pseudo-RR that would return the entire chain would make this more convenient, and couldn't really be used for DoS because only recursive resolvers and not authoritative servers would answer that query. Validating clients without validating caches could still fall back to asking for each individual record.

7) Authoritative denials leak information. NSEC5 is supposed to fix this, but there is a much easier way: Sign a denial key which can itself only be used to sign denials and keep it on the authoritative server. The idea that someone who compromises your authoritative servers will be able to deny service to your DNS clients is already sort of implied.

The problem is, even though you could theoretically do all of those things to DNSSEC itself, those aren't even the only problems, and trying to patch all the warts in something nobody is even really using is only going to make something which is already unnecessarily complicated even worse. What is needed very much is a clean slate.

But that doesn't mean it isn't worth doing.

tptacek18• 11 years ago

The most important TLDs are controlled by world governments. COM and NET are essentially USG properties. What's the most popular TLD outside the original TLDs? IO. Guess who controls IO? GCHQ, the world's most unhinged signals intelligence agency.

Moreover, how can it possibly be sane for us to deploy a security system that protects end-users from NSA only if Google is willing to move Gmail off of COM?

If NSA subverts a CA today and uses it to MITM Gmail, a substantial fraction of all browsers on the Internet will detect that and alert Google, because of key pinning. When that happens, Google will nuke that CA from orbit. If NSA is dumb enough to subvert a CA that's hard to nuke, Google will start a process of employing code-level restrictions on that CA that will for a substantial portion of all Internet users make that CA asymptotically approach "useless" for NSA's purposes.

If NSA does a QUANTUM INSERT-type attack to selectively poison .COM lookups in order to use TLSA to get a target to eat a fake certificate, what does Google do? Nuke COM from orbit?

DNSSEC is a terrible, terrible idea.

danyork19• 11 years ago

Several people have confirmed to me that they were unable to see the site from Google's Public DNS Servers (8.8.8.8, 8.8.4.4 and the IPv6 addresses). This makes sense given that Google has been performing DNSSEC validation since early 2013.

I've updated the post to more clearly note the fact that this was not just an issue on Comcast networks. (I mentioned that it would be an issue on any network performing DNSSEC validation, but then only gave Comcast as an example.)

st3fan• 11 years ago

"As a result, the many networks around the world that perform DNSSEC validation to ensure that customers are getting to the correct sites (versus being redirected to bogus sites for phishing or malware) were blocking customers from getting to the possibly bogus order.hbonow.com!"

The "versus being redirected to bogus sites for phishing or malware" part here is funny. Because that generally happens when some scammer registeres hb0now.com. Not when someone is intercepting your DNS.

And actually, in case of the latter, all bets are off anyways. Because client resolvers DO NOT CHECK DNSSEC.

(Yes, I know you are on your custom configured Linux box or OpenBSD firewall and that DNSSEC works wonderful for you. But the majority of the internet using world with OS X or Windows behind a $15 DSL router or WiFi AP does not.)

Stop. The. DNSSEC. Nonsense.

xorcist4• 11 years ago

There have been many instances of unknown entities intercepting DNS which went unnoticed for quite some time. Hence DNSSEC.

danyork19• 11 years ago

And actually...

And actually, in case of the latter, all bets are off anyways. Because client resolvers DO NOT CHECK DNSSEC.

... please visit APNIC's DNSSEC Statistics site where you will see that about 12% of all DNS queries globally ARE being validated by DNSSEC:

http://stats.labs.apnic.net/dnssec/XA?c=XA&x=1&g=1&r=1&w=7&g...

In Sweden this is about 71%:

http://stats.labs.apnic.net/dnssec/QM?o=cXAw7x1g1r1

Slovenia 67%, Estonia 55%, Denmark 48% ... on down to the USA at 23%:

http://stats.labs.apnic.net/dnssec/XQ?o=cXAw7x1g1r1

So DNSSEC validation very definitely * IS * happening out there!

icebraining2• 11 years ago

Checks being made doesn't mean the clients are checking. What's the point of validating the communication between the servers if the last mile is unprotected?

tptacek18• 11 years ago

This is "checking" in the sense that "everyone at Comcast was 'checking' when HBO Now broke".

Panino• 11 years ago

DNSSEC causes more problems than it solves. And even if it were reliable, it's still based on obsolete crypto and isn't even encrypted. But on top of that, it's an outage factory.

samplonius• 11 years ago

Why would it make sense to encrypt queries and replies from a public data source? Signing the data is all that is needed.

jessaustin• 11 years ago

Some users might not want intermediate hosts to see a request for the IP addr of "ginger-midget-porn.com", for example.

HBO Now DNSSEC Misconfiguration Makes Site Unavailable from Comcast Networks

Related Stories