DNSSEC failures are how you get people to disable DNSSEC

https://utcc.utoronto.ca/~cks/space/blog/tech/DNSSECFailureDrivesDisablement

by zdw • 3 years ago

56 55 3 years ago

utcc.utoronto.ca

yakkityyak• 3 years ago

The experience around just setting up DNSSEC when I tried it on Google domains was so incredibly frustrating due to the time delays and pain points surrounding correcting mistakes.

Don't know if it has gotten any better.

silisili5• 3 years ago

Correcting mistakes is a pain point, but really applies to DNS as a whole rather than just DNSSEC.

I think a lot of the problem stems from people using really long TTLs for the keys, which used to be the standard advice.

That said, as someone who used to manage DNS and DNSSEC at a TLD level, I will admit that documentation and best practices are poor. I remember asking someone why it was this way, and he told me it was because people want to make money contracting, so were less than willing to make it accessible.

Once you understand what's happening, you can make a little cheat sheet and it's actually really simple. But it seems like everyone forges ahead their own way, myself included.

esaym• 3 years ago

Show us your cheat sheet...

cyberax2• 3 years ago

One trick to make it easier: set all relevant TTLs to 1 minute before trying something dangerous. Plenty of resolvers ignore that, but the main ones (1.1.1.1, 8.8.8.8, etc.) respect that.

tptacek7• 3 years ago

The protocol wasn't designed on purpose to do this, but I think it's worth keeping in mind that this is why centralized DNS providers do DNSSEC features and promote DNSSEC: it's a thing that keeps you dependent on SAAS providers, because the DIY is so difficult. Keeping your domain anchored to a particular platform is a powerful customer retention mechanism; you're likely to do everything you can with that domain with that same platform.

xvilka• 3 years ago

We need a truly decentralized alternative, DNSSEC isn't that.

silisili5• 3 years ago

DNS is de facto decentralized. Stand up your own root, it's that simple.

DNSSEC has no bearing on how centralized something is - it works the same on the current system as it would on your own new DNS system.

noman-land• 3 years ago

Wait, what? How can I be my own certificate authority?

HaroldBolt78• 3 years ago

I had the power turned off at home about a year ago. I think it was due to maintenance by the utilities company, or maybe I did it for DIY reasons.

Anyway, after it came back on, it took me far too long to realise why my internet was down. All DNS responses were being rejected. It turns out, my Pi-Hole DNS server - which had forgotten the time, of course - was trying to resolve the IP of an NTP server so it could fetch the current time, but this name resolution was over DNSSEC which requires the current time. You need the time to get the time.

I turned off DNSSEC and haven't looked back since.

bouke• 3 years ago

I had exactly the same issue happening to me! I’ve since added a Real Time Clock (RTC), so the Pi shouldn’t forget about the current time anymore. That is until the battery runs out of course, for which there is no monitoring in place. So yeah maybe I should disable DNSSEC as well. At least we need to get better at building fault tolerant systems. Maybe Pi-Hole needs to be aware of and handle this scenario better: a special “we’re booting up and we don’t have the current time so don’t do DNSSEC yet”.

toast02• 3 years ago

fake-hwclock[1] is useful for this. I don't know much about Pi-Hole, but if it involves writable storage and has a package manager, fake-hwclock may be available from there.

At least we need to get better at building fault tolerant systems.

Security and fault tolerance are at odds here. Failing closed when you can't validate is correct, but not so useful. Properly modeling the clock might give you a way to get bootstrapped, but then an adversary would arrange for a bootstrap situation so they could interfere with your clock and then make use of the incorrect time for their nefarious purposes.

[1] https://git.einval.com/cgi-bin/gitweb.cgi?p=fake-hwclock.git

boringuser2• 3 years ago

Does pihole even have value anymore with DNS over https?

I'd just implement that in 0.2 nanoseconds if I were a Samsung engineer making an intrusive adserver.

citrin_ru• 3 years ago

For me it's a reason to use IP address in /etc/ntp.conf instead. DNS over HTTPS and HTTPS in general also need current time to validate a certificate so DNSSEC is not unique here.

Avamander• 3 years ago

This makes Network Time Security difficult.

This is a reason you add an RTC to your RPi.

mnd999• 3 years ago

When I’ve enabled it in systemd.resolved it breaks every hotel / conference centre / airport WiFi. Maybe that’s a good thing but it’s pretty inconvenient.

elp• 3 years ago

I run the signing systems for a bunch of TLDs and honestly it scares me.

.NZ was just the latest public failure. If multiple large registry operators employing people who wrote the DNS and DNSSEC rfcs have had failures then how can you expect regular mortals to get it right?

Transferring your domain and your old ISP had dnssec on but the new one knows nothing about it and didn't check ... Down you go.

Mess up the key signing key change over procedure while transferring to a competent DNSSEC provider ... down you go again (although hopefully for a shorter time).

We do best practice DNSSEC, using fancy HSMs to do so, but if my fancy, proprietary, badly documented Thales HSM dies and the next person with my job has trouble getting the new box configured... no changes to the zones until its fixed. Take long enough that the RRSIGs expire ... down YOU go again. If at that point they throw out the HSM and roll regular DNSSEC using Knot or something it will probably be at least 24 hours for the zone to work properly even with IANAs emergency procedures.

I'm paranoid so we have 3 identical HSMs and detailed restore procedures but there is zero guarantee that if I ever leave my replacement will get it right. You can't hire dnssec staff, they are impossible to find, you have to train on the job.

The basic idea is excellent but the implementation in the real world is horrible. If dnssec usage was wide spread we might reach some decent maturity with the tools and protocols, but I can't see that happening any time soon.

donttellmypeers2• 3 years ago

If dnssec usage was wide spread we might reach some decent maturity with the tools and protocols, but I can't see that happening any time soon.

DNSSEC's lack of maturity is a symptom of DNS itself not having a healthy ecosystem. I feel like it'd take as little as a half-percent of the people writing new HTTP tooling to noticeable improve the DNS ecosystem as a whole.

silisili5• 3 years ago

Totally. DNS was always an IETF RFC bound thing. Recently, big players just kinda started making their own decisions.

I'm not sure which is better. Having big companies control the internet is definitely bad, but the pedantry and bikeshedding the IETF offers can be equally bad.

I'd probably rather see the two cooperate/converge, but that's a pipe dream.

abwizz2• 3 years ago

I feel like it'd take as little as a half-percent of the people writing new HTTP tooling to noticeable improve the DNS ecosystem as a whole.

you mean like the ppl that made it more centralized and failure prone?

belorn• 3 years ago

One major problem with new security technology is that testing is generally created as the result of failure, rather than preceding the failure so that it doesn't occur in the first place. Even very basic things like verifying that a new key actually fit the intended lock (ie, that a new key correspond to the published data) is a concept that people tend to implement only once they have had experienced a failure.

One general solution is to have standardized tools with standardized testing. Transferring a domain should always involve testing the old data, testing the old keys, testing the new data, testing the new keys, testing the chain of delegation, testing the chain of trust and so on. Some TLD's has gone as far as doing basic sanity test before a domain owner is allowed to transfer a domain. (industry term is to delegate a domain, since transfer is strictly about legal ownership)

Thankfully there are initiatives to create standard tools and standard test. Zonemaster (https://zonemaster.net) is one I recommend for testing, which is the result of a collaboration between .se and .fr. There is also initiatives for tooling that create multi-signing (https://github.com/DNSSEC-Provisioning/music), which is to have multiple keys operating for the same domain.

Redundancy in HSM's are nice, but redundancy is not a replacement for testing and backup procedures. It was one of the hard lessons learned by the IT industry, especially with data. No amount of RAID will save a company if a rogue script deletes all the files, or if the database becomes corrupt. A lot of experts learned the hard way that redundancy is mostly just a complement to sanity tests and backup procedures.

tptacek7• 3 years ago

This isn't new security technology. The service model for DNSSEC was designed in 1994.

tptacek7• 3 years ago

The basic idea isn't excellent. It's wrong. It's firmly rooted in mid-1990s cryptography (the original design was a DOD contract to TIS); in particular, to the idea that cryptography is too expensive to do on edge systems. That's why DNSSEC's core design has offline signers (my belief is that the supposed security benefits of offline signing are back-rationalized --- remember that a best-practices "leaf" DNSSEC deployment today will have online signers, to foil enumeration attacks).

If you were to scrap DNSSEC-ter and start from scratch in 2023, there is no chance the design would look anything like DNSSEC. It would be fully encrypted for privacy. It would strike a different balance between authenticated denial and offline keys. It would use different primitives, and it would probably have a different DNS RR schema.

There are lots of implementation problems with DNSSEC, but the real issue is that the design is the best we could do in 1998, and that most of what's "improved" since then have been more effective ways to shoehorn the same model into the evolving core DNS infrastructure, rather than really improving the fundamental design.

roadiase• 3 years ago

Dnssec is so hard to manage right

jongjong2• 3 years ago

Why isn't Blockchain tech used to solve the problem of domain name resolution? Why aren't these solutions getting any traction? Tokens would serve as a built-in payment system for purchasing domain names and also as a spam prevention mechanism, it's decentralized and therefore wouldn't rely on any centralized authority, the incentives for running nodes would be built in... They are highly replicated and can replicate without limit which means it would be able to scale to support unlimited reads (which is what matters most when it comes to a DNS system). It would be distributed geographically and therefore could provide the fastest lookup times possible as you could run your own node to get the lowest lookups for existing domains... Also, ownership of domains would be cryptographically secured. WTF is this not getting any traction? Why aren't browser makers adopting this or enabling Blockchain-based domain lookups by default? There are solutions like Unlimited domains but they're not actually being rolled out...

jongjong2• 3 years ago

I forgot the mention the most relevant aspect; that because you could run a node on-premises which syncs up with all the latest domains, may not even need encryption... But if you didn't want to run your node on-premises, you could have an encrypted tunnel to someone (or a company) which does.

rightbyte• 3 years ago

I think the main problem is the expiration dates. Like, why do they exist? I see no point in having a ticking bomb in your setup.

mike_hearn5• 3 years ago

The whole issue of expiration in cryptographic systems is extremely fraught. My gut sense is that X.509 certificate expiry is the most common cause of mega-outages these days, where by "mega outage" I mean the sort of thing that takes down entire cellphone networks or major online services for hours at a time. In the past I've tried arguing against this practice of immediately cutting all communication because a deadline expired, but cryptography as a field is highly resistant to business requirements analysis of any kind. No CEO in their right mind would ever agree to having expiry in their infrastructure but the decisions are made by people far away, very deep down in organizational stacks, and it honestly doesn't occur to most business-side people that anyone could be crazy enough to litter miniature time bombs all throughout mission critical HA infrastructure anyway.

Part of the issue is misalignment of requirements. Keys don't actually expire, of course. In the old RSA days there was a notion of a sort of slow rusting as factoring attacks tended to get better over time, so every so often you'd want to bump the size of an RSA key to keep it secure. But that wasn't something that had to be done on a deadline. When elliptic curve cryptography went off-patent and started getting deployed it fixed that, and a secp256r1 key made today is as good as one made 15 years ago.

So what is the purpose of expiry, then, in today's world?

For DNSSEC, it seems to be a kind of encoding of an assumption that keys can leak without anyone being aware of it, whilst the holes being used to do so get sort of ambiently patched anyway again without anyone being aware that they've done so. If you assume a world where keys are constantly being stolen and abused, then regular rotation can help. However, that's not the real world! These keys are stored in HSMs. Has there ever been a leaked DNSSEC key spotted being abused in the wild? Certainly it's questionable whether the juice is worth the squeeze given the outages it causes, and the way these undermine the entire system.

For certificates there's that, but the original justification was actually about constraining the size of CRLs. If certificates don't expire then in theory any decommd cert has to be recorded and stuck on a revocation list forever, which would mean they can never shrink. How exactly to handle phaseout of old keys and certificate details has experienced a lot of churn over the years with CRLs, OCSP, OCSP Stapling and nowadays CRLsets all doing similar tasks.

Still, this is primarily a logistics and cost management issue rather than an actual security problem. The task could also be done with gradual phase out in which at first expired certs only trigger warnings to developers via the browser console, Google Search console etc. Then later it graduates to a warning in the URL bar. Then eventually TLS stacks start increasing the latency on connections, etc. Then once artificial latency passes a certain threshold that's basically equivalent to an outage anyway and you may as well just return an error immediately.

Lately there's another reason for expiry, with short expiry times being enforced by the CA/B Forum so they can change the rules for certificates quicker, not keys. It took a long time to phase out SHA1 and introduce CT, the people who set the certificate rules didn't like that as their job is security, so by making certs constantly expire they hope to push people into setting up automated renewal pipelines which in turn means they can then do things like mandate new algorithms, change the contents of certificates and so on. Constant expiry certainly helps them do their job, but it makes everyone else's job harder as automated renewal isn't always easy especially outside the context of simple web servers that use LetsEncrypt.

Summary: If tomorrow I was temporarily made King Of The Internet and told that my only mission in life was to increase availability, I'd start by altering X.509 libraries to implement gradual degradation instead of hard expiry, e.g. with a sleep(time_since_expiry) call somewhere in the stack. I'd also default to simply ignoring expiration outside of TLS context, because X.509 gets used in a lot of places that aren't TLS and they're often hard to reach and fix e.g. certs that are hard-coded into embedded devices. You do NOT want your electricity grid to have an outage caused by certificate expiry. In those contexts it hardly matters because the PKIs are often custom and not very large, so CRL sizes just don't matter. You can easily list every revoked certificate for the entire lifetime of the system, if you even need PKIX revocation support at all, and for those use cases expiry is probably a much bigger business risk than attacks using obsolete algorithms.

teddyh6• 3 years ago

If a key is ever potentially compromised (say, a key employee quits), you should rotate the key. If this is too hard, people will just not do that, and we will have a world of hurt, with old keys being hard-coded all over the place. But you know what they say, “if something hurts, do it more often”. If key rotation is hard, automate it. Then, when a key really does need replacing, it’s easy. I prefer the pain of key rotation over the pain of all keys being hard-coded everywhere.

tptacek7• 3 years ago

There's some irony here: short expiry has some clear practical benefits for at least some of the goals of the WebPKI, and that works because the WebPKI has stakeholders that will push things like "if you issue one more SHA1 certificate we will distrust your CA". Certificate expiration failures are common, but not that common, because "needs a certificate" isn't the default for every host.

Key expiry is a much bigger problem for DNSSEC, since once you sign, every hostname in your whole zone depends on no RR in your DNS configuration expiring. But there are no stakeholders extracting any value from those expirations: nobody can say "if you sign one more record with RSA-1024 we will break your whole TLD".

Further, unlike with the WebPKI, where verification is conceptually offline --- you have a trust anchor and a signature chain and the verification is all done locally --- DNSSEC resolution is online: you have a TTL issue with revocation, but not a "this bogus signature will be valid for an entire year because there is no effective revocation system".

So, arguably, the RRSIG expiration field is another fundamental flaw with DNSSEC.

teddyh6• 3 years ago

Exhorting people to not screw up things like DNSSEC KSK rollover clearly hasn't worked, so the only real solution would be better ways to automatically recover from it.

No, the solution is to automate the rollovers, so that they can’t be screwed up by manual handling. <https://github.com/DNSSEC-Provisioning/music>

teddyh6• 3 years ago

Also, validation errors is how you get people to ignore XHTML.

Strict validation is tricky. With HTTPS, validation was initially very lax, and AFAIK, most SMTP servers still do not validate the certificates of the destination SMTP servers. It is hard to get people to move to stricter validation once habits and working systems have been established which depend on lax – or no – validation.

Some try to avoid this by specifying strict validation in advance, like XHTML and DNSSEC, which obviously has its own drawbacks.

tptacek7• 3 years ago

Like DNSSEC, XHTML is an infamous global technology failure.

teddyh6• 3 years ago

My point exactly (well, not about DNSSEC being a failure, just XHTML).

DNSSEC failures are how you get people to disable DNSSEC

Related Stories