Why is DNS still hard to learn?

https://jvns.ca/blog/2023/07/28/why-is-dns-still-hard-to-learn/

507 254 3 years ago

shon2• 3 years ago

It’s not. It’s one of the few things that hasn’t changed much and it’s operation is fairly straightforward.

dig is a little confusing. It’s more capable but less straightforward than good old nslookup (which still works fine BTW).

I think partly DNS and the core protocols may seem confusing to younger people in the industry because so much stuff “just works” now.

For example, today wifi routers “just work” right out of the box. In the early 2000s it would have taken a network engineer with knowledge of DNS, IP, Ethernet, RFC1918, actual routing protocols and whole bunch of other stuff to set something like that up and they’d have well known how it worked and why it was configured the way it was.

If you think DNS from a client can perspective is confusing, try configuring BIND ;-)

/OldNeckBeardRant

Arcanum-XIII2• 3 years ago

Yeah, BIND is hard to configure. Unbound/nsd are so much easier to deal with (once you find the correct documentation which is an exercice in frustration)

The principle behind DNS are not that hard, once you understand it's recursive. Now to configure it with security in mind, the proper infrastructure and the final details... lot of things to learn, but not that hard. Without BIND I mean.

KRAKRISMOTT• 3 years ago

Many modern APIs are more ergonomic and easier to use due to the benefits of hindsight. A redesign and upgrade of DNS is long overdue.

mrits• 3 years ago

A network engineer or a teenager motivated to communicate with his girlfriend when not at his dads office.

cduzz• 3 years ago

These things you're talking about are a small fraction of DNS though.

For instance, you lookup "thing.behind.cdn.it" and get one answer, someone else looks up the same thing and gets a different answer. Pretty obvious, but when someone asks the reasonable question "can you open a firewall hole for thing.behind.cdn.it"

Some servers forward requests, some delegate, some will look stuff up for you others won't. And there's the magic with search domains on clients, and if clients or internal resolver libraries will honor TTLs or not.

There's also the myriad different types of records, and sometimes the server will tell you to reconnect in TCP instead of UDP, etc.

So -- DNS is pretty complex; it has the illusion of being simple because it works so well and most of the fiddly bits are abstracted away by stuff that mostly just works.

bityard• 3 years ago

Pretty much what I came here to say. As a young system administrator, DNS was the second thing I learned after setting up my first Apache server and I didn't find it hard to learn at all.

I will admit that it when you get to a certain point, you have to be careful not to shoot yourself in the foot when operating a production system but that is a slightly different concern which is more implementation dependent. Eg BIND.

hiAndrewQuinn• 3 years ago

Re/ knowing older protocols, I recently took a few weeks to read _Networking for System Administrators_ and take+review copious Anki card notes. It's incredible just how much more confident I feel around understanding networking at a high level, including both DNS and all the stuff underneath it, like `ethtool` and Ethernet frames and stuff.

I suppose this isn't surprising, since knowing things "from the ground up" is why I went for electrical engineering instead of CS in college.

gjvc2• 3 years ago

20 years after doing a CS degree, I wish I had done EE instead.

monksy3• 3 years ago

I just wanted to add on to what you're saying:

I think partly DNS and the core protocols may seem confusing to younger people in the industry because so much stuff “just works” now.

I've noticed it's become much worse since universities have been teaching Python to start with and with the whole aggressive comodization of developers. To some extent the social justice polices inacted in our communities to exclude people*. ("unsavory" people)

We no longer have the culture where we had kids in early ages get a desktop, learn the ins and outs, play video games, trying to pretend to be a hacker, etc. We're getting developers who barely can script in javascript, barely do html, ignore the edge cases, and generally don't have a lot of interest in the craft. It's pretty frustrating to see this.

NikolaNovak• 3 years ago

I may agree with your point but don't understand the social justice aspect at all; makes it feel like it's something just added in. I think both our perspective as a society and thereforo educational goals have changed over decades, laterally to any social justice aspect. If nothing else, there are order of magnitude more developers of all sorts today than 25 years ago, both as absolute numbers and as relative percentage of population. Basically the enthusiastic nerds who geek out about anything and everything and want to be hands on are still there, but also a LOT of people who just want a paycheck or just have more limited / specific interests. And they do ok. And education system will provide for them.

But I just don't see e.g. for-profit bootcamps with their simplified curriculum as any kind of social justice project.

Arcanum-XIII2• 3 years ago

I don't know, I would be curious to meet out of school dev from 20 years ago. Those I meet at my current job are... young. The lack of experience shows.

I guess at the time I was not better. Different, because using C I could still destroy hardware. Hard to program a CGA card from Python :D Maybe with MicroPython on an arduino now?

alexjplant• 3 years ago

When I was 14 I (poorly) administered an Active Directory environment with mail, web, and CIFS for a restaurant without understanding DNS or DHCP. Instead of setting the WRT54G's DHCP server to hand out the domain controller's static IP as the DNS server for proper name resolution I just used IP addresses and host file entries to make everything work. I also had the MX record for the domain set to the router's WAN IP and didn't have any PTR records set - the fact that e-mail delivery went as smoothly as it did is an absolute miracle in retrospect. A few years later I figured out how DNS actually worked and in my early 20s I inherited a corporate intranet where BIND was used as the nameserver for all external corporate domain zones. Moving this setup to VPSes for increased reliability taught me a _lot_ (mostly zone transfers, SOA, etc). I'm grateful for the experience but these days everything is pretty much done for you so this is a low-value activity... "IT" isn't valued the same way that "software engineering" is for better or worse.

bdavbdav• 3 years ago

Same boat at a similar age. (Ab)using AD taught me a lot about DNS, largely after the fact. Lots of “so that’s why that didn’t work…” moments later down the line.

TheNewsIsHere5• 3 years ago

This is why I continue to maintain that ops and SE excel at remaining distinct fields, though I certainly don’t mind overlap.

Most of my friends are software developers and/or software developers working on cloud-based stacks.

Two of those friends lead the platform engineering groups at their respective companies. One of them has a very basic understanding of networking and could figure out how to do subnetting that didn’t come out of the box, but they’d need to stop and go learn that.

That isn’t everyone by any stretch, but I see it more often than not these days.

I agree fully with the idea that younger professionals aren’t as used to the infrastructure underneath the infrastructure being as complex as the higher layers of the OSI model that they’re frequently more experienced with.

What I hope we don’t end up with is a future where all the data center people and all the network engineers (and so on) are almost exclusively employed by a small number of mega-corps. It’s important that knowledge and experience in the fundamentals of the networked world remain widely distributed and openly accessible.

donretag• 3 years ago

That is what I assumed as well, until one day I got hit by a bug involving Extension Mechanisms for DNS (EDNS). Never knew it existed. All of a sudden DNS was failing and could not understand why. Took me a long time to fix the issue.

orangepurple• 3 years ago

If that is the case why is the recommended way to use DNSSEC is to turn it off?

https://www.fastmail.com/blog/dnssec-dane/

redeeman• 3 years ago

In the early 2000s it would have taken a network engineer with knowledge of DNS, IP, Ethernet, RFC1918, actual routing protocols and whole bunch of other stuff to set something like that up

You remember things differently than I

icedchai3• 3 years ago

Me too. Plug-and-play consumer wi-fi routers were common place by the early 2000's.

fps• 3 years ago

in the very early 2000s, home routers weren't a thing. Cable modems hooked up to a single computer. If you were a business, you got a PIX, but home setups were frequently done with a computer that had 2 ethernet ports and either used Windows's "home internet sharing" or Linux's ipchains and NAT. This was typically fine, because very few houses had multiple computers. I knew many people who would get a separate cable modem for each computer in their house.

By the mid 2000s, Linksys started coming out with their little WRT routers, which were affordable by home users and mostly just plug and play.

deathanatos• 3 years ago

Yeah. It was a bit different from today, but not by much. Really if anything, stuff has regressed since then.

Our setup in the early 2000s was the same as today, except the AP, router, and modem were separate devices. This was somewhat beneficial: today, my monopoly ISP forces me to use their all-in-one junk box. There's not even a possibility of competition, and lo and behold, I don't think the forced-box does anything more than the three pieces of equipment did. (Except, finally and only as of the last year, IPv6. But perhaps if there had been real competition, that'd happened a decade sooner, esp. since I first saw it in '07 on a non-ISP connection…)

Combining the AP/router I can get behind. It's when it became ISP domain that it went south.

The 90s were rougher, as the networking stack in Windows was a bit more terrible. I have vague nightmares about installing drivers for TCP … where the TCP driver was somehow specific to the network card. That sentence doesn't even make sense by today's standards. Stuff got better in the 2000s.

icedchai3• 3 years ago

I agree. I remember learning about DNS when I was a teenager. And I've been running my own authoritative DNS servers for almost 30 years now. Remember the O'Reilly book, "DNS and BIND" ? It's still out, those this would've been first edition, around 1993.

xtagon• 3 years ago

The downside to things that "just work" is that they become magical black boxes where learning how they work isn't a requirement until things really go wrong.

yodsanklai• 3 years ago

I think partly DNS and the core protocols may seem confusing to younger people in the industry because so much stuff “just works” now.

Younger people aren't dumber than old one, they build even more complex stuff on top of these old abstractions.

vichle• 3 years ago

Because all great developers create More complexity (-:

shon2• 3 years ago

Agreed. Not dumber, just building higher level stuff.

The value of deeply understanding all 7 layers has diminished because generally things are standardized and work well so it’s not required. This frees up some calories to work on higher level stuff like LLMs. That’s good.

It’s also good to encourage some people to understand the wisdom of the ancients in case SHTF.

vel0city• 3 years ago

Linksys was making home routers which were about as easy to deploy as any home router today starting in 1999. Their earliest Wireless G router came out in like 2002. The beloved WRT54GS came out in 2003.

https://arstechnica.com/gadgets/2000/08/befsr41/

deltarholamda• 3 years ago

It’s one of the few things that hasn’t changed much and it’s operation is fairly straightforward.

It's relatively straightforward, ignoring all of the potential ways that things can go wonky, e.g. random servers not respecting TTL.

But I'll never forget when Firefox put out an update with DNS-over-HTTPS turned on by default. All of a sudden, I was inundated with "Email is gone! Everything is broken!" because we run an internal DNS server handed out to workstations by DHCP. We have internal webmail and intranet Web servers that were just gone.

It took a lot longer than it should to figure out what was happening, partially because it's DNS! Why should things go blooey? But it's pretty clear that Mozilla did not anticipate this (easily forseen, IMO) sort of issue.

throw0101b3• 3 years ago

But I'll never forget when Firefox put out an update with DNS-over-HTTPS turned on by default.

Plenty of DNS old timers / neckbeards (e.g., Paul Vixie) warning that DoH was not a good idea; there was lively discussion on HN at the time.

We used split-horizon DNS as well and I implemented the "disable DoH" canary where I was working at the time.

* https://support.mozilla.org/en-US/kb/configuring-networks-di...

* https://support.mozilla.org/en-US/kb/canary-domain-use-appli...

arjvik• 3 years ago

Is configuring BIND hard just because it's got an obtuse zone and configuration format? Or because there are a lot of DNS-server-level decisions that need to be made?

creeble2• 3 years ago

See above. It's not only an obtuse configuration issue, but it has very little validation for issues that can get you in trouble, like CNAME s that also have A records.

dv_dt• 3 years ago

DNS concepts are pretty straightforward, but I agree with the article that there are a lot of little holes to fall into. No mention in any thread on dig vs /etc/hosts. Or of ISPs with bad actor DNS behavior... etc..

haroldp2• 3 years ago

Can you help me find the mistake in my zone file?

  $ORIGIN example.net.
  $TTL 900
  @    IN    SOA    ns1.example.com. hostmaster@example.com. (
        20230728001
        1800
        300
        3600
        172800
    )
  @    IN    NS      8.8.8.8.
  @    IN    NS      8.8.4.4.

  @    IN    CNAME example.com.
  @    IN    MX    10    172.253.124.27
  www  IN    CNAME example.com

throw0101b3• 3 years ago

Off the top of my head (haven't had to do zone files for ~2 years):

* hostmaster@example.com -> hostmaster.example.com

* NS records are usually hostnames (not sure if IPs are even valid)

* Ditto for MX records ; also add a period to the end, otherwise example.net will get appended

* Also appending with the www record

somat• 3 years ago

Also the CNAME if you have a cname you don't want any other records with the same name. It ends up being a confusing and ambiguous situation to be if. You are supposed to use the cname to jump the the actual record. but now there is also a MX record here are we supposed to do anything with it?

WarOnPrivacy4• 3 years ago

For example, today wifi routers “just work” right out of the box. In the early 2000s it would have taken a network engineer

Or a nerd buying a WRT54v1 to install hyperwrt.

giobox• 3 years ago

For example, today wifi routers “just work” right out of the box. In the early 2000s it would have taken a network engineer with knowledge of DNS, IP, Ethernet, RFC1918, actual routing protocols and whole bunch of other stuff to set something like that up and they’d have well known how it worked and why it was configured the way it was.

I think you are stretching how bad wifi was in early 2000s - sure its easier today, but in the actual year 2000 you could walk into a store and take home an Apple branded wifi base station (original AirPort unit) - 802.11b stuff of the era was largely as easy to connect new stuff to as today, generally with a passkey. It all largely worked with DHCP out of the box just like most routers today too, if anything the experience is much the same minus the faster speeds, slightly better range and encryption today. Oh and probably some kind of ipv6 support...

You certainly did not need network engineer level knowledge - lots of smart professional folks installed wifi in the 2000s, and solutions like the AirPort base station and many others were about as "turnkey" as they came.

mkeedlinger• 3 years ago

I found out about https://www.nslookup.io/learning/ recently, which greatly increased my knowledge of DNS. If you look at the list of DNS record types [0], you might be surprised at how many their are. Knowing how to use those can be a bit much.

[0] https://www.nslookup.io/learning/dns-record-types/

teddyh2• 3 years ago

the list of DNS record types

Actually authoritative list: <https://www.iana.org/assignments/dns-parameters/dns-paramete...> That list also has linked references for each entry, whereas the list you gave only has references for 9 of the 51 types it lists.

If we exclude entries explictly marked as experimental, obsolete, deprecated, or reserved, the list you gave is still missing these:

• AMTRELAY

• ATMA

• AVC

• DOA

• EID

• GPOS

• ISDN

• L32

• L64

• LP

• MINFO

• NID

• NIMLOC

• NINFO

• PX

• RKEY

• RT

• SINK

• SPF

• TALINK

• WKS

• X25

(I know, many of these are de-facto deprecated: SPF is abandoned for TXT, GPOS was replaced by LOC, and the entire usage of WKS was obsoleted by the recommendation of RFC 1123. But they are not marked as such in the list from IANA, and I still often see SPF records in the wild.)

Also incomplete, but often has better references: <https://en.wikipedia.org/wiki/List_of_DNS_record_types>

(Not to mention TYPE, which I have also occasionally encountered.)

labcomputer• 3 years ago

These all seem to be super-niche or obsolete though?

* ATMA, ISDN, NIMLOC, EID, X25, are all for relatively niche or obsolete physical layer protocols (I guess ATM isn't that niche, but most people never run into it).

* WKS, PX, NID, LP, L64, L32 seem to be defined but unused in practice (I had never even heard of ILNP, which what NID, LP, L64 and L32 are for, until I googled it just now).

* RKEY, NINFO, MINFO and several others are expired without adoption or never made it to an RFC

* GPOS is an earlier version of LOC

joshcafe• 3 years ago

Shameless side project plug: they mention a "debug" mode for dns resolving being nice to have. ComfyDNS has this in its web UI :3

https://comfydns.com/

It's the picture that says "TRACE google.com A IN" at the top.

ComfyDNS is partly scratching a personal itch - I was tired of hand modifying bind9 zone files. And also I was curious as to how DNS works - I knew surface level stuff but no details. So I implemented the RFC from "scratch" (I used netty but no DNS libs). It was a lot of fun.

(Also if/when the site goes down from hugging, forgive me, it is a rails app running on the oracle cloud free tier lol)

pgray• 3 years ago

the joke i've always heard is DNS combines 2 of the hardest problems in CS: naming things and cache invalidation

fauria• 3 years ago

you forgot the second, off-by-one errors.

Dylan168072• 3 years ago

It comes with a validity counter in seconds, and you can be very very loose about counting those seconds.

It's not the hard kind of cache invalidation. You don't really have to do "invalidation" at all.

And on the server side, it's perfectly acceptable to send a mix of old and new versions for a while.

andrewaylett• 3 years ago

That definitely helps things to work, but it makes it very much more difficult to work out why things might not be working.

Not least because an unexpected cache can lead to things looking like they're working when they're actually broken at source, as well as things looking like they're still broken when you've actually fixed them at source already.

dclowd99015• 3 years ago

It's not the hard kind of cache invalidation. You don't really have to do "invalidation" at all.

One of the points he brings up is negative cache, or caching the dns into a state that it won’t retrieve a resolved address even if it’s available simply because the negative case is cached.

Invalidation is definitely a part of it, mostly because you kind of can’t.

8organicbits• 3 years ago

You sometimes can perform an invalidation, but it's a manual process and you need to know who to ask. Slack did this when they botched their DNSSEC rollout[1]:

Our team quickly started contacting major ISPs and operators that run public DNS resolvers, with the request to flush all cached records for slack.com.

DNSSEC is another part of DNS that is still hard to learn.

[1] https://news.ycombinator.com/reply?id=36910054&goto=item%3Fi...

paulddraper3• 3 years ago

Yes, but not a joke.

TZubiri• 3 years ago

Non-issue

teunispeters• 3 years ago

One of the gotchas I encountered is that DNS is asynchronous, with possibly a long delay before reply. C apis make it look synchronous - which I think makes it harder to work with. There's also the detail that order of replies can be any. (I found too many developers expected synchronous and instant replies)

lanstin2• 3 years ago

with the caching, it's a nice bimodal distribution, 99.99$ 0 ms response time, 0.01% 30 ms response time (with a small chance of having that query packet be dropped, with retries in the 1000s of ms). I've seen people write caches that use the old value and kick off a new query in the background to hopefully populate the cache again.

NoZebra120vClip• 3 years ago

I remember the mid-90s when we were writing MUD servers and clients. You'd start the client, go "/world ZebraMUCK" and then the TUI would hang while the DNS name resolved.

So then we figured out asynchronous DNS (this was in the days when you linked with "-lresolv" on SunOS) and it was like a breath of fresh air! You could go "/world ZebraMUCK", control was returned to the keyboard, and even if it took 120 seconds to resolve zebramuck.nozebra.example.com, you could go about your business, like in another world, or issue some other client commands.

And client developers learned a little about select(3).

bratgpttamer2• 3 years ago

I feel like DNS is one of the more straightforward protocols, especially on a practical level, and especially given that most interfaces are a dropdown and two text boxes.

I have noticed a lot of developers shy away from it, probably because they don't use it much or it's not their job (rather than it being hard).

paulddraper3• 3 years ago

"Dropdown and two text boxes" undersells it.

Here is the list of several dozen record types: https://www.iana.org/assignments/dns-parameters/dns-paramete...

bratgpttamer2• 3 years ago

Sure, but I'm talking about a day-to-day practical level. Most people will only ever need to modify A/CNAME, occasional MX and TXT, and maybe an SOA/PTR.

Even the more arcane record types (as far as I've ever used them) are essentially key-value pairs with the record type analogous to a namespace.

naniwaduni• 3 years ago

Yeah, that's the dropdown.

msie• 3 years ago

Exactly. Why waste time learning something I will only use once or twice a year or 10 times in my career? Or that someone else (who is an expert) can fix for me?

dclowd99015• 3 years ago

I guarantee the problem space that dns solves is something you will run into in your career. Best to have some knowledge of systems like that so you can design them.

BeefWellington• 3 years ago

Of the similarly-aged protocols, I think it's the most difficult (which is not saying much).

SMTP and HTTP can be pretty easily done by hand, which makes them more accessible to a person learning the protocols themselves.

DNS the protocol is simple, but I do think there's something to be said for how complex it is if you want to say, set up your own domain from which to reliably send e-mail.

gjvc2• 3 years ago

I remember being stymied by the BIND zone files (which are plain text) being called .db files (they may actually be called anything -- some people used the .db extension). I had just learned about sendmail, and I knew it kept alias tables in Berkeley DB files (which are binary), which in sendmail parlance were also called .db files.

As Alan Kay is fond of quoting Bob Barton... "Systems programmers are high priests of a low art."

laserbeam2• 3 years ago

Here's what's cool about the article:

- Presents some nice theories which make things hard to learn (infrequent use, poor tools...)

- Describes how DNS tools could be improved.

- Gives you a few gotchas for how one may shoot themselves in a foot with DNS.

Here's what's a bit (not much) less cool:

- I really have no clue if those things ACTUALLY make things hard to learn (because it's not a research paper on learning).

- It's a plug for other content on the side which actually describes the DNS protocol. I'll admit the sold content looks cool. I haven't purchased and can't vouch for the actual quality.

dgb232• 3 years ago

As for the last point: Check out the author’s blog. She’s a real hacker and can convey technical things in friendly and simple terms.

laserbeam2• 3 years ago

Yup, I have no problem with the author or the work they do. I think it's cool. And there's still value in the article.

My only point is it slightly undermines this article per se. Claiming that something is hard to learn and immediately providing a thing that can teach it well is usually marketing. And that's all ok :).

andrewfromx• 3 years ago

sometimes it's hard to even know WHERE to change the settings. Last week a friend was trying to setup heroku with api.foo.com and he needed to add a CNAME to the domain so heroku would make the cert and turn it on.

I used dig, i used host, I used whois, I got invited to their aws route 53 and saw all sorts of stuff in there but each change had no affect. Finally I noticed from whois that the name servers weren't even aws they were google.

So they gave me access to the google account but no domains in there.

Finally I asked, have the CEO log in to his personal google account and sure enough, that's where the change could be made.

shadowgovt• 3 years ago

This. The protocol isn't hard, but the protocol isn't the service.

The service of DNS is a decentralized, distributed, held-together-by-spit-bailing-wire-and-an-unprecedented-post-WWII-era-of-international-peace-and-collaboration hash-job of individually configurable nodes kind of agreeing on a shared worldview of the information the service contains, unless your local government hates piracy or pictures of Winnie the Pooh, YMMV.

It's like saying "I don't know why people struggle with databases; SQL isn't hard" and then the database contains ten thousand tables, a thousand indices, a hundred functions and triggers, and all of it was documented by someone who built it and never had a neophyte review the docs.

Oh, and the database operates on eventual-consistency guarantees out to 24 hours of "eventually."

Spooky23• 3 years ago

DNS is easy in the same way that chess is. The game mechanics are straightforward, and it gets more complex from there.

DNS bears the burden of delivering you to complex IT systems. It’s abused in various ways to enforce geographic restrictions, service levels, etc. It generally works, so long as everyone upstream knows how to configure things so that downstream things they don’t know exist work well.

When things don’t work… that’s not easy.

gunapologist99• 3 years ago

It's probably a good idea for all IT people to have a working knowledge of how to debug DNS issues.

DNS has historically been a vector for significant security holes and it's likely that this will continue to be true for the indefinite future. These holes also lead to other vectors in nearly every other protocol like SMTP. Even the CA system used for HTTPS is highly dependent on a basically insecure protocol. (Would you notice if your bank bought a DV certificate instead of OV? likely not)

So, perhaps it's not such a bad thing that it seems hard to learn to those who don't have enough interest, since even now we see people building DNS things without taking the time to really understand the history of things like port randomization, cache poisoning, AXFR, etc.

dgb232• 3 years ago

It seems to me that everything which broadcasts/asserts routing decisions in a network (any layer) is deceptively simple and potentially dangerous.

chasd002• 3 years ago

deceptively simple and potentially dangerous

Also, there's not a lot of people keeping the whole thing running. iirc there's only like 13 or 14 root DNS servers on earth.

tristor3• 3 years ago

I don't agree with this article. I think DNS is something few people take the time to learn, but it's not actually hard to learn. One of the great things about DNS is that the system itself will tell you about it's internal state in response to queries. It's very easy to inspect a DNS server for a known zone and understand how it works, and there's very good tooling that's free and widely available to do this (like dig).

It's always been a big surprise to me that my DNS expertise is what seems to be most memorable for a lot of folks I've worked with through my career, when I don't believe I know anything mystical or special. DNS is extremely well standardized, the most common server and client implementations rigorously follow the standard, and it's very easy to inspect with free tooling. It just takes some effort and time to learn, but it's not really hard.

jeroenhd4• 3 years ago

I think it is hard to learn... using the tools people used to learn DNS with.

BIND is great at what it does, but its configuration files suck and its manual is long, terse, and unnecessarily complex sometimes. Dig is powerful, but abbreviates everything like we're on an 80 column terminal. At times Wireshark was a better tool debugging DNS issues than Dig was.

Give someone PowerDNS or another modern DNS server and I think they'll have a much better time configuring a working DNS server. I don't know a good modern DNS client, so I've learned to deal with Dig instead. As a user of the "--color" flag for the `ip` command, I'd love to see tools like dig produce more modern output (I'll alias the command line flags, just add it to the command!)

Seriously, "MSG SIZE rcvd: 71" did not need abbreviation. "flags: qr rd ra" could've been full words as well. I don't know what the semicolons before the lines are supposed to convey but they're only making things confusing.

I find it no wonder people get confused learning DNS with the materials provided to them.

knome3• 3 years ago

I don't know what the semicolons before the lines are supposed to convey but they're only making things confusing

all of the lines that aren't part of the query answer are prefixed with semicolons. so it's basically a comment character. presumably to ease processing of the data it spits out.

You know. So you can `dig google.com | grep -v '^;' | grep . | awk '{ print $5 }'` easily.

I can imagine people using it in a shell script 20 years and more ago to grab a list of IPs to do some domain's health check or whatever

Not that you would want to in any modern stack. obviously you'd just use dig's `+short` option :)

chefandy9• 3 years ago

I'd conservatively estimate 90% of the people who make core FOSS software interface decisions haven't had to learn anything technical in an entirely unfamiliar domain where there existing mental models didn't apply in decades. Beyond that, many consider having learned these arbitrary, terse interfaces as a badge of honor, and for some reason thinks that makes them better technologists. I'll bet they'd be even better had they been able to focus on the core concepts rather than trying to get into some systems engineer's head who worked at Berkley in 1984.

j16sdiz3• 3 years ago

BIND expose (almost) every details of DNS. That's why learning their config would teach you 90% of the DNS system.

dig output make sense after reading the RFC. It exposes every bit flag in the protocol level

j16sdiz3• 3 years ago

rcvd, qr, rd, ra

are defined in the RFC. It need the abbreviation to fit in the ASCII art in RFC text.

back in those days, dig users are those who read rfc

zehemer• 3 years ago

As a user of the "--color" flag for the `ip` command, I'd love to see tools like dig produce more modern output

https://github.com/ogham/dog is pretty good in that regard

manuel_w2• 3 years ago

  $ ip address --color
  Command "--color" is unknown, try "ip address help".

dataflow• 3 years ago

To the extent I learned DNS, it was via nslookup, definitely not dig.

firexcy• 3 years ago

I think old CLI tools are hard to use because they were designed to be machine- and human-readable at the same time. They are effectively APIs in some scenarios (e.g., scripting and as linked libraries) and in others handholds. As a result they are restricted to be barely viable as either.

neilk• 3 years ago

How did you learn DNS? And when?

Terretta• 3 years ago

How did you learn DNS? And when?

I wrote one of the world's first dynamic DNS servers for our dialup modem customers in the early 90s, so when connected to our Livingston Portmasters with an assigned IP address you could be username.isp.com while online.

Later updated this so you could also be http://www.username.isp.com all the time just by dropping an index.html file in your ~username/site directory.

I learned both by spelunking BSD, reading man pages, and rapid prototyping in Perl. So I'm with tristor, it felt relatively straightforward.

// I don't say I wrote the first because I don't know. I am not aware of any other regional or national commercial dialup ISP in the U.S. that offered this before we did.

tristor3• 3 years ago

I had a slow day at work in the beginning of my career (I was a help desk monkey then) and I wanted to learn more about how Active Directory actually worked. One of the things that I found out and became more curious about was why DNS was so critical for Active Directory, upon where I discovered the magic of SRV records. After that I decided I wanted to learn DNS in detail, so I read the RFCs, poked around with nslookup and dig, and then set up a local copy of BIND on a Linux VM to experiment with. In the numerous intervening years I've picked up more over time and been involved in numerous situations where I had to assist someone by doing DNS spelunking, often learning something in the process. I jumped from being a help desk monkey to being an L2 sysadmin in my very next job largely on the basis of my DNS and Active Directory knowledge, all of which was learned by reading documentation and poking at things, nothing particularly difficult about either.

What always causes challenges is weird edge cases, but if you understand how things are supposed to work at a protocol level you have the basis of knowledge to troubleshoot the edge cases, which is how you learn about the more esoteric or implementation specific behaviors of things. I wouldn't expect anyone to particularly have deep knowledge of AD or DNS edge cases, but the basics of how they work aren't particularly hard knowledge to acquire and it's a constant surprise to me how few tech people understand DNS in particular, since it's a critical system for nearly every aspect of modern computing.

As far as resources, when learning these things way back when I only used the official Microsoft documentation, the RFCs, man pages, and sites like nixCraft to learn about it. These same resources still exist (although Google sucks now and makes anything decent hard to find), and these protocols have changed very little in the intervening nearly 20 years, so I don't think it's any harder for folks to learn today than it was for me back then. This stuff is very literally not rocket science, DNS is an extremely basic protocol. In fact, nearly all its weird edge cases is because it's such a basic protocol that some behavior isn't clearly defined, however /most/ behavior is. It's very nearly all text on the wire, and all of the zone records, queries, and responses are human readable. As someone else mentioned, you can just use Wireshark and learn about DNS. It's inspectable on the wire, not just through tooling.

FWIW, Active Directory is far more difficult to learn in detail than DNS, largely because Kerberos is deeply complex, and Active Directory has many unique complexities (like how FSMO roles work). DNS in comparison to most other things you are expected to know as even a semi-competent systems administrators / SRE is mindbogglingly simple.

dahfizz2• 3 years ago

I was tasked with setting up & maintaining a dnsmasq server in college, which handled DNS, DHCP, and pxe booting.

I learned it like anything else. Googled it, read the docs, and trial & errord my way into a working setup. I was comfortable with all the technologies in a couple weeks (working part time).

WarOnPrivacy4• 3 years ago

Setup an in-house server to sinkhole malware requests.

mrweasel• 3 years ago

For me: Volunteer systems administration for a student organization. We had a subdomain under the domain of the university and wanted to add another one. We already had Bind configured with an A and MX record. No one really wanted to deal with it, so I asked some people I knew, who recommended that given we already ran qmail, then maybe tinydns would be the best option.

So I installed tinydns, read the documentation, experimented with the subdomain we didn't use yet. Took a few tries, but not really difficult.

Later one we added more stuff, like zone transfers, all of it is pretty easy once everything is running.

One of the things that scare people away from running DNS servers is Bind and it's configuration language, it's not great. You just don't have to use Bind, or you can switch to it once you understand the concepts of DNS better.

Vaslo• 3 years ago

You have the curse of knowledge my friend. It’s hard to learn and way more complicated than it needs to be.

dogleash• 3 years ago

I think DNS is something few people take the time to learn

I kinda agree and think DNS one of those technologies where you can go an entire career without picking up more than bits and peices here and there. Those things gains a sense of mystique in industry as more complicated than it otherwise would if more people had to tackle it full on.

WarOnPrivacy4• 3 years ago

I kinda agree and think DNS one of those technologies where you can go an entire career without picking up more than bits and peices here and there.

As far as that's true it's weird, because DNS basically does one straightforward thing. But then you get into all the places where that one thing has to be done in different ways.

Where I wouldn't mind some more magic is with reverse DNS. Too many tables don't know a name pointing to an IP until something tries to resolve that name.

Reliably historical rdns would be even more awesome but that's more of a service than a spec thing.

josho• 3 years ago

If you read the article the author points out why it's hard to learn. The concept is easy, but when teaching the concepts we don't include all the details of the modern internet.

As an example what are the rules that your browser uses to cache and expire DNS entries? Are those rules consistent between browsers?

antonjs• 3 years ago

And does your browser have settings which bypass or supplement the host's DNS configuration. Secure DNS (DoH etc) is great, but damn that's confusing when you first run across it. Not to mention how phones do it; you can't override a DoH DNS server when connecting to a VPN which offers internal DNS on Android, for instance.

f1shy• 3 years ago

Absolutely agree. Back in the days, I was very inexperienced, I was thrown to the task of administering DNS (with BIND) and Sendmail. I had 100+ servers. The first couple of month was a lot of reading, and understanding things, but relatively fast I got a good understanding of it. After 6 month I was teaching DNS to other teams in other countries for the same company. It was not at all hard. I'm a very average engineer, and from 0 to explaining to others in 6 month, is by no means a difficult topic.

pessimizer• 3 years ago

DNS isn't at all hard, it only takes two months to learn when you're being paid to, and after only 6 months you'll be knowledgeable enough to teach!

from 0 to explaining to others in 6 month, is by no means a difficult topic.

You should have seen almost anything else. Many things can be learned within days.

jvns• 3 years ago

Hello! I wrote this post and I have a couple of things to say about this "DNS is not actually hard" take. It took me many years to feel totally comfortable debugging DNS problems, and I wrote this post to explain why I think it was hard for me.

I also used to think that "no, actually, it's easy!" was an encouraging response to "this is hard to learn". And I kind of get it! I love DNS! I think it is surprisingly simple in many ways, and I've written about that a lot, for example in https://implement-dns.wizardzines.com which shows you how to implement a toy DNS resolver from scratch in some pretty simple Python code.

But over the years I've learned that "no, it's easy to learn!", instead of coming off as an encouraging comment ("you can do it!"), often gets received as "no, it's not hard, actually the problem is that you're dumb". Like, I've been confused about this for years, and you're telling me that, no, actually it's easy? Not that helpful!

So I've stopped telling people that, and instead I put a huge amount of work into trying to understand _why_ people find certain things hard and work to help remove some of those barriers.

anyfoo• 3 years ago

Maybe everyone sees this differently? Or maybe is more than one viewpoint to it.

For example, if someone tells me that something is "actually really simple", and I did not get it yet, I tend to think that I likely (not certainly, but likely) have not found the right approach to that something yet, and once I found that way to look at it, things will resolve itself.

But if people assure me that something is "hard to learn" or "hard to get", I'm rather suspecting that that thing is full of unwieldy complexity and/or unintuitive aspects, and unless I'm either up for the challenge (definitely the case sometimes), or learning it is crucial for achieving a certain goal (in which case it may turn out to be fun after all, even if still hard), I might think twice if I want to put the time and energy into it.

In the case of DNS, which I've learned in the nineties, I found the principle simple and elegant, the tools and software unwieldy and sometimes arcane, and a large multitude of little details that are not strictly necessary for understanding and using DNS in basic ways, but that can get important when either debugging or working at scale.

So, in the case of DNS, I'd say the answer is "depends on how deep you want to go", but don't shy away from the simple and elegant principle, even if the arcane software (it has gotten better, though) stands in the way.

I think that matches the content of your article overall?

riffic• 3 years ago

an aside your content is top notch in quality and I always point to your site(s) when I want to share how this stuff works with an easy explainer. You make it seem easy :)

13of40• 3 years ago

Just my 2 cents: I worked on the DNS in Windows Server back in the 200x's (validating the UI and back-end functionality, not dev work), so I did have to learn about it at one point and I recall that there was a vicious level of complexity to it compared to what we ask it to do. The 99% scenario is turning a domain into an IP via an A record. Next after that is doing the same with an MX record. Beyond that is this deep well of other things people thought someone might want to do with DNS, but that almost nobody takes advantage of in real life. So DNS is probably easy to understand if you're just concerned about the one or two first class scenarios, but if you had to implement an RFC-correct instance of a DNS server, it's a bit of a brain melter.

redog• 3 years ago

I get it. Your take resonates with me. Playing piano is easy too right? Right after you can do it. Until then it's mystically hard.

It just takes practice. Training helps but nothing can replace practice.

We can't jus b gud

parentheses• 3 years ago

jvns, your articles and conversational writing style are great. I hear you saying "'X is hard' is a way to not lose a reader by insulting their intelligence". My immediate reaction is "'X is easy' is a reminder that things that look hard upon first examination can be quite easy".

I think in general this level of "kid-gloves" is overall unhealthy for society. It encourages mental weakness.

At the same time, I understand that your aim is to reach as large an audience as possible so your educational material can have max impact. For this reason, I can justify you having this stance. That said, I don't think it applies globally.

emmelaich2• 3 years ago

Perfect response!

My own response to "X is actually hard to learn" is ... compared to what?

   1. compared to your first impressions?
   2. compared to general opinions that you're aware of?
   3. are you talking about general principles or deep down stuff?
   4. are you talking about the learning curve?  Is it irregular? Steep? Shallow?

"X is not hard to learn" is ultimately a pretty empty statement.

emmelaich2• 3 years ago

My own hassles with DNS, some of which are mentioned.

    1. windows and apache httpd and others cache things
       for a long time, seemingly forever.
    2. The 'A' record is used in case of missing MX records. Who knew that?
    3. (in the olden days) confusion with winbios and other naming schemes
    4. The defn of 'authoritative'
    5. the misleading semantics of 'domain' in resolv.conf
    6. (minor) the casual use of CNAME to mean alias, whereas it fact
       means the opposite and is used to refer to the record, not the name.
    7. That DNS can use tcp/53.  The number of misconfigured firewalls I've
       dealt with argh.
    8. Split vs forwarding DNS.  e.g. impact on the .dev
       domain.
    9. The urge of some to put underscores in their names.
       I believe that some (early?) implementations allowed
       them.
    10. Someone putting in a lower 'serial' in named.conf and
        the struggle to restore sanity.

sitzkrieg• 3 years ago

i thought to post about how i thought "dns looks fairly straightforward" (which i guess it is on paper) then learned all the nuances over the years of maintaining a few services across a few domains and debugging network issues as a networking fw dev. i paused and the difference is perfectly between the the parent and grandparent post really.

some things seem deceptively easy and you can get by for years on basic features. then it happens, you try something new you learned you can do with $THING , add to it and things start to break and you dig in and realize you never fundamentally understood how the basic features that got you here worked. and thats ok

codeisawesome• 3 years ago

Julia, I agree with your take here, thank you for bringing this thought process into the mainstream. There are many other concepts this happens with and in other ways: people sing praises of the pedagogic excellence in Skiena’s & Sedgewick’s algorithm textbooks (and the awful EPI book by Aziz et al) but I hate reading them, they’re actually written horribly to explain things, IMO. I think this form of difficult writing on top of already difficult concepts is a weird type of gate keeping (perhaps sometimes unintentional?) and it should be tackled all over Computer Science by better writers who have an explanation style like yours.

einichi• 3 years ago

Simple is not always easy. Too simple can be difficult to grasp and sometimes adding a little complexity can actually make something more intuitive and easier.

DNS is simple, but it is not necessarily easy to understand.

txutxu• 3 years ago

As always, thanks for the article (and the debate).

DNS itself, as an internet protocol, is "easy" in principle.

I did implement and maintain DNS services at the beginning of my career, for more than 200 customers, without any title and without help, except "the internet".

Everybody understands "you ask for a name, you get an address to connect as response", or "you can have more than one MX entry and play with the weight", "what is a TTL", etc.

But as every internet protocol... as soon as you add to the recipe: historical changes and more than one RFC, upgrades to the protocol, cryptography, different client and server implementations, vendors, operative systems and libraries (and versions), levels of compliance, software applications, misconfiguration, bugs, layers and more layers (inside client, reaching intermediate server(s), final server issues), cache!, enterprise interests (from client to server side) with a wild global (or local) positioning, government and agencies interests, ageing, innovation, all issues related to the long running opensource projects related to the protocol, all issues related to the "new is better" opensource projects (client and server) related to the protocol, user/developers/customers/managers asking (or assuming) impossible things or that there are no limitations, a long history (and a future one) of known attacks and known defenses or mitigations, response ordering (/etc/gai.conf), clients that implement from the last century /etc/hosts to mdns and avahi to nsswitch and ncsd, clients that act as servers (developed by enterprise bug creators)... you know... not easy anymore... things gonna be wild.

If there are complementary protocols, let's multiply the complexity: for DNS issues, there is also domain registration issues, whois, DHCP, VPNs, intermediate firewalls, rate limits, anycast, replication, IP, ARP...

With any internet protocol, or anything related to computers:

1) While it works, it's fine (and everybody is happy)

2) While we get an issue previously seen, understood and fixed, it's still calm

3) While we get an issue that we can quickly search by symptoms and fix, maybe ok

The "not easy" comes after that.

When we get a new and unknown problem, a weird issue, unexpected things, and there are 20 or 30 possibly failing actors around the protocol, or XX involved parts.

Recently I did a written breakdown of all things related to DNS in the "strace -y -yy -fq -v nc -vz google.com 80" in a linux laptop, and many people at work was surprised of all the things that happen there, inside the "give me the IP" step locally.

We could say exactly the same against any issue related to any internet protocol:

HTTP? it's easy or hard

Depends. It's easy to explain in a sentence what it does.

But you don't want to be the guy fixing issues against a float of load balancers used by all the customers, with a legacy (non compliant) device from a high priority customer at 4 A.M. oncall.

tristor3• 3 years ago

Howdy Julia,

I debated whether I should respond to you at all, however I feel that I must. You and I don't know each other, and based on the last sentence of your reply I suspect we agree with one another more than we disagree. Nonetheless, your reply seems to imply that my comment and my earlier disagreement were done out of malice or arrogance. A different commenter used the term "gatekeeping".

Fundamentally most of the building blocks of the Internet that people interact with regularly, including DNS, are well-defined and relatively easy to understand /relative to other technical things/. Protocols like DNS are effectively shoving text into packet. All of the hard stuff isn't in the bare protocol, it's in understanding the complete stack of abstractions the protocol relies on, the abstractions that allow the protocol to be simple but can sometimes not work in unexpected ways. If there is anything I've learned in my life, it's that integrations and scaling are what make technical things hard, the basic protocols are generally very simple and they are robust primarily because they are simple. They are simple primarily because they get to rely on all the underlying abstractions to be there. Technology now exists in a world of abstractions on top of abstractions, and understanding the basic building blocks is now rare since many technologists focus on mastering a particular layer of abstraction, yet if you understand the basic building blocks, even in an incomplete sense, it can greatly empower you to be more capable and comfortable with the abstractions.

"Hard" and "Easy" are relative terms, and I do believe that saying things are hard can lead people to avoid them altogether. I have spent a considerable portion of my life working diligently to mentor people in their technical careers, foster technical understanding, and bring people along with me on knowledge journeys. I have done this through extensive written documentation (internal to companies), teaching classes (internal to companies), public conference talks, guest lecturing at universities, and mentorship of people who are interested in learning technology but come from non-traditional backgrounds. I myself come from a non-traditional background. I've also worked hard to improve the UX and accessibility of every piece of software I touch, because I think that technology as a tool and the information about how it works should be accessible.

I really appreciated your article and I think you have a great writing style to explain things in an understandable way. Your Implement DNS project is really great, and I will be linking it to others in the future as they go through the process of learning DNS. In no way was my disagreement before or this reply meant to throw shade on you, your article, or otherwise. It's because I believe saying things are hard actually discourages people from learning them, leads to avoidance, and in particular I believe DNS is one of the more simple of the many possible things someone can learn about how the Internet works. I am /very much/ not trying to in any way disparage the intelligence of anyone who is struggling with learning any technology. I believe that learning technology is much like learning anything else, it requires interest and time. Saying that some of the simpler things are hard can discourage people from having the interest, and if people do not have the interest they will not invest the time.

Very specifically, I think for the audience here on HN, they've had to learn and use technologies that are far more difficult than DNS in their lives and careers with an almost certainty. In a relative sense, I do not believe DNS is difficult to learn. DNS has a /lot/ of edge cases though, which can be hard to troubleshoot, understand, and resolve, so from this perspective it is hard to master and I wouldn't claim to have fully mastered it myself. As an example of what I mean, another commenter mentioned Git. I have a pretty deep understanding of Git as well, yet I consider it significantly more difficult to learn and understand than DNS, largely because DNS is better documented, easier to inspect, and doesn't require the technologist to understand complex algorithms like merkle trees. Yet almost every tech worker in the world uses Git every day, often with many frustrations and weird edge cases, as in regular usage it fails far more often than simpler technologies like DNS. The entire strength of DNS from a robustness, adoption, and resilience perspective is largely its simplicity, which is why I think relative to the many other things that exist that it's not hard to learn.

Our opinions are always colored by our experiences, as is mine, which I stand by. One of those experiences I'll recount here, which is that I have known a woman for a little over 6 years now who is currently working in IT and finishing a bachelor's in IT through WGU. She's one of the top performer's on her team and has a gift with understanding and explaining abstract concepts to users, which has made her well-liked by all at her company. Not long after I met her she shared with me that she'd always wanted to go into IT, but felt like she couldn't do it because everyone she'd met along the way had told her it was too hard for her because she suffers from a learning disorder called dyscalculia which interferes in her ability to do math. For reasons I don't fully understand, math ability is used by many technical people as a determinant of your ability to learn and work with technology. I was one of the first people she'd met in her life that encouraged her to pursue her interest in technology and told her she could do it. I did so by pointing out that many of the things people were saying were hard were concepts she already understood in the abstract and offering to go on the journey with her. She had been so dejected by others telling her things were too hard for her she never pursued a college education. I invested significantly in learning with her and 3 years ago she finished an associates and multiple certifications and started her first IT job, and she's been excelling in it the entire way.

I've continued to mentor her, along with many others, because I legitimately do believe that much of the technology around us is much easier to learn and understand than people commonly believe, and that understanding how it works is a key to understanding how our world works now and is incredibly empowering even for people who don't work with technology every day. I am adamantly opposed to any sort of gatekeeping of technology knowledge, and in no way am advocating that gatekeeping or trying to disparage those who struggle to learn. If anything, one of my existential fears is that as a society we fall into a situation where so few people understand the technology we rely on every day that it creates a new and more dire social gap beyond the wealth gap, a knowledge gap that fundamentally separates those who can effectively participate in the levers of power (including democratically) from those who cannot. I am heavily invested in ensuring that knowledge is free, accessible, and that there is an open pathway for people to learn and it's one of my core optimisms about the Internet.

nvahalik• 3 years ago

Seems like maybe you are both saying: DNS the protocol is easy to understand well. DNS in he wild is much harder to understand well.

acheron• 3 years ago

I wish I had jobs where people cared that I knew DNS.

Aeolun• 3 years ago

DNS itself is easy to learn. Trying to figure out why domain x doesn’t resolve to ip y is a hard problem.

Like the article points out, there’s so many layers of potential caches in between you starting a lookup and that lookup being resolved.

temporallobe• 3 years ago

Likewise, a lot of people know (and use) me for my supposedly deep knowledge of Git, even though it took me years to fully understand and feel comfortable enough with it so that I’m no longer terrified when using the CLI — I can cherry-pick and rebase with the best of them. YET, I still feel it’s a bit intimidating and that there’s some mysteries behind the internals I don’t fully grasp. I suppose like DNS, it does take time and effort to learn, and when I really think about it, it’s actually not that hard, but for some reason, so many devs struggle with it, mostly (front what I’ve observed) because it’s intimidating due to o it’s somewhat odd terminology and unintuitive workflow, but also because there are so many GUI/IDE tools that hide a lot of the complexity, which simultaneously good and bad.

mekoka• 3 years ago

I don't agree with this article.

But you address none of the article's points. You're basically disagreeing with the title. And I'd say that your answer seems to conflate DNS is hard and DNS is hard to learn.

It just takes some effort and time to learn.

How much effort, how much time? Many people in these threads say that they learned by implementing a server. Some mention that it only took them some months to grasp, but then just echo in unison that "once you understand it, it's actually pretty easy". So can we agree that for something as ubiquitous and conceptually simple, it's actually hard to learn?

kmoser• 3 years ago

I think DNS is something few people take the time to learn, but it's not actually hard to learn.

Hard disagree. For me, DNS is like doing taxes: I touch it once a year or so, find it Byzantine, know enough to be dangerous, but am always frustrated that I don't use it often enough to remember exactly how to configure things without having to consult poorly written and/or overly technical tutorials.

I'd like to see a better version of web-based tools like mxtoolbox.com that will analyze DNS records, let you know what's wrong, and give you actual examples of what settings you need for things like DMARC/DKIM/SPF records. In my experience, online tutorials for setting them up come tantalizingly close to giving me what I need, but I often end up getting stuck with the last few details (usually the weird punctuation required) because, again, I touch this stuff so infrequently I just don't remember from one time to the next. Ideally I'd want a form-based tool that gives you drop-downs to select from and, when submitted, just gives you the actual record you need.

denton-scratch• 3 years ago

I also don't agree with the article. Many aspects of sysadmin are far more difficult than DNS.

the most common server and client implementations rigorously follow the standard

Not servers! in particular, many servers mess with the TTL (as author notes). It's not that these servers are defective; it's that the hostmaster has interfered with the configuration, presumably to reduce the load.

lazyant• 3 years ago

DNS _protocol_ and _servers_ are conceptually easy. In practice, the implementations and what actually happens is not. When you type google.com in your browser, do you know exactly the workflow in your computer and the different caches it uses (browser, OS etc) and how they work?

Do you know which Linux tools use gethostbyname vs getaddrinfo and why they could give different results?

andrethegiant• 3 years ago

Julia also created this comic about DNS, which is helpful for learning: https://wizardzines.com/zines/dns/

roomey• 3 years ago

There's an old saying: if someone tell's you they know how DNS works, they're lying

1bent• 3 years ago

djbdns is simple, easy to understand, easy to configure; it embodies a clear understanding of how DNS works.

Unlike BIND and dig, it was designed after DNS had been in use for a while.

Like sendmail, BIND suffers from being designed before anyone knew what it would need to do.

fullstop• 3 years ago

I still use tinydns, but I've moved on from dnscache to unbound.

djbdns was a great tool, clearly built with security in mind, and it forced you to understand how the whole system worked. It struggled with things added later like txt and srv records but they could still be added.

qmail was also well ahead of its time.

rconti2• 3 years ago

and logging in hex is awful and should be punished

jesuspiece• 3 years ago

+human flag is a sick idea, would love to see that PR.

inb4: you do it

iAm25626• 3 years ago

I used to specialized in these service enable services. DHCP/DNS/AAA/LDAP and etc; low level stuff tend to get take for granted. It's not difficult but there are much nuances. Not typical visible to layer7/Front end development per se. Embrace the "and". If you are a FE/BE dev AND understand system/dns/network. You just set yourself apart from the next person.

swayvil• 3 years ago

Because complexity is easy. It practically generates itself. It has a natural economic advantage.

Making sense of complexity, otoh, takes much diligent effort.

So complexity tends to win and everybody tends to befuddlement.

dap• 3 years ago

I went through this a few years ago, deciding that I had only a piecemeal understanding of DNS based on the specific things I had run into. I knew about `dig(1)` and BIND and a CS101 idea of how recursive DNS resolution works, etc. But I was missing the working knowledge needed for designing and implementing anything non-trivial or for debugging non-working systems. So I read "DNS and BIND" (not quite cover-to-cover, but close, skipping over the details on some of the more exotic features). I set up a real BIND server for some unimportant personal web sites. None of it was hard, but it did involve a bunch of (time) investment. To be clear, BIND is not the right thing for many use cases, but a lot of DNS ideas and terminology still come from BIND and I found that stuff very valuable.

I think books are underappreciated for learning stuff like this. Most resources you find on the web are high-level theory ("here's a block diagram of how recursive lookup works"), task-oriented ("how do I get `dig` to do a recursive lookup"), or otherwise low-level (e.g., reading the source of your local DNS client to understand its retry policy). To understand the pieces and how they fit together, from the theory (what they're trying to achieve) down to implementation (what caches exist where), I find there's no substitute for a holistic approach that you usually find in books, but rarely (not never) find on the web.

magicmicah85• 3 years ago

"(like, what if you could run dig +debug google.com and it gave you a bunch of extra debugging information?)"

The flag they're looking for is +trace and it provides exactly what they're looking for which is a path of how resolution occurs from root nameservers all the way down to the domain.

supportengineer• 3 years ago

This was an outstanding writeup, and I have felt the same frustrations in many other contexts. I enjoy going the "make your own, friendlier tools" route.

kabdib• 3 years ago

DNS itself is pretty simple. Implementing a basic DNS server or client is not a big deal.

The management software around it is what is generally terrible.

jl6• 3 years ago

It's one of those things where there is a mismatch between how easy it seems to be, and how hard it turns out to be.

We all use DNS every day, and it seems really easy. The everyday language of DNS is: domain names, lookups, IP addresses. This language is exposed in browsers for all to see, and through this exposure we develop a mental model of how we think it works.

But under the covers there is a whole new language: zones, resolvers, delegated authority, that weird dot after the top-level domain...

dclowd99015• 3 years ago

It’s the distribution part that makes it hard right? There’s a shit ton of dark magic happening above the atomic level of an individual node that both introduces the majority of the complexity and also the majority of the obfuscation.

DNS is easy. An organization agnostic distribution of information is _really_ tricky.

gerdesj2• 3 years ago

"that weird dot after the top-level domain"

That weird dot is called root. Without it, a name is unqualified, with it the name is completely defined. That means that context is everything. Without the dot, a resolver might add the resolver's domain or parts of it, repeatedly.

Now, you and I know exactly what: host.example.co.uk is supposed to mean but without the trailing dot a resolver could try to look up host.example.co.uk.example.co.uk

Windows out of the box, if this happened would also try host.example.co.uk.example.co then host.example.co.uk.example and then host.example.co.uk.example, then host.example.co.uk. and get a result. However I never saw Windows actually try the first effort and I think the behaviour was designed to deal with large corp with DNS federated monstrosity Active Directories.

Your browser is probably toddling off to a DNS over https (DoH) server these days without your say so and canoodling with all sorts of ne'er do wells. Your DNS lookups are basic data - your ISP used to love seeing where you go. Your OS vendor (if you buy your OS) obviously can pass back "telemetry". Mr Google, doesn't own the desktop but does own the browser, so by ensuring you use "safe" DNS servers for your browser instead of whatever you have configured, its all good. All these shenanigans does make IT troubleshooting far more exciting than it used to be.

I shouldn't worry too much about trailing dots. You will almost certainly not be using the DNS servers you think you are. I get why DOH was invented and there is a good reason for some "civilians" to use it - ie a non IT specialist using a nasty wifi hotspot will be protected from some harm by their browser going home securely to do DNS lookups. However is it up to the browser vendor to trample all over the user's choice of Endpoint Security?

DNS is way more complicated than simply looking up addresses. Its about money these days (actually it always has been since around 2000) and there are now a lot of very opinionated mega corps who want to decide who profits off you.

briHass2• 3 years ago

You might actually be surprised, or at least I was, that browsers -- and typically worse, IoT junk, still generally respect the DNS server(s) served to them via DHCP or they use plain ol' 53 to the hardcoded Google/Cloudflare standards. Some (like the Facebook Portal) are tricky enough to try DNS over TLS, but insofar as my big-list-of-common-DNS-IPs in the block rule is accurate, nothing reaches out over DoH unless explicitly configured to do so.

Like you, however, I see the writing on the wall. Being able to easily see (and block) how these devices/operating systems/apps relentlessly and obsessively phone home to upload as much telemetry as they can get their hands on is not something these corporations want. Eventually, they will all just use your connection for the bandwidth and pass everything over a VPN to hardcoded endpoints.

hackboyfly• 3 years ago

I actually tried to learn about sockets today. I gave it around 3 hours but I never reached the end of the rabbit hole which led me to give up.

mgaunard• 3 years ago

DNS is hard because cloud/web companies insist on taxing you to do very basic things.

jstx1• 3 years ago

People who use their DNS knowledge often - what is your job and problems do you solve with your DNS knowledge?

m30472• 3 years ago

I use DNS to define topology and services (what you'd expect) and of late I'm using it for federating telemetry (the actual data; think of "tags" in the industrial control sense).

I've used it as an observable for asset discovery and classification, as well as for characterizing infrastructure.

m30472• 3 years ago

In a kind of corrollary of Dunning-Kruger, there is often a chasm between what people think DNS is and all of the things it can be. The article points some of that out (stub resolvers, even different implementations of libc); recursive versus authoritative responses; the recursion process; recursive versus authoritative servers.

A lot of deployment implementations are "bottom of the barrel" and aren't correct to begin with, although they work for the intended purpose. There is no checklist for server implementations which I am aware of (I've asked where people should know!). There's a lot of folklore which persists because if it works it's presumed correct.

There's "DNS" and then there's "The DNS", the "one true root" with arbitrary restrictions on the contents of labels. There's political interference with implementations in terms of the "one true root" doctrine, which interferes with marrying a resolver which e.g. queries a control plane and serves an application plane (where they have different roots): there are e.g. notions of forwarded zones, but there is no notion I am aware of of a "always recurse and lie that you're authoritative" zone (you can hack source code to accomplish this of course).

Even MITRE ATT&CK doesn't always get it right. They had it listed that DNSSEC "traffic" had to be examined with SSL tools, until I pointed it out (I didn't get credit). DNSSEC related records are ordinary DNS records, nothing is being encrypted.

I could go on: the experts were wrong on (UDP) frags, anycast...

neilv• 3 years ago

At one point, I had to make and maintain my own DNS zone text file, knowing only a little. Today, cheap providers give me various Web forms interfaces, but that doesn't mean it will always "just work", so knowing a little can still help.

I still had to break out `dig` the other day, when a DNS provider started answering records days past TTL, breaking email and Web.

To know when if/when that happened again, I looked around for a monitoring service, but didn't quickly find one that did what I wanted (and was overwhelmed by SEO, and very aggressive robo-sales emails from one of them), so I wrote a script that essentially runs `dig` and `diff`: https://www.neilvandyke.org/check-my-dns/

The monitoring script discovered one of a provider's DNS servers had ongoing problems, so I sent them a monitoring report, saying I wasn't a DNS expert, but perhaps, if there's a problem, the report would be helpful to their DNS experts. This seemed to immediately get past any "have you tried rebooting Windows" front-line flowchart response that I wouldn't have been surprised to hear, and they said they're working on it.

echan00• 3 years ago

I thought you talking about aws lol

ertian• 3 years ago

'Still'? There's a famous talk[1] in the network community about how DNS complexity is growing so fast that effectively nobody can keep up with it. There's so many competing, overlapping, and overriding RFCs that it's hard to make sense of it anymore.

[1] https://blog.apnic.net/2018/03/29/the-dns-camel/

Joel_Mckay• 3 years ago

DNS has been repurposed for everything from security validation to load balancing.

DNS over HTTPS and DNSSEC attempted to address some longstanding issues, but in the end everyone still has a host they know is going to get hammered harder.

Not too difficult to understand, but it is complicated given the number of sub-optimal use-cases that emerged. =)

chasd002• 3 years ago

i hate to be that guy but it's not hard to learn. The tools are just from a different era where expectations where... different. However, even in my day BIND was avoided in favor of other servers like that one by the qmail guy.. can't remember his name.

this comment reminds me of one of my favorite Dilberts

old guy watching Dilbert at his computer: you kids today and your fancy graphical user interfaces. Back in my day all we had were ones and zeros ...and sometimes we didn't even have ones.

Dilbert: you had zeros? we had to use the letter 'o'

egberts1• 3 years ago

Doing:

bastion named server

Multizone

Hidden Master

Split-Horizon

IXFR/AXFR firewalling

DNSSEC and local network

resolv.conf hijacking

Remote admin security

All of above would and should be hard.

pavel_odintsov• 3 years ago

Great article. Thank you!

I used to develop and maintain one of the World largest DNS services and summarised my experience with DNS protocol in this blog post: https://pavel.network/please-stop-using-dns-protocol-for-you...

thunderbong• 3 years ago

Because the only three hard problems in computer science are cache invalidation and naming things.

And DNS is a caching system for names of things.

https://reddit.com/comments/15c2ul2/comment/jtty9dy

yard2010• 3 years ago

Also, off by one errors, which is implied

pphysch• 3 years ago

To be fair, DNS is one of the best examples of "naming things done right".

It's globally-curated (IANA), hierarchical, federated, easy to modify.

chaps• 3 years ago

One of the best, but also strangest explanation of DNS I've seen is from "A Cat Explains DNS". It's wonderful.

https://www.youtube.com/watch?v=4ZtFk2dtqv0

mqus• 3 years ago

For me, the issue isn't so much DNS itself, but more that there are more things that try to resolve names on your PC, like /etc/hosts and all the stuff you put into resolv.conf.

Then you read online that you should try dig or so and those tools don't really match up with what your tools do(=resolving via libc), because dig really only does the DNS part.

cafard• 3 years ago

Some years ago, I had a miserable couple of days when an internal DNS server quit working. Eventually I traced it to a corrupted cache file. Once that was deleted, life returned to normal.

Though unpleasant, it was an enlightening experience. I had never imagined how much of the internet ran on Amazon by then (2010?).

nineteen9992• 3 years ago

Another one of these articles. We learnt very easily back in the 1990's when the Internet was smaller, and the computers were much much slower and less capable.

DNS, LDAP, SMTP, IMAP etc were the bread and butter of ISP's back then and people actually referred to the official documentation (RFC's etc). You had to learn them if you wanted to run servers on the Internet at all, and with a bit of an investment of time (ie. your paid time on your job) you learned it.

This generation of developers and devops people don't have the patience or initiative and expect to be spoon fed and just cut and paste crap from StackOverflow and various low value blogs. Rather than learn the infrastructure that the Internet is built on, they grab the latest fashionable wrapper tool of the week, follow some shitty blog instructions, and then cry foul when it all falls apart and they cost their company lots of money. Just because they didn't take the time to learn the foundations of how things actually work on the Internet.

I've seen it time and time again. It's not actually that hard kids. You just need to do your homework.

lannisterstark2• 3 years ago

Ah another one of those "dese damn kids and millennials I tell ya hwat, back in my grandpappys days we used to mine our own copper before laying them lines!"

nineteen9992• 3 years ago

The "get off my lawn" vibe was totally intended. Doesn't change the fact that there are a bunch of "AWS architects" out there running Terraform and building future disasters because they don't properly understand the infrastructure that the whole thing sits upon.

It's just like the morons who think that dynamic linking should be abolished for everyone, because they don't understand the use cases for distributors from a security standpoint, developers/users who require binary modules, or how to use their distributions packaging tools to avoid conflicts, and are too lazy to learn. But oh no, "ma DLL hell". Good grief.

1vuio0pswjnm7• 3 years ago

I'm not a "developer" and I learned DNS without any problems. Therefore agree with other commenter that DNS is not actually difficult to learn. I like the output from DNS utilities such as BIND and tinydns format.

DNS is worth learning for any internet user, IMHO. I've written primitive utilities that when used together can do stuff none of the popular DNS utilities can do. I use these every day.

Here's DNS challenge for readers. Try to look up the domain "bootable.com". Use whatever is the preferred method.

People writing about DNS often compare it to a telephone book. IMO the way most people use DNS is more like "directory assistance".

IP addresses do change but by and large most stay the same for long periods. Old zone files, publicly available scans^1 and other bulk DNS data collected from web crawls and public DNS caches comprise the "phone book" for me. Absolutely essential.

1. Sadly, in recent some of these sources have changed to non-public. No phone book for you! Call directory assistance.

rconti2• 3 years ago

DNS is not actually difficult to learn.
tinydns format.

You earned my disagreement right there!

hairui• 3 years ago

TIL that `dig` does not have TLDR page https://github.com/tldr-pages/tldr

derrickrburns• 3 years ago

Use chatGPT to understand dig output:

Explain this : dig +human +norecurse @8.8.8.8 google.com HEADER: opcode: QUERY status: NOERROR id: 11653 flags: qr ra records: QUESTION: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

QUESTION SECTION: google.com. IN A

ANSWER SECTION: google.com. 21 IN A 172.217.4.206

ADDITIONAL SECTION: EDNS: version: 0, flags:; udp: 512

ChatGPT This is a result from the "dig" command, a DNS (Domain Name System) tool used to query DNS servers and retrieve information about domain names. Let's break down the output:

dig: This is the command being executed.

+human: This flag is not a standard dig option. It is likely an extension that is meant to provide a more human-readable output, making it easier for people to understand the results.

+norecurse: This flag tells the DNS server (in this case, the server at IP address 8.8.8.8, which is a public Google DNS server) not to perform recursive lookups. Recursive lookups involve the DNS server querying other DNS servers if it doesn't have the requested information in its cache. By using +norecurse, the server will only provide information if it has it directly available.

@8.8.8.8: This indicates the DNS server being queried, which is the Google Public DNS server located at IP address 8.8.8.8.

google.com: This is the domain name being queried.

Now, let's look at the different sections of the response:

HEADER: This section provides general information about the response. opcode: QUERY: Indicates that the query type is a standard query. status: NOERROR: Indicates that the query was successful, and there were no errors. id: 11653: This is a unique identifier for the query, used to match queries with their respective responses. flags: qr ra: The flags indicate that this is a response (qr) and that recursion is available and allowed (ra). QUESTION SECTION: This section shows the domain name and the type of record being queried. google.com. IN A: The domain name "google.com" is being queried for its IPv4 address (A record). ANSWER SECTION: This section provides the answer to the query. google.com. 21 IN A 172.217.4.206: This line shows the answer. It says that the IPv4 address of "google.com" is 172.217.4.206. The "21" is the time to live (TTL) value in seconds, which indicates how long the information can be cached by other DNS servers or clients before it should be refreshed. ADDITIONAL SECTION: This section provides additional information related to the query. EDNS: version: 0, flags:; udp: 512: This line indicates the use of EDNS (Extension Mechanisms for DNS), which is a set of extensions to the DNS protocol. The version is 0, and the flags are empty. "udp: 512" refers to the maximum payload size that can be used for UDP communication. In summary, the query asked for the IPv4 address (A record) of "google.com" from the Google Public DNS server at IP address 8.8.8.8. The response confirms that the IPv4 address is 172.217.4.206, and it is valid for 21 seconds.

ggm• 3 years ago

DNS is complicated because it's old, and has been continuously revised in flight, to uplift to TCP, DoH, DoT, add DNSSEC, EDNS0, TSIG, notify, ECS, EDE. It continues in one ietf WG having outlasted another, it has cross area review. It's political, and technical, and has its black helicopter cabals.

I predate the DNS, and I continue to befuddle myself about details of the DNS on a daily basis despite a life embedded in DNS.

If you love DNS join DNS OARC.

oneplane• 3 years ago

I think the article hits the nail on the head. DNS itself isn't hard, but learning real-world DNS is, due to the (as described) hidden nature of many parts between intending to do a lookup and getting the expected result.

Some of this also stems from the divergence of what used to be a 'default' way to be connected to the internet (one interface, one gateway, one DNS server provider) and what is now far more common like having to figure out how to actually do the lookup in the first place (connected to multiple WANs, sometimes concurrently like LTE and WiFi) and as a user, not actually being able to tell which resolving path was used (was it the browser? the standard interface in the system C lib? was it some local resolver or recursor that sits in between the lookup? do any of them have a local cache? do any of them add special options by default?).

Even if everything works, you'd still not be able to blindly trust if the query and response used the same path in one application as it did in another. Three browsers might all use something different, and the OS itself might use something different yet again, only to have mDNS add a fifth option.

rsync• 3 years ago

The question I have is:

Why isn't there a combined MTA/DKIM/DNS server ?

Why am I installing and configuring and running and monitoring all of these different server software packages ?

I'd like to have a single config file that handles everything required to run a mailserver.

If I have more complex needs I can split things up make them modular ... but I don't.

quesera• 3 years ago

Why isn't there a combined MTA/DKIM/DNS server ?

If the nameserver was part of the MTA, where would you configure a CNAME for your webserver?

I'd like to have a single config file that handles everything required to run a mailserver.

I think the kids call that Ansible. :)

mparnisari• 3 years ago

I love the flamewar that this post has started

There's the "it IS hard!" gang

and the "it's not hard, you're just dumb" gang

and the "it's not hard, you just haven't spent the time required to properly learn it" gang

readonthegoapp• 3 years ago

i had hella trouble with it, but not just it - all the related topics - reverse dns/proxy setups, 10 ways to configure it/them, cname flattening and the 15 ways to configure that, the 15 ways you can do it with cloudflare or not, dis/advantages of all those setups/combinations/permutations, etc.

doubt a course could teach me half of what i would need to know in some reasonable amount of time - say, two or three months.

but open to the idea.

StillBored• 3 years ago

As a nslookup user since before dig, i find the default dig output to be less than ideal. The top->bottom flow, the "important" information is always the third option on a line, etc.

Basically how do I put this, uh, its obviously written by someone for themselves and people who use it everyday.

So, for most things I continue to use nslookup, it gives me what I want 99% of the time, and for me its easier to parse.

So, various other people have said something similar but it reminds me of a couple utilities in AIX that an office mate of mine wrote. The thing was a very sharp tool but the 3 cases that comprise pretty much 100% of its usage required three+ mode selection parameters, etc just to print the equivilant of "its working" or "its not working". The result was 50 characters of typing that quickly turned into a shell script with two parameters. I sorta have the same thing to say about the "ip" command which showed up at the same approx time as dig. Yup it can do everything, but its a terrible UI too for all the same reasons. The defaults are ugly/hard and the hard stuff still takes 10 mins reading the man page for the once every two years you need to do it.

So worst of both worlds.

PS: I sorta feel the same way about bind, its written for people who run big public DNS servers, the rest of us are better served with "unbound" or the half dozen other utilities that are more oriented towards users looking to run an in-house split dns/caching resolver setup.

minusf• 3 years ago

behold the dns camel:

https://datatracker.ietf.org/meeting/101/materials/slides-10...

pjungwir• 3 years ago

Wow, her "human" output for dig is wonderful. I learned nslookup first, and to this day I still use it because dig's output is such a mess. If dig looked as she suggests, I would have switched years ago.

ks2048• 3 years ago

I find the syntax and formatting confusing. The article mentions a desired "+human" flag. How about DNS tools that output everything in JSON? I want to see the structs as key+value.

sigjuice• 3 years ago

Wireshark

mike31fr• 3 years ago

DNS-noob here: For what use case is it useful to debug DNS like described in this article? I'm a programmer since 2001 and I never ever had to investigate what is going on behind name resolution, I never encountered any kind of DNS-related issue.

lorenzotenti• 3 years ago

https://isitdns.com/

Croftengea• 3 years ago

DNSSEC was already mentioned, but yet another dark corner of DNS are wildcard records.

udev4096• 3 years ago

A fun and colorful explanation of how DNS works. https://howdns.works/

eftychis• 3 years ago

tl;dr: suggestion: DNS is really hard to unlearn.

I do not agree with the implied sentiment of the article. And thus with some of the comments (from "both" sides) of the arguments presented here in the comments.

Hard is not defined, but is implied to be "harder than you believe it is."

That is a general characteristic of perhaps most topics worth pursuing in our field: computer science/engineering. (Defining it in this closed field space on purpose for brevity.) And especially in distributed systems, which DNS in some way falls under.

Further you have business interests, which have affected and will affect DNS -- we like it or not. And have added on top of it.

A number of individuals then use hard as "it is intractable" [to learn].

Is DNS harder to learn than BGP routing: I think not.

Does one need to spend a decade focusing on using and debugging DNS. I would argue not.

Does one need more time to dive into BGP than DNS to do state of the art work: I would argue yes.

DNS is simply not the hard and hot scientific problem people go around tackling from a research perspective.

It is a federated business (and political perhaps) problem at this point.

I do agree there is a lot of caching involved, which creates a ton of coherence and invalidation issues. That does not make the topic hard, it makes DNS problematic. (The other word I would use, and that I think Ms. Evans [the author] implies perhaps.) I agree totally with that.

I also think it is a point of approach the learning path to it for the average scientist here. A lot of people are introduced to DNS (or email to that extend) as: here is the sketch idea, we make sure it works, trusts us, and there is this single deified entity that makes sure it works, and it is not patched with other branches "constantly." All of the aforementioned sentence of course is a lie. But a lot of engineers learn that. (I was lucky to not have that introduction.) And then it comes time to debug a DNS issue -- when they actually realize it is a DNS issue.

So I would recommend a revised blog title to: DNS is really hard to unlearn.

deafpolygon• 3 years ago

DNS is like Othello. Simple to understand, but takes a lifetime to master.

Why is DNS still hard to learn?

Related Stories