A Caddy Cert Expired Because Systemd-Resolved Was Selectively Broken

https://rant.mvh.dev/a-caddy-cert-expired-because-systemd-resolved-was-selectively-broken/

by PaulHoule • 2 months ago

41 33 2 months ago

rant.mvh.dev

farfatched2• 2 months ago

There was interesting commentary on https://lobste.rs/s/z0ozbb/caddy_cert_expired_because_system...

Was there? I see 14 comments, of which only 3 have any value at all which are: fanf finding an irrelevant bug, oliverpool explaining the use of LetsEncrypt staging, and hexadecimal suggesting setting up expiration alerts.

The rest is:

  - People bitching about systemd and replies to their bitching totaling 7 posts (literally half the comments) 
  - conversational detritus like "well written blog post" and the whiteboard comment

eqvinox8• 2 months ago

There's this adage about consuming whatever newspaper reporting, everything seems fine and dandy, until one day they report on something in your field. Say, chemistry, and you're a chemist, and... it's all wrong. Even the basics.

My field is networking, roughly from Ethernet physicals to TCP/UDP.

systemd has no f*cking clue what they're doing on networking. You need to not use systemd-resolved, and not use systemd-networkd or systemd-timesyncd either.

I really hope they are service manager people and know how to write a service manager, and I have no counterindications on that. But don't let them touch your networking, aside from service-managing on that.

(My recommendation for DNS - on servers - would be to install unbound locally and use that. It's not great for clients since it doesn't deal well with frequently changing network connectivity, or rather, it does, but only if you restart it on network attachment change, which implies flushing all cached data.)

gucci-on-fleek2• 2 months ago

systemd has no f*cking clue what they're doing on networking. You need to not use systemd-resolved, and not use systemd-networkd or systemd-timesyncd either.

Can you give some more details, or do you have some examples of where systemd is bad here? Because I'm quite happy with systemd-resolved and systemd-networkd, but I admittedly know hardly anything about networking.

[...] But don't let them touch your networking, aside from service-managing on that.

What do you recommend as an alternative then? systemd-resolved and systemd-timesyncd are easy to replace, since there is lots of good DNS/NTP software available, but systemd-networkd seems trickier: I don't like NetworkManager, I'd rather not write a bunch of ad-hoc shell scripts, and I'm not aware of any other alternatives.

eqvinox8• 2 months ago

Can you give some more details, or do you have some examples of where systemd is bad here?

I can't, because it's death by a thousand cuts. I remember systemd-timesyncd having security issues, but not what exactly. systemd-resolved was initially missing basic security best practices (source port randomization, if I remember correctly), despite their being well established and well known in the DNS community - AFAIK the systemd people just didn't know of them.

systemd-networkd seems trickier

Indeed; I was primarily speaking about servers, and there my answer is: either ifupdown-ng or ifupdown2. (The latter if your networking needs are more complex.)

On client devices the choice is between bad and worse. Personally speaking, I consider NetworkManager bad and systemd-networkd worse. So I'm using NetworkManager on my laptop... it still occasionally breaks IPv6 for me.

I will admit: this is burned-in experience for me, learned over several years, and the original reasons have expired from my brain's cache. And I probably didn't give systemd-networkd much of a chance (I don't remember anything on that, for resolved and timesyncd I have at least vague bits), with that being more recent (at least for me) than systemd-resolved and systemd-timesyncd.

Is it possible it has gotten better? Definitely. But TFA indicates at least systemd-resolved hasn't.

kotaKat2• 2 months ago

hot take "predictable network names" should have been a kernel flag - give us all our eth0 in peace. i shouldn't need to set a flag to get back a default feature.

just what i needed to do: configure ens328452356aflhjdslhfda to get my network going.

vga1• 2 months ago

If I ever have a networking problem in a personal Linux system, it gets solved by removing the weird networking thing that the distro decided to use and move to full systemd-resolved / networkd. In my experience, that works every time.

I'm not doubting commenter's expertise. It's entirely possible that resolved and networkd are utter horror inside but still work from at least this user's POV.

farfatched2• 2 months ago

There's this adage about consuming whatever newspaper reporting, everything seems fine and dandy, until one day they report on something in your field. Say, chemistry, and you're a chemist, and... it's all wrong. Even the basics.

Gell-Mann amnesia effect: https://en.wiktionary.org/wiki/Gell-Mann_Amnesia_effect

systemd has no fcking clue what they're doing on networking. You need to not use systemd-resolved, and not use systemd-networkd or systemd-timesyncd either.

I'm not sure how to reconcile that with enjoying each of those:

systemd-resolved is the default on many distributions, and does DNS-over-TLS stub resolution for me. What do you suggest as an alternative that handles changing network connectivity, and DoT?

* systemd-networkd is very well documented/organised. I find it pretty useful on servers.

* systemd-timesyncd doesn't seem to cause issues for me. It being an SNTP client (rather than NTP) seems appropriate for distributions. I would like to move to NTS or Roughtime at some point.

justsomehnguy7• 2 months ago

My recommendation for DNS - on servers - would be to install unbound locally and use that

And now your developers are running around and cursing you because nothing works anymore.

Because Docker silently retargets the interna Docker resolver to 8.8.8.8 if it sees 127.0.0.1 as a resolver address on the host.

Because people who wrote Docker have no fucking clue how the system works.

NB: see https://news.ycombinator.com/item?id=47441785 to solve the Docker issue with a local resolver.

It's not great for clients since it doesn't deal well with frequently changing network connectivity

This is is something Linux specific I guess, I run Unbound locally on my Windows laptops for years and never had a problem which would require the Unbound restart.

which implies flushing all cached data

It doesn't really matters in 2026. Just look in your cache and note the default TTLs for like 90% of records.

zaphirplane• 2 months ago

Some docker issues archaeology says that there was tension between docker team and redhat / systemd

eqvinox8• 2 months ago

Running on 127.0.0.53 isn't a systemd-resolved exclusive magic trick, you can just configure unbound to do the same thing.

quectophoton• 2 months ago

My recommendation for DNS - on servers - would be to install unbound locally and use that.

At least on Ubuntu 26.04, you can't easily bind Unbound on a WireGuard interface because of services dependency order, so the Unbound service errors during system boot because the WireGuard interface doesn't exist yet. And IIRC neither `ip-transparent` nor `interface-automatic` fixed it.

On Alpine Linux all this just works.

gucci-on-fleek2• 2 months ago

What's the server's hostname? If it's the same as the domain that's not working, perhaps you're running into this bug [0]? If that's the case, then adding

  # /etc/systemd/system/systemd-resolved.service.d/override.conf
  [Service]
  Environment=SYSTEMD_RESOLVED_SYNTHESIZE_HOSTNAME=0

might fix it.

[0] https://github.com/systemd/systemd/issues/34897#issuecomment...

A Caddy Cert Expired Because Systemd-Resolved Was Selectively Broken

Related Stories