Why Certificate Lifecycle Automation Matters

https://www.hezmatt.org/~mpalmer/blog/2024/01/30/why-certificate-automation-matters.html

by pabs3 • 2 years ago

103 29 2 years ago

Macha2• 2 years ago

I do wish the ACME ecosystem was as fire and forget as it's always aimed to be. It feels like an annual occurrence where I need to fix... Something misfiring with acme cert issuance and I've tried certbot, acme.sh, Lego and they've all had issues. Like maybe when first issued the tool decided to use ZeroSSL but on reissue decides to use Let's Encrypt and fails because one requires an email and the other doesn't. Or the verification method gets deprecated and requires manual software updates. Or the tool decided to rename it's environment variables and auto updated so now it can't find it's DNS auth keys.

akerl_2• 2 years ago

Interesting. I was using acme.sh and switched to lego, largely because I manage my servers with Puppet and acme.sh’s config format didn’t lend itself well to config management.

I renew certs for several boring things like nginx instances, but also things like my printer’s web portal, a Synology NAS, and a Unifi console that require bespoke commands to reload the cert.

I’ve got monitors that fire if any of my certs drop below 27 days before expiry, and over the past ~2 years, 100% of the problems have been with the custom hooks. I’ll log into the server and find that it did reissue the cert, but the command to make the Synology ingest the new cert failed for some arcane reason. The servers where it’s just running systemctl restart nginx or similar don’t fail.

Macha2• 2 years ago

Huh, the environment variable thing was specifically aimed at acme.sh which rather arbitrarily changed the config value from ACMEDNS_UPDATE_URL to ACMEDNS_BASE_URL, never acknowledged this in a changelog and then silently failed after an automatic upgrade as recommended by the default install:

https://github.com/acmesh-official/acme.sh/commit/2ce145f359...

It's also cleared out my .account.conf files when run on the suggested cron.

I've started using updown which also monitors my TLS certs simply because I no longer trust the process to work as documented.

Aissen• 2 years ago

I'm using certbot and acme/autocert, and it has been a few years since I last fiddled with this (even though I have CD happening !).

I was a bit mad at certbot for having chosen snap as its recommended installation platform, but my solution to use pypi + CD seems to be quite stable (famous last words).

The go services I run with autocert have had no manual intervention ever since the ACMEv2 move.

placatedmayhem• 2 years ago

EFF offers certbot in a Docker image, which is what I've been using for years:

https://eff-certbot.readthedocs.io/en/latest/install.html#al...

crotchfire• 2 years ago

This is why I use .onion urls instead of WebPKI.

memset• 2 years ago

Shameless plug: I've built a tool that automatically generates certs and uploads to destinations. https://github.com/poundifdef/certmaster

It uses Lego under the hood to issue certs, and then has custom connectors to upload to destinations. Right now those are email (indeed, more humans handling certificates), sftp, and hetzner load balancers. I'm working on adding the ability for it to automatically renew and re-upload when certs are 30 days from expiration.

throw0101d5• 2 years ago

A 'competitor' to this would be GetSSL which is a pure-shell ACME client (plus OpenSSL and cURL) and can be executed on one host, but send verification tokens to remote systems (where you may not have cron access):

Get certificates for remote servers - The tokens used to provide validation of domain ownership, and the certificates themselves can be automatically copied to remote servers (via ssh, sftp or ftp for tokens). The script doesn't need to run on the server itself. This can be useful if you don't have access to run such scripts on the server itself, as it's a shared server for example.

* https://github.com/srvrco/getssl

at_a_remove• 2 years ago

I get why it is important, but certs can be a tremendous pain to deal with when you're in a niche industry and reliant on the only vendor in town, and the consequences of a broken cert were minimal.

0xbadcafebee2• 2 years ago

The whole cert regime is wrongheaded. Issuing certs is vulnerable to exploits, the methods are clunky and problematic, certs expire so quickly that stuff is breaking more often than security issues could ever come up, there's no consideration to private networks, we still don't have content integrity without encryption, content proxy caching is dead, CAs can do what they want (there's too many CAs, cert logs aren't required to be validated in real time, you can't force a CA not to do something shady under a government gag order, etc), and there's a fundamental conflict between the owner of a domain and the issuer of a cert and the owner of a cert and the owner of some IP space or DNS server records.

But because it's this giant behemoth involving multiple industries, technological components, and vested interests, it will never change, save to implement bandaid after bandaid after hack after kludge. Literally the only thing that changes it is when someone with enough power (a market-dominant browser) forces it and everyone has to comply, or face creating a total shit-show due to lack of conformance. And we're all hopelessly dependent on it so it's not like we can do anything about it. The security of the web is a Kafkaesque nightmare.

XorNot• 2 years ago

Here's the thing though: the alternatives are also god-awful, and in part it is because the people who make them wind up being very stupid about their choices - i.e. if you want a distributed trust environment fine, but your first, primary and most important use-cases are still going to be government, and corporate, and specifically financial.

If your "libre" plan does not involve a way to bring those entities onboard and direct priority to servicing their needs, then no one is going to use it because we all actually live in the real world where those are the things we need security in communicating with.

gur48• 2 years ago

More revocations != more customers losing keys.

It may just as well mean more capable audits to detect lost/compromised keys.

The article also says nothing of certificate validity periods, which are critical to their lifecycle and analysis.

The conclusion that humans degrade information security is true but ultimately removed from the reality of operating a business.

throw0101d5• 2 years ago

A reminder that if you an internal-only server where the typical http-01 verification connection method will not work, one can use dns-01 by using DNS aliasing/CNAME, especially if you cannot easily/dynamically update DNS records:

* https://dan.langille.org/2019/02/01/acme-domain-alias-mode/

* https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se...

So if you want a cert for www.internal.example.com, you will first have do a one-time change to have a _acme-challenge.www.internal… CNAME created to point to any other (sub-)domain where you can easily update things dynamically, e.g., www-internal.example-dnsapi.com.

When request the cert for "www.internal…", LE/ACME will look up the corresponding _acme-challenge record, and go to "_acme-challenge.www-internal.example-dnsapi.com. The nonce token will be there in the 'final' destination following the CNAME in a TXT record, which shows LE/ACME that you control the DNS chain.

To do the DNS updating, you can use a CLI/Python library like Lexicon, which supports dozens of APIs:

* https://github.com/AnalogJ/lexicon

ryandv• 2 years ago

One advantage of the DNS01 approach is that you can also order certificates for standby instances behind a load balancer, all fronted by the same domain name. Since the standbys are not serving traffic, you can't get an HTTP01 challenge/response through, but you can still do so with DNS01 and get a certificate delivered to that machine, so that it's always ready to serve TLS in the event of a failover.

JoshTriplett4• 2 years ago

tls-alpn-01 woks really well as well. You don't have to mess with DNS at all, or have an HTTP port open; you just listen on HTTPS.

throw0101d5• 2 years ago

Yeah, except for the the internal-only server part where the hostname may not even be accessible to the public Internet (certainly not as an A(AAA) record).

And given the number of internal applications many large/r organizations have, giving all those internal dev teams update access to the main corporate domain is not something I'd really feel comfortable doing.

But being able to give API access to (e.g.) foo.dnsauth.example.com is something I have done: main domain needs to be touched once for the CNAME, and the sub-domain isn't critical for day-to-day operations of a company.

diarrhea• 2 years ago

I own example.com and actually created NS records for internal.example.com, to manage that entire zone at an entirely different provider. I did that because the original provider had terrible API access for Caddy to use. The new one has better support, and a breach there does not pwn the root domain. I was happy with that setup, but wasn’t aware one could delegate to an entirely different domain using ACME primitives. Makes sense though, it’s like the NS approach but simpler, and allows using some throwaway domain. Thanks!

yourapostasy• 2 years ago

Can anyone with good experiences with a commercially-supported ACME stack recommend a vendor they’re happy with, where the stack was integrated into their own in-house certificate authority? I’m especially looking for vendors that have very good documentation and tracing introspection to let our own engineers who read the documentation rapidly troubleshoot the stack, and our own developers build robust integrations to the client to rotate the certificates.

tlyman27• 2 years ago

You should look at Venafi and their offerings. They have integrations that allow you to connect to inhouse CA for certificate issuing. You can also add your own integrations when provisioning certificates to machines or choose from some of the many existing integrations.

Disclosure: I work for Venafi

schoen• 2 years ago

I haven't used it (so I might be excluded from answering your question), but I think SmallStep may be relevant to you.

https://smallstep.com/docs/certificate-manager/acme/

At least they are doing a lot of "your own organizational PKI" tools and they care about ACME integration.

Why Certificate Lifecycle Automation Matters

Related Stories