Ask HN: What's your solution for SSL on internal servers?

https://news.ycombinator.com/item?id=30272101

by Tepix • 4 years ago

65 77 4 years ago

news.ycombinator.com

Lets encrypt has made SSL/TLS a no-brainer for websites on the public internet. Behind home routers, it's still a tedious topic. What's your (non-enterprise) solution?

latch• 4 years ago

Personally I avoid it where I can and use wireguard for all internal traffic. Hopefully someone more knowledgeable here can tell me if it's a good or bad idea.

LilBytes• 4 years ago

Same here, Tailscale and their DNS LetsEncrypt service does all the HTTPS management for me. Works just fine really.

dindresto• 4 years ago

I'm using subdomains on a domain I own and request Let's Encrypt certificates with the DNS challenge.

viovanov3• 4 years ago

Then you're routing internal traffic through a public IP? Or do they support wildcard certs?

Macha4• 4 years ago

I have local DNS setup to resolve my personal domains to hosts on my home network. They do support wildcard certs, _only_ if you use some form of DNS challenge.

bryanlarsen2• 4 years ago

SSL certificates contain the name, not the IP. So the IP address can be anything, including internal ones.

tyingq3• 4 years ago

Then you're routing internal traffic through a public IP?

No, not typically. There's various methods to do the LetsEncrypt challenge/verification that don't require internet connecting the internal host you're generating the certificate for.

The downsides are:

- You can generate a wildcart cert for *.internal.yourdomain.tld. But then, it's a pretty big master key if you lose control of it.

- You can generate a cert-per-server but it exposes your hostnames (at least) in certificate transparency logs, which gives outsiders some view into how big your internal network is, perhaps some detail on what it's like via hostnames, etc. This is worse if you also expose the internal DNS records externally, then everyone sees those records as well, exposing more internal info. You could mitigate these things somewhat with various strategies around hostnames, DNS setup, etc.

fulafel• 4 years ago

Other replies already explained how this is orthoginal to IP addressing, but also there's not many virtuous virtuous things and many downsides about using ambiguous addresses your server to server communications. Also invariably you'll eventually end up networking them in a new way you didn't originally plan. It ends up being bad for security because it breeds unneeded complexity and makes your system harder to understand.

Tsarbomb• 4 years ago

Seconding this approach.

Macha4• 4 years ago

Yeah, this is the way to go. If your DNS provider doesn't let you setup limited scope access tokens, then you can use something like acme-dns with delegated dns challenges, as long as you have a single external server.

perbu• 4 years ago

Just beware that the records of the TLS certs are public. So you'll leak a some hostnames. This might or might not be ok.

tmottabr• 4 years ago

not if you use wildcard domain in the cert. LE already support wildcard certs

efitz• 4 years ago

SPIFFE: https://spiffe.io/

tqwhite2• 4 years ago

For a long time I was all fussy about having to create a security exception for self-signed certificates. One day I realized I was acting insane, as if there was some glorious principle involved. There isn't.

I trust my own (or coworkers) certificates. It's a dev site for heavens sake.

Ever since, ssh-keygen all the way.

savolai2• 4 years ago

It appears to me the issue is browser warning dialogs that imply it is always very dangerous. There should be either more context explained in those dialogs or a recognition of/mode for sites that are supposed to be self signed.

dboreham• 4 years ago

There is a long history to this. The original browser warnings were along the lines of your suggestion. Then it was discovered that regular users just clicked through the warning when an attacker MITM'ed their bank. There followed decades of making the warning ever more scary sounding and ever more difficult to bypass.

ThermalCube• 4 years ago

(Untested idea) I'd suggest creating your own Root CA with an expire-date far in the future. Install it's public key as a trusted certificate and all derived certificates should not prompt any issues anymore

tyingq3• 4 years ago

It can be tricky to get self-signed certificates put into all the various places where they need to be. OS level certificate stores, browsers, mobile devices, curl, python/requests, VPN clients, etc. There's always some weird exception case.

ninkendo3• 4 years ago

Yeah, I think OP probably mostly does web development through the browser, where there’s just one trust store to worry about.

As soon as you need some automation involving backend services or even just curl and some scripts, this gets tougher. (And please please don’t use `curl -k` to disable checks… if such scripts accidentally make their way to production, you may as well not use TLS at all.)

jhugo• 4 years ago

I have a private CA (managed with cert-manager) and trust it on my systems, and issue certs for internal services from that.

isthisfree• 4 years ago

If you can automate DNS, create a wildcard LE cert and have a cronjob to maintain it around your different places from the one place you issue it. That is what I do.

Before that I just bought one wildcard cert and used that. Can be bought at less than 50 bucks and then no hassle.

If I could not automate DNS and I don't have 50 bucks per year for it, I would create a small CA myself, trust it in my browsers and issue certificates from that.

Cockbrand• 4 years ago

I have only one internal server which is coincidentally also accessible from the public Internet. I use NAT hairpinning for the external interface of my router and forward all packages on ports 22 and 443 to my server, so its TLS certificate is also valid from inside my LAN.

bjt2n3904• 4 years ago

EasyRSA. Install the CA on my devices, distribute the SSL certs to the devices I care about. (Mainly NAS, OctoPi, PiHole). The me PiHole serves hostnames too, and DHCP.

justsomehnguy• 4 years ago

I have my own 2T PKI.

Pro: you would know how some apps crash (and sometimes burn) when PKI breaks. Also you would know what every other distro guy thinks he is smarter than everyone and do the PKI his own way.

Cons: see Pro.

NB: Windows PKI in the ADDS environment (at terms of distributing the RCA/ICA trust) is a walk in the park compared to everything else.

NB2: Java keychain is PITA.

Behind home routers, it's still a tedious topic

Buy a domain, park it at Cloudflare/Ghandi/whatever ACME supported provider, use DNS-01, push or pull certs to the local network.

There is no problem with the process, only with laziness and automation.

yosamino• 4 years ago

I have a private CA for all internal hosts. It's a bit of a pain to run, but it's neccessary because "internal" includes VMs that run colocated servers and many services to mutual authentication.

It's hyper annoying that it's barely to not possible to include our own CA into mobile Browsers so that they can use internal websites.

The alternative would be to depend on LE where neccessary, but this introduces an external dependency that I would rather avoid.

gjvnq• 4 years ago

Why not have your applications accessible through two URLs with yor CA and another with LetsEncrypt.

Also, it may be possible to sign an SSL certificate with two authorities.

xoa• 4 years ago

It's hyper annoying that it's barely to not possible to include our own CA into mobile Browsers so that they can use internal websites.

You can? Any MDM should be able to deploy internal CAs, and at least on iPhones the free Apple Configurator 2 software will let you make profiles you can then easily deploy a bunch of ways. Doesn't scale easily but it's free and easy for small numbers. Also very useful period even for an individual or family to bootstrap devices and make changes if you've got significant numbers of email accounts for example. I'm sure Android has some equivalent.

giaour• 4 years ago

Private CAs are the enterprise solution, but that can get expensive or difficult to manage for a home setup.

You could get a cert for a wildcard subdomain and then use whatever private subdomains you want on your home network

guilhas• 4 years ago

Create a self-signed certificate on the server

Install it as trusted on each client

yabones• 4 years ago

I have one main Nginx server that all other services are behind, regardless if they're internal or external. This box is where NAT port forwards 80,443 to.

I also use only subdomains of the domains I own, even for internal stuff. This means I also run a small bind9 DNS server with minimal zones to direct traffic to the proxy inside the network, and most of the records just don't exist outside, ie. they return NXDOMAIN from my public DNS provider.

On the nginx box, I have a snippet like this:

    deny all;
    allow 10.x.0.0/16;

Then, when I configure something as 'internal only' I just add this line to its config file:

    include /etc/nginx/private.conf

This means that I can decouple the certificate status from the internal/external status of the site. All sites get valid certs, and most of them get 401's from outside the network.

In reality, I manage the nginx config with ansible templates, so what I really do is set a boolean "public" flag to "true" on sites I want accessible outside, everything else is private by default.

api• 4 years ago

At ZeroTier we are working on a solution for this that will implement ACME. Not ready for release quite yet but getting close. Could be used on ZeroTier networks but doesn't have to be.

AnonHP• 4 years ago

Please post a submission on HN whenever this is released.

phsm• 4 years ago

You can use https://github.com/joohoi/acme-dns to issue letsencrypt certificates to your internal hosts using DNS validation.

All it takes is to setup an ACME-DNS server somewhere (or just use author's public ACME-DNS server if you don't care much), and create one CNAME record in your DNS.

itgoon• 4 years ago

Hashicorp Vault can generate certs via a web form or REST call. Easy to set up and maintain, and it's free.

oulipo• 4 years ago

Can you explain your use-case for this?

mikedelago• 4 years ago

At a previous job, I used Smallstep Certificates[0] as a hosted CA, though for certificates to communicate with our Kafka clusters. It worked pretty well, and was relatively easy to set up.

[0] https://smallstep.com/docs/step-ca

res0nat0r• 4 years ago

I've not used this directly, but it may be useful: https://github.com/FiloSottile/mkcert

    mkcert is a simple tool for making locally-trusted development certificates. It requires no configuration.

    Using certificates from real certificate authorities (CAs) for development can be dangerous or impossible (for hosts like example.test, localhost or 127.0.0.1), but self-signed certificates cause trust errors. Managing your own CA is the best solution, but usually involves arcane commands, specialized knowledge and manual steps.

    mkcert automatically creates and installs a local CA in the system root store, and generates locally-trusted certificates. mkcert does not automatically configure servers to use the certificates, though, that's up to you.

wyuenho• 4 years ago

Depends on how your home network is setup and how you deploy your services. If you have a public domain for you home IP, and it's the usual docker bridge network setup with only a couple of containers, using Traefik or Caddy as a reverse proxy will suffice. They'll automatically provision TLS certs for your services with very little to no effort at all. If it's something more complicated than that, such as needing a separate IP and mdns host name per container running on a vlan, or some multicloud kubernetes setup, you pretty much have to setup your own CA. In that case, look into mkcert and/or step-ca.

daper• 4 years ago

If you have your own domain, then move it to one of the listed DNS providers and use DNS challenge with ACME:

We are using using certbot + cloudflare whis way. There is no HTTP request, certbot makes a temporary DNS record using the Cloudflare API to satisfy the challenge so you can run the script anywhere. Then copy the cert to the device that needs it.

DNS providers supported bycertbot:

https://community.letsencrypt.org/t/dns-providers-who-easily...

chenxiaolong• 4 years ago

None of my self-hosted things are internet facing, so I use the DNS-01 challenge type. For internal DNS, I run the powerdns authoritative server. Each ACME client uses RFC2136 (TSIG) to update the _acme-challenge TXT record. I have a pdns lua policy script set up so that the only record that's allowed to be updated is the TXT record matching the TSIG name.

To allow Let's Encrypt to hit the DNS server, I run a public-facing dnsdist load balancer. It forwards the relevant TXT, CAA, DNSKEY, and NS queries to pdns and silently drops all other queries not required for ACME challenges.

I'd prefer that the internal hostnames wouldn't be leaked in the certificate transparency logs, but given that no services are exposed to the internet, it doesn't bother me enough to look for alternatives (eg. wildcard certs).

AtlasBarfed• 4 years ago

This all comes down to two things:

- keytool and openssh absolutely suck from a usability standpoint

- testing the pipeline of generation of keys/files/certs/stores and importing/generating/signing etc is difficult

- error messages, if you get them, are completely unhelpful, and often the errors are superficially not even ssl/security related.

Every time I do SSL, it is a 1-4 day job, and that's with StackOverflow saving my ass on translating "why this weird error means this failure".

Between the above two issues, SSL on every platform, application, database, operating system (or version of OS) has different errors.

If you have a non-mainstream language, I have NO IDEA how you would get SSL up. Python, JVM, javascript, C/C++ there's a lot of eyeballs on this.

bravetraveler• 4 years ago

I still pay for wildcard certs on a few domains; it's crazy - I know, I just haven't bothered to jump over to LE and add it to the list of things to mind.

For some services I'll put them behind those domains and simply use the appropriate certificate.

Generally though if this is strictly internal, my domain can issue internally trusted certificates.

madduci• 4 years ago

A private PKI using Ejbca Community Edition from Primekey. Pretty solid.

Another one is step-ca

captn3m0• 4 years ago

Wildcard certificate using Traefik on a secondary domain generated using DNS challenge. Since this gives Traefik too much permissions to my DNS (CloudFlare now has better RBAC, but I haven't switched to it) - I use a secondary domain for my homeserver.

For the few services where I want to use my primary domain, I run dehydrated once in a while from my local setup (which uses my password manager on a script). These services do end up exposed publicly on the CT Logs, but I'm okay with that.

I proxy these services over a DO VPN, but resolve these internally to private IPs using NextDNS. There's also a internal domain for every service (service.in.example.com resolves to internal IP and service.example.com resolves to public IP, and service.example.com resolves to the private IP within my home network)

sdevonoes• 4 years ago

Good question. I'm also interested. My use case would be:

- a have an nginx server load balancing traffic between N web servers that talk to one DB. Everything is inside a VPC (I'm using digital ocean), and only nginx is public to the internet

One approach I have read is that I could terminate SSL at the nginx level, and handle normal http between my web servers. Question would be: how secure is that? Can I (should I) trust that everything within my VPC is only accessible to me? Is terminating SSL good enough when handling, let's say, account creations and payments via Stripe?

9wzYQbTYsAIc• 4 years ago

I use an internal subdomain wildcard certificate (e.g., *.internal.example.com) and traditional methods of configuring ssl for internal services / sites on the internal servers.

For cases where the service or site doesn’t natively support ssl, I run a local reverse proxy with the above certificate.

rad_gruchalski• 4 years ago

Running an internal CA is really easy in 2022: https://gruchalski.com/posts/2020-09-09-multi-tenant-vault-p....

throw0101a• 4 years ago

DNS alias mode:

* https://dan.langille.org/2019/02/01/acme-domain-alias-mode/

* https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo...

* https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se...

You want the name "internal.example.com". In your external DNS you create a CNAME from "_acme-challenge.internal.example.com" and point it to (e.g.) "internal.example.net" or "internal.dns-auth.example.com"

When you request the certificate you specify the "dns-01" method. The issuer (e.g., LE) will go to the the external DNS server for the look up, see that it is a CNAME and then follow the CNAME/alias, and do the verification at the final hostname.

So your ACME client has to do a DNS (TXT) record update, which can often be done via various APIs, e.g.:

* https://github.com/AnalogJ/lexicon

You can even run your own DNS server locally (in a DMZ?) if your DNS provider does not have an convenient API. There are servers written for this use case:

* https://github.com/joohoi/acme-dns

* https://github.com/joohoi/acme-dns-certbot-joohoi

* https://github.com/pawitp/acme-dns-server

dlangille• 4 years ago

I approve this post. /s

ganondork• 4 years ago

I do a lot of the same things as others with a custom domain at home with a NAT 80/443 forward to nginx, but I use https://nginxproxymanager.com/ as it gives a dead-simple hostname proxy to forward traffic to internal hosts, and will request/renew Let's Encrypt certs for them automatically.

acsigen• 4 years ago

For a relatively small company I use PiHole Unbound with LE certificates and I'm routing the private subdomains to internal IPs.

pcunite• 4 years ago

I became my own CA. Install certs in the system/browser and issue keys that are 10 years out. Make everything with the CertManEX tool.

CaptainJustin• 4 years ago

I asked a similar question previously: https://news.ycombinator.com/item?id=29995812

stephenr• 4 years ago

I'd already written a small (server focussed) tool to make use of certbot (or any ACME client really) certs a bit more automated (getting the right combination of certs/key in a file, converting to alternative formats, fetching the OCSP data, syncing across machines, restarting services after they're updated etc);

Last year I added in a 'create' mode where it sets up a self signed root CA, and issues certs using that. The other logic (convert, sync, combine) is obviously all the same still.

galenguyer• 4 years ago

I wrote this tool[1] to help me create a CA, generate certificates, and automatically renew them

https://github.com/galenguyer/hancock

Ask HN: What's your solution for SSL on internal servers?

Related Stories