Stop MitM on the first SSH connection, on any VPS or cloud provider

https://www.joachimschipper.nl/Stop%20MITM%20on%20the%20first%20SSH%20connection,%20on%20any%20VPS%20or%20cloud%20provider.html

by JoachimSchipper • 3 months ago

137 77 3 months ago

joachimschipper.nl

londons_explore2• 2 months ago

A big class of attacker is nation state attackers who do not want to risk discovery.

A big way to deter them is to keep remote log files which, if analyzed, will reveal any attack.

For example, if both ssh-client and ssh-server kept a fingerprint of the session key in some append-only logfile, then a later administrator could compare the logfiles to know if an MITM happened.

Suddenly, nation state attackers won't be interested in MITM-ing at all.

Unfortunately it appears openssh doesn't even have an option to create such a logfile!! Why not??

hnlmorg2• 2 months ago

Couldn’t the MITM ssh server just forward the client’s fingerprint to the legitimate server?

If so, the legitimate server wouldn’t have anything in their logs that would help detect such an attack.

OpenSSH does log other telemetry though.

flumpcakes2• 2 months ago

Couldn’t the MITM ssh server just forward the client’s fingerprint to the legitimate server?

Fingerprints are derived from the certificates/private keys. Unless I don't understand some basic crypto, or SSH works in some obtuse way, I do not think it would be possible for the MITM attacker to present the server with the true client's fingerprint unless they also had had the client's private key.

cesarb3• 2 months ago

Couldn’t the MITM ssh server just forward the client’s fingerprint to the legitimate server?

The client sends not only the public key, but also a signature, and that signature depends on the output from the key exchange, so it's "bound" to the shared keys negotiated between the client and the server. If the MITM server does separate key exchanges with the client (pretending to be the real server) and the server (pretending to be the real client), the signature won't match; if it forwards the key exchange between the real client and the real server, it won't be able to decrypt the packets.

That's the best thing about SSH public key authentication (and HTTPS client certificates): even when MITM can impersonate the server to the client (because the client didn't verify the host key), it can't impersonate the client to the real server.

daveguy2• 2 months ago

Because log processing is handled in the kernel/root/system? Is this a trick question?

See also: rsyslogd

dboreham2• 2 months ago

I think the idea is the attacker didn't compromise both the local machine and the remote log sink machine. If you want to get really fancy the techniques used in cert revocation logs/blockchains could be used.

iberator• 2 months ago

how would you create REAL write only logs?

syslog > /dev/lpt0 printer?

bitfilped• 2 months ago

Forwarding to WORM tape is one way. Of course that still involves network and likely a buffer somewhere.

heinrich5991• 2 months ago

Example of a nation state attacker that risked discovery and got discovered: https://notes.valdikss.org.ru/jabber.ru-mitm/.

codethief• 2 months ago

Oof. The answers by Hetzner and Linode read as if they were straight out of a dystopian novel.

ranger_danger• 2 months ago

How do you know this was a nation state attacker?

skydhash• 2 months ago

The technique appears to be new: I haven't found a proper write-up of this, nor of any other provider-independent solution (but I'd welcome a correction).

To be frank, anyone that serious about security would probably log in via console, generate and retrieve the host key that way. And then any client would have strict verification enabled.

It's kinda the 101 of communication using public keys cryptography. You have to get hold of the public key in a secure manner first (direct contact or attestation by a third party).

Section 3.1 in Bruce Scheiner's Applied Cryptography discuss how to automatically solves MITM. But that's only important for M:N communications (TSL). For 1:1 communications where you can have secure exchange before hand, no need to go that far.

dboreham2• 2 months ago

Attacker might have MITMed the console.

Arrowmaster• 2 months ago

If they MITMed the console then they have probably MITMed the entire deployment process and now have the one time use key already.

crypt0r84• 2 months ago

provision the hosts with an SSH CA, use the CA as a trust root in openssh. they are various version out there from the big players.

INTPenis• 2 months ago

The author essentially bootstraps their servers with a known trusted host key, so that first connection is recognized, instead of having to trust a new and recently generated host key when you first connect.

It's a neat little trick if you're often deploying VPS in shared cloud environments.

flumpcakes2• 2 months ago

This is something I have struggled with:

How to deploy secrets during bootstrap to a new virtual machine running in the Cloud that does not leave a trace in the infrastructure. And in a way that I can completely automate the deployment.

One answer is providing the secrets in cloudinit - but this leaves a trail on the host/provider's infrastructure, I do not know if those configs I paste into the portal then get saved off somewhere.

The other option (more secure) is having the keys/secrets generated on the host itself at first boot. But then this is difficult to automate as I would need to scrap them (even just the public parts) in a secure way. One option would be to have the public keys printed to the terminal/VNC - but this is much more trouble than it is worth to automate.

I'm not sure on a good solution. This is taking quite an adversarial security model though, assuming the host/provider is not completely trustworthy. Of course not owning the hardware means that the host/provider could be performing other attacks without my knowledge (copying memory, etc.)

IgorPartola2• 2 months ago

1. Use cloudinit but give it a one time password to download the secrets on first boot.

2. Use certificates and your own CA.

3. Use the virtual serial console for first login.

4. Use cloudinit to add a custom software repo, then use that to install a custom package that does the initial work.

unsnap_biceps• 2 months ago

I've been meaning to investigate Nitro Enclaves[1] for exactly this but haven't made the time yet. Have you looked into them and found them lacking?

[1] https://aws.amazon.com/ec2/nitro/nitro-enclaves/

anygivnthursday• 2 months ago

You can bootstrap from your custom ISO with some embedded starter key, upload ISO, loads into RAM and opens SSH, connect and run a playbook to encrypt the root drive where you deploy your OS with your SSH key. If you went with encrypted root, you might need to enter pass via console, or dropbear server in a pre-boot env you connect to via SSH to enter the key if you dont trust the console, or setup some custom network unlock mechanism, etc. But once unlocked your provider can still dump your keys from memory. There are also things like AMD SEV-SNP for some more confidential use cases.

asalahli• 2 months ago

It baffles me why VPS providers don't display the fingerprints on their dashboards or expose them through the API.

dspillett• 2 months ago

A few do. Though a more common way of securing the loop is to take the desired user public key during the build process.

Assuming you trust that the host control panel (or API server) hasn't been hacked, which you are assuming anyway if you trust a host fingerprint given to you that way, that should be as secure. For a small bit of extra assurance, to protect from an extra very unlikely attack, generate a new key pair just for this VM's creation so that you know you aren't connecting to some other VM that happens to have a known public key of yours and you've been redirected to by DNS poisoning.

comex2• 2 months ago

AFAIK this is not secure. The SSH protocol doesn’t require the server to know the client’s public key for the connection to succeed. Instead, the client actively sends its public key to the server while authenticating, and the server just decides whether to allow a connection or not. OpenSSH will typically match the client’s key against a fixed list of keys, but nothing stops a rogue server from allowing connections from any client key. In fact, OpenSSH itself can be configured to do this using AuthorizedKeysCommand.

zuzululu2• 2 months ago

but how real is this threat? you buy a box from hetzner or some VPS provider and its rooted when you try to ssh ?

projektfu3• 2 months ago

You get a box and someone is sitting on its IP address (in the middle), proxying to the real one, so everything is getting logged. Other comments say that this mitm stops working when you use public key authentication.

cesarb3• 2 months ago

Other comments say that this mitm stops working when you use public key authentication.

It doesn't completely stop working; a MITM can still pretend to be the server, it just can't authenticate to the real server on your behalf. You could be doing all your work in a fake server controlled by the attacker, while the real server sits there untouched.

tabwidth• 2 months ago

The box itself is probably fine. It's the path between you and it. In shared infrastructure one compromised hop somewhere upstream is enough and now you're SSHing into the wrong thing without realizing it.

tptacek2• 2 months ago

If you don't assume the network between you and Hetzner is compromised, you might as well just use rsh instead of ssh.

zuzululu2• 2 months ago

Hope your home internet isnt compromised if it is thats a whole other problem

PunchyHamster• 2 months ago

Seems like way easier way would be sshing for the first time and just typing `sudo reboot`. If VM reboots, it is yours

Or cat-ing some secrets that would be on target machine but not attacker

kardos• 2 months ago

A mitm does not redirect your ssh login to their machine

IgorPartola2• 2 months ago

There was ages ago a project called monkeysphere that let you sign the host ssh key with your gpg key and verify it automatically. The downside was that it was very slow.

lvlabguy2• 2 months ago

MitM is not possible if one uses public key authentication.

gruez2• 2 months ago

I was about to downvote this for being obviously false, but after some research this does appear to be true, because ssh uses some channel binding mechanism to prevent your public key authentication from being replayed/reused by the "man" in the middle.

ekr____2• 2 months ago

This is one of those situations where it's necessary to be very precise about the security properties.

Specifically, if you bind authentication to the connection, then an attacker who impersonates the server (in this case because it's the first connection, but in other settings because they have a fake certificate), then client authentication is not portable to another connection, so the attacker can't mount a classic MITM attack. However -- and this is a big however -- that doesn't mean that there aren't serious security problems. For example:

* If you use SSH to copy a secret such as an API key to the server, then the attacker still knows the API key.

* If you download some file (e.g., a script) from the server and then trust it, the attacker can use that to provide a malicious script.

lvlabguy2• 2 months ago

Basically, the client signs the shared key obtained through Diffie-Hellman key exchange, which then gets verified by the server. This ensures that the client and the server have the same shared key, hence no man-in-the-middle.

riffic• 2 months ago

zero touch prod might be something folks should keep as a north star - SSH should be break glass and a last resort if possible.

https://www.usenix.org/conference/srecon19emea/presentation/...

athrowaway3z• 2 months ago

So this is a tangent on a thought i had after reading the title, but it might be a cool idea that I'll not have the time to do anything with so feel free to use it:

Human checkable fingerprints for pubkeys/hashes dont really work. None of the schemes i've seen hold up under somebody willing to spend compute to get a near-enough collision to fool most people most of the time.

But we can take those random bits and transform them and feed them into a seeded image generation LLM, and then have a person remember/compare the deterministic output image.

You might even make the case its the perfect machine to create memorable-2-human image artifact from random data.

wiml• 2 months ago

There are a bunch of methods for transforming a hash into something that's easier to compare. You've probably already seen the RandomArt thing that openssh uses for comparing host keys on first use. Some apps produce a sequence of emoji for fingerprint comparison. It's a small but fertile little research niche.

I can't offhand think of anything that an LLM image generator would do to improve the process; it'd be an interesting research task. You'd need a way to transform the 256-bit hash into LLM input in a way that would maximize the perceptual difference in generated images. The problem is that it's absolutely critical that two different implementations work the same, which means the spec would need to specify the exact set of model weights to use.

washingupliquid3• 2 months ago

I'm supposed to believe MitM with the same exact keypair is somehow possible? Private keys are never exchanged. Did everybody forget how crypto works?

Yes you implicitly trust the public key on first login.... then just... immediately compare it with what's on your box?

Might as well seal your doors with duct tape to prevent ghosts from entering your home because this is equally effective.

leni536• 2 months ago

How do you compare? What trusted channel do you use to retrieve the real public key?

washingupliquid3• 2 months ago

Public keys go over untrusted channels. That's why they're public.

I'm not confident you understand how crypto works.

You do realize the entire threat model here is a house of cards perched atop someone else's software hosted on someone else's hardware all of which you implicitly trust and discard in favor of some unlikely cloak and dagger interception scheme.

tptacek2• 2 months ago

The author of this post understands that private keys are never exchanged. Read it more carefully.

theteapot• 2 months ago

The technique appears to be new: I haven't found a proper write-up of this, nor of any other provider-independent solution.

Maybe I'm missing something but SSH already has a built-in solution for this, key-certs. Just sign the server key with a private CA key you trust.

Borealid4• 2 months ago

SSH has *ANOTHER* built-in solution, in the form of the SSHFP DNS record.

If the DNS record for the host has an SSHFP (SSH FingerPrint) record, SSH will compare it to the retrieved public key(s) and refuse the connection if there is a mismatch. It can be configured to require DNSSec for this, or to only reject if it gets a secure rejection (to prevent DoS).

It works perfectly, has no notable down sides (just add a DNS record when you generate the host's SSH key), and has been around for many years.

beagle3• 2 months ago

It is very insecure unless you use dnssec, isn’t it?

Just means an attacker also needs to mitm DNS if you MITM the host. Not trivial, but depending on setup might not be harder.

cromka• 2 months ago

Is there something like this for Wireguard?

xorcist• 2 months ago

The article describes a method of securely seeding private key data to newly provisioned servers. It's a bit of bootstrapping problem, and not something a PKI can solve.

You usually need some form of trusted communication with a new server until you can give it its real identity, in the form of host names and cryptographic keys. In an enterprise environment this can usually be done with some sort of isolated management or provisioning vlan. In a cloud environment, I've seen all sorts of more or less hacky solutions but since it depends a lot on specific details of your networks and APIs, bespoke solutions are fine.

TacticalCoder• 2 months ago

inject a temporary SSH host (private) key via cloud-init...

Reading the comments here I'm tempted to believe that if cloud-init is available and if we consider Heztner (and OVH etc.) provides a secure access to cloud-init (i.e. the box running cloud-init is really the box you think it is), then there are many different ways to solve this problem.

jmholla• 2 months ago

I've been thinking of setting up a simple server that publishes the public keys at a known endpoints. You point an A record for one of your subdomains at the machine and it can provision a TLS cert. Then you can be reasonably confident your connection hasn't been MitM (assuming your trust your cert provider) when you query for those public keys.

The one presented in the article has fewer moving parts though. I'm also curious what ideas are bouncing around your head if you're willing to share.

kayson2• 2 months ago

I've been fighting against this chicken and egg problem in my homelab. I have a step CA SSH CA set up along with automated deployment, bootstrapping, and cert renewals with an ansible playbook. It sets StrictHostKeyChecking yes in the ssh_config for my domain so I'm happily protected against a very unlikely attacker that has a foothold in my network. Theoretically, it means I should never have to worry about host keys again.

Unfortunately, ansible does everything over.... SSH. So if I spin up a new VM or host, I have to manually trust the certificate for the first connection, which is the whole point of this article. I always have console access so I can log in and check the pub key.

There are various ways around this, including the authors suggestion with cloud init. None are very helpful for new physical hosts, though. I'm leaning towards a feature step CA supports that lets hosts authenticate themselves with an X509 cert, which you can easily get with ACME. It's so easy that you could even do it over console on a new physical host.

What I really wish is that there were something like the ACME TLS-ALPN challenge but for SSH servers. They can already present a self signed certificate, so it would just be a matter of connecting all the plumbing.

denkmoon• 2 months ago

Though the name is cloud-init, you can cloud-init physical hosts. Seems excessive though, how often are you adding new physical hosts to the homelab?

kayson2• 2 months ago

True. But it means you have to get it into the installer/image which just doesn't seem worth it for something that, as you point out, doesn't happen all that frequently.

SoftTalker• 2 months ago

So you inject a host ssh key pair at VM creation using cloud-init, then connect one time trusting that key and immediately replace the host keys in the VM. So there is still a brief race condition where the attacker could impersonate the VM if he had intercepted your cloud-init file, correct?

Also I don't see anywhere that the script reloads the sshd daemon, which AFAIK is necessary to get it to start using the new host keys and stop using the deleted initial host key.

burnt-resistor• 2 months ago

I guess they don't teach public key-enabled SPA secure port knocking anymore to defend ports of jumpboxes. And using jumpboxes and an internal vlan rather than putting control ports on the wild internet, right?

Stop MitM on the first SSH connection, on any VPS or cloud provider

Related Stories