Duplicate SSH Keys Everywhere

https://blog.shodan.io/duplicate-ssh-keys-everywhere/

by achillean • 11 years ago

310 84 11 years ago

blog.shodan.io

astrocat• 11 years ago

Not my domain of expertise; can someone explain why this is problematic?

bbrazil• 11 years ago

If you can read the private key from just one of these machines then you can impersonate any of them.

4mnt• 11 years ago

That is: MITM SSH connections to these devices without getting any warning. Of course you first have to get in a position to MITM the person who connects to these devices.

superuser2• 11 years ago

On the same topic, what are the other pitfalls of re-using OS images?

I can think of:

- Hostname conflicts

- SSH key duplication

- Driver issues if the hardware is not the same

Are there more?

This seems like a compelling argument for bootstrapping new instances with configuration management, rather than trying to re-use OS images.

achillean• 11 years ago

Speaking of hostname conflicts: an ISP in Vietnam has configured all their devices to have a reverse DNS name of "localhost"

https://www.shodan.io/search?query=hostname%3Alocalhost

JonRB• 11 years ago

I've experienced this too - I seem to get a few people joining my gameserver from that ISP and I notice the DNS appeared as 'localhost'. Not ideal and I imagine it could break certain things. I always assumed it was a misconfigured internet cafe or something. 'Nice' to know it's a wider issue

Xylakant• 11 years ago

MAC Address duplication on virtual interfaces for VMs started from a template.

warrenm3• 11 years ago

Never seen this in years of using VMs

shitloadofbooks• 11 years ago

No (mainstream) Hypervisor does this and you'd run into networking problems pretty quickly if any of those VMs share a network.

icebraining3• 11 years ago

An alternative to typical configuration management (puppet, ansible, etc) is making your own OS installer. I've adapted the Ubuntu installer to turn a blank machine into a running instance of our app in just a couple of days. The client just has to pop-in the CD or flash drive, answer a couple of questions, and go grab a cup of tea while it installs.

warrenm3• 11 years ago

in just a couple of days

Days?! That is a crazy-long deploy time.

daurnimator• 11 years ago

Even things like /etc/machine-id

emmelaich• 11 years ago

MAC addresses. HP-UX saves them in its startup configs for some reason. A place I worked at used a sample build for its 'gold' build. All builds had the same MAC. Hilarity ensued.

elij• 11 years ago

shared inode number

feld2• 11 years ago

This also applies to those of you building "template" VMs in VMWare or Xen -- before you save that template please delete the SSH keys in /etc/sshd/ so they are generated on first boot of the deployed template!

korethr• 11 years ago

It would not surprise me if there are other bits of unique info on the template VM that need to be cleared before freezing the image, e.g. hostnames.

This is a solved problem in Windows. The sysprep tool wipes out the hostname, security keys, user accounts and a bunch of other things one can configure, so the image initializes properly the first time it's booted.

I am curious if there are similar tools for Linux & Unix that the image builders simply failed to use here, or if this points to an opportunity to build a useful tool.

mintplant• 11 years ago

There's virt-sysprep [1] and sys-unconfig [2].

[1] http://libguestfs.org/virt-sysprep.1.html

[2] http://linux.die.net/man/8/sys-unconfig

verbatim• 11 years ago

virt-sysprep from libguestfs does this.

http://libguestfs.org/virt-sysprep.1.html

pliu• 11 years ago

cloud-init

https://cloudinit.readthedocs.org/en/latest/

warrenm3• 11 years ago

in VMware, you use Customization Specifications to do the equivalent to sysprep for Linux (and sysprep on Windows VMs)

13• 11 years ago

generated on first boot of the deployed template!

I've always been uncomfortable about that. How much boot time entropy can a VM have?

staunch• 11 years ago

http://blog.dustinkirkland.com/2014/02/random-seeds-in-ubunt...

feld2• 11 years ago

Depends on your OS :-)

e12e4• 11 years ago

With standard ssh-keys, you now have a new problem: how do you know the key of your new vm? Trust-on-first-use is one option -- but a better option would probably be to move to ssh server/user certificates, and generate and sign a new server cert outside the vm, as part of finalizing/readying it for deployment. Might not work well with elastic generation of vms based on load (not sure how well ssh deals with mulitple CAs -- one might imagine a structure with an ultimately trusted, possibly long-lived, off-line cert on the top, and the various sub-CAs organized based on security domains (eg: elastic vm, user auth etc)).

Still doesn't exist any sane, open automation for that, as far as I know. Maybe we should be happy people use keys at all, never mind if the trust/validation is flawed... (not to mention keys never expire, and can only be revoked via a black-list).

jamiesonbecker6• 11 years ago

I've been thinking about adding server key synchronization (optional, of course!) across a Userify server group (cluster) but it seems even riskier than trust-on-first-use. What are your thoughts? Userify is definitely getting expiring keys soon (tracking when the key was last updated).

joelthelion• 11 years ago

I believe Vagrant takes care of that for you.

chubot• 11 years ago

Related: if you don't have enough entropy when creating keys, your public keys might not be identical, but they can share primes. Sharing primes allows this cool factoring attack by DJB and others.

https://scholar.google.com/scholar?cluster=92037504584829650...

This paper explains how an attacker can efficiently factor 184 distinct RSA keys out of more than two million 1024-bit RSA keys downloaded from Taiwan's national "Citizen Digital Certificate" database.

These keys were generated by government-issued smart cards that have built-in hardware random-number generators and that are advertised as having passed FIPS 140-2 Level 2 certification. These 184 keys include 103 keys that share primes and that are efficiently factored by a batch-GCD computation. This is the same type of computation that was used last year by two independent teams (USENIX Security 2012: Heninger, Durumeric, Wustrow, Halderman; Crypto 2012: Lenstra, Hughes, Augier, Bos, Kleinjung, Wachter) to factor tens of thousands of cryptographic keys on the Internet.

abrodersen• 11 years ago

This is why we can't have nice things. And by nice things, I mean identification of people via cryptography rather than rectangular pieces of plastic.

kordless2• 11 years ago

rather than rectangular pieces of plastic

Finally, the U.S. gets a mention.

lifeisstillgood• 11 years ago

Taiwan has a citizen digital certificate process - cool !

I mean that's a seriously big advance over anything I imagined a govt capable of. Estonia was the only other place I heard even close.

And is 183 out of 2 million so bad? Just sign the next round of certificates with this lot and move on.

pedrocr2• 11 years ago

Portugal also has a national ID card that's a smart card and includes a key. The last time I tried to use it in the justice ministry website it required accepting a self-signed HTTPS certificate. When I reported the issue I got a "oh, just accept it and it will be fine" response from a supposedly technical person.

So yeah, these things are getting deployed in some places but I'm not too hopeful it will end up being much more than a gimmick. At least I hope not because the next step on the minds of some of these "technical" people is electronic voting and that won't end well...

caf• 11 years ago

Yes, 183 out of 2 million is so bad, because apart from anything else it indicates that the output of the random number generators used had far, far less entropy than it should.

(How can you use a completely compromised private key as a root of trust for another certificate, anyway?)

kalleboo• 11 years ago

Sweden also has a digital certificate ID, but it was developed by the major banks instead of the government (with the government originally only providing a legal framework to allow digital signatures to be legally valid - BankID is now the primary method to file taxes and manage government services online) http://www.bankid.com/en/What-is-BankID/

zhte415• 11 years ago

I'm not sure why this is a good thing. An advance would be for not requiring any certification, and not having a single point of failure, despite the tech, as tech will fail, and be subverted before it does [reference: 5000 years of modern history]. A nation without 'citizen' certification system is, to me, cool.

kerkeslager• 11 years ago

I almost didn't read the paper because I thought it was only about the batch-GCD method: that's fairly easy to understand. But I'm glad I read the paper because it goes on to explain how using the shared factors allowed them to understand the failure of the PRNG and use that understanding to break keys that didn't share factors, using a more sophisticated attack. It's definitely worth reading that paper.

fvold• 11 years ago

Another implications of this is that if a security flaw is found in this re-used image, it's trvial to write a worm, as it can very easily seek out and find vulnerable hosts.

Ooops.

hannob• 11 years ago

This is not really news, this was already discovered by Nadia Heninger and others in a paper in 2012. From their faq about their research: "We found significant numbers of insecure RSA and DSA public keys in use on the Internet: we found that 5% of HTTPS hosts and nearly 10% of SSH hosts shared keys for reasons we considered a vulnerability." See here: https://factorable.net/faq.html Mostly embedded devices that auto-generate their key on bootup. There was a pretty interesting talk at ccc where part of this research is mentioned: http://media.ccc.de/browse/congress/2012/29c3-5275-en-factha...

maaaats2• 11 years ago

What about "personal" private keys? I find it a hassle to add new keys for every device to each host I ssh into, so I usually just end up copying them from device to device..

ckuehl2• 11 years ago

Are you talking about SSH host keys? Those are normally generated automatically, and should be unique for each server.

(These are the things you're asked to verify the first time you connect to a new server.)

maaaats2• 11 years ago

Sorry, meant my key, the one I use to connect with and have added the public part of to numerous servers.

dlitz• 11 years ago

SSH agent forwarding should eliminate the need to do that.

iso8859-1• 11 years ago

All these questions have insightful answers. I'd say "use separate set of private keys for each originating device".

http://superuser.com/q/189355/33303

http://serverfault.com/q/170682/80866

http://security.stackexchange.com/q/40050/13510

http://security.stackexchange.com/q/10963/13510

http://unix.stackexchange.com/q/27661/14305

jamiesonbecker6• 11 years ago

That's what Userify is for... don't copy your keys around by hand! :)

jquast• 11 years ago

This is particularly problematic for virtual machines, which are starved of entropy (no sensors, audio device, etc.).

I have VM images begin without ssh host keys and have a service dependency on haveged: http://www.issihosts.com/haveged/ and http://security.stackexchange.com/questions/34523/is-it-appr...

atoponce• 11 years ago

For Qemu 1.3 and later, there is VirtIORand where you can have VMs attach to a HWRNG on the hypervisor, then food that into the kernel CSPRNG. This should be imaged by default, so on first boot, each VM has enough entropy for key generation.

rlpb• 11 years ago

On Ubuntu, pollinate solves this problem: http://blog.dustinkirkland.com/2014/02/random-seeds-in-ubunt...

goalieca• 11 years ago

Dumb question, why not have all processors contain a new hardware RNG based on thermal noise or some such thing? That was even VMs can get some stable entropy.

jamiesonbecker6• 11 years ago

Great point. and Intel's and AMD's black box RNG generators (RDRAND etc) provide entropy... but they're thought to be compromised (which would be nearly impossible to detect). At least other firmware attacks would/should/might be detectable. By definition, RNG's produce unpredictable output -- detecting predictability is much harder than with other potential attacks (versus, for example, the sum of two numbers, which should always give the same result).

So, FreeBSD and Linux both use hardware RNG's as entropy inputs and mix-in other sources of entropy as well (which hopefully mitigates any loss of available entropy and also adds in other believed-good sources of entropy such as timing, network traffic, etc).

Ubuntu is using a new network-attached source of entropy which is itself constantly reseeded with the network traffic used to access it. (There's some inception joke there somewhere..) Of course, you may not be able to rely on the SSL/TLS connection that you're using to access it, so you might be seeding with an attacker's steady stream of 1's...

Of course, getting access to real hardware entropy in a hypervisored or virtualized cloud server/instance is the second part of the problem.

The third part is to make sure that SSH server and client keys are distributed properly. That's what Userify is for, but it really only helps with the client/user keys.. it doesn't help with server keys. (Yet?)

mkj2• 11 years ago

I'd suggest that a larger security issue is that the 250,000 hosts are unnecessarily running SSH to the world. A MITM issue doesn't matter if noone's using the connection!

cbsmith• 11 years ago

Given the purpose of the key, this doesn't necessarily seem like a security flaw. If individual owners ssh on to the system, then it is a flaw, but if the primary purpose is to allow system updates, this seems like a perfectly reasonable approach.

jamiesonbecker6• 11 years ago

This was also the case with Digital Ocean's droplets on Debian for a time (they were re-imaging without re-genning keys.) It's not as uncommon as it sounds.

Userify (cloud ssh key manager) is getting a feature to force re-generation of server-side keys across an entire cluster remotely.

andor• 11 years ago

I'm not sure how I feel about Shodan. Their statistics are interesting, but apparently the primary use is finding vulnerable systems and matching exploits?

This is the list of most popular searches: https://www.shodan.io/explore

Duplicate SSH Keys Everywhere

Related Stories