Identical Droplets in the DigitalOcean: Regenerate your Ubuntu SSH Host Keys now

http://missingm.co/2013/07/identical-droplets-in-the-digitalocean-regenerate-your-ubuntu-ssh-host-keys-now/

by jlund • 13 years ago

285 106 13 years ago

mey2• 13 years ago

I must say, I'm impressed with how this was handled both by the original researcher and DigitalOcean.

ihsw3• 13 years ago

Responsible disclosure can benefit us all, unfortunately some vendors -- SaaS, PaaS, physical, or otherwise -- use their legal departments as blunt weapons to needlessly attack well-meaning security researchers.

In case anybody is wondering, I'm referring to Volkswagen.

http://www.engadget.com/2013/07/29/uk-court-volkswagen-megam...

Personally I'm an advocate of full, anonymous, and public disclosure.

suhailpatel• 13 years ago

I too am an advocate of full public disclosure but the concerns by Volkswagen are warranted and fixes to the issue can not be deployed as easily as DigitalOcean has done here. I don't agree with the gag order from the UK courts but it isn't totally unjustified.

philtar2• 13 years ago

That's not a fair comparison. DigitalOcean just needs to update some stuff to fix this issue. VW would need to recall millions of vehicles (10s billions $). You would do the same thing if you were in their position and had shareholders to worry about.

specto4• 13 years ago

Except I informed them of this issue in January of this year

http://imgur.com/GTi2UxJ

Apparently they ignored it :)

kintamanimatt• 13 years ago

That worries me far more than the actual security issue. Security issues happen to everybody, but so long as too many don't occur, it's the response that shapes my ongoing confidence in that company or product.

What's happening on the left side? Is that you or the rep?

mey2• 13 years ago

Ok, that is disturbing...

ceol• 13 years ago

Except I don't see an email from DO or a notification when I log in to the admin panel. So if I didn't check HN at this exact time and saw this article, I would have no idea.

It's not a huge deal to me, but if Linode did the same thing, you all would be foaming at the mouth. Just thought I would point this out.

tallanvor• 13 years ago

Do you have any Ubuntu instances running or saved? If not, then they would have no reason to notify you of the issue.

makomk• 13 years ago

This is now one of the first things I check when setting up a new VPS or other VM instance, because it's really common.

voltagex_2• 13 years ago

I think I need a list of things to check. This sounds pretty scriptable though:

* SSH Host Key

* SSH Authorized Keys

* SSH PermitRootLogin

* Disabling password auth in favour of keys

* Security updates from the distro

* SELinux (maybe?)

Anything else?

icebraining• 13 years ago

The difference is that with most of those, you can create a snapshot of an instance and then duplicate the configuration, but Host keys are special, since they need to be re-created for each instance.

In any case, there was discussion on those issues not long ago: https://news.ycombinator.com/item?id=5316093

patio11• 13 years ago

a) Move SSH off port 22. (Really limited security gain when implementing the rest of the suggestions, but saves spam in your logs.)

b) Software firewall via e.g. IPtables is generally the first thing I turn on after rebooting SSHd with the new settings.

c) (Optional) Consider using an architecture where you have N boxes and SSH only listens on a local interface on N-1 boxes, with the Nth box running nothing but your VPN. (This is also a good architecture choice for admin consoles, folks. www.example.com resolves to a public IP, admin.example.com resolves to a private IP, so even if they're technically speaking on the same box/boxes you won't lose the admin console if someone unwisely uses the same password for a WordPress blog somewhere.)

gtani• 13 years ago

these are pretty decen outlines

http://www.quantisan.com/securing-a-fresh-ubuntu-server-with...

http://plusbryan.com/my-first-5-minutes-on-a-server-or-essen...

__david__• 13 years ago

SSH PermitRootLogin

I assume you mean to disable it? I see that listed in various places, but I don't understand why it matters… I like to kill all passwords on my VMs (so there's nothing to brute force), uninstall sudo and only use ssh keys to authenticate. I would like to know why this is a bad idea.

chimeracoder• 13 years ago

What do you mean by 'check' in this case? Check the keys against the keys for all other VM instances that you've provisioned from the same VPS provider?

jackalope2• 13 years ago

You don't even have to compare the keys. If they exist at all, they are probably copied from the image itself. The first time you start up sshd, it will create new keys. If you are unable to witness this, assume that they are not trustworthy, delete them and restart sshd (or replace them with keys you've generated locally).

The common problem is that sshd was started before creating the final image, so it has keys that are duplicated by provisioning. Always delete the keys before committing the final image, so that sshd will create new keys the first time it runs.

schappim• 13 years ago

Props to the way you handled this. That's how you do responsible vulnerability disclosures!

joshmn• 13 years ago

Now that it's said, I did notice something strange once.

I had loaded up an Ubuntu Desktop droplet with the purpose of checking something out through the browser on the node.

The startup page was https://www.americanexpress.com/

Since when is that default?

Didn't think much of it at the time, but now... whoa.

davidhollander• 13 years ago

After you have run those commands, simply restart the SSH daemon so it starts up with the new keys in place

I believe if your version of OpenSSH is up to date, sshd will read the host key each time a session is opened and does not need to be restarted.

throwawayh4xor• 13 years ago

Just verified this is also the case with at least some AWS-hosted servers. Coupled with the fact that many people simply ignore the MITM warning that SSH throws, this is scary stuff.

vacri4• 13 years ago

Curious. I just upgraded a small server to a medium, detaching the existing volume and reattaching it to the new instance, and it has given me a new fingerprint.

novaleaf• 13 years ago

unfortunately, devops (like me) need to ignore these as part of our work system.

i need to automate deployment / reprovisioning of 30 digital ocean servers. as reprovisionined servers frequently use the same ip address, i always run into this. for me I had to disable the check :(

Titanous• 13 years ago

By disabling that check, you are destroying a huge component of the security that SSH provides. Perhaps you could clean up the authorized_keys file as part of your teardown script?

gregf• 13 years ago

I ran into this issue as well. As the person below mentioned, make this part of your teardown script. You can use ssh-keygen -R hostname to remove host from ~/.ssh/known_hosts.

karlkatzke3• 13 years ago

I've ended up in this situation too. We migrate between datacenters with duplicate hosts fairly frequently at my primary place of employment, and best practice is to use the cname of the service to access the server currently acting as the primary for it (e.g. foo.bar.com as opposed to foo.datacenter.bar.com). That leaves me with a lot of foo.bar.com entries to clean out of known_hosts, or a lot of spurious MITM errors.

What I've done is added a whitelist to my .ssh/config to disable the alerts only for those hosts. The foo.datacenter.bar.com address (which I use often enough, usually when migrating it between datacenters) still alerts.

And yes, I know I'm living in the 90s what with my datacenters and whatnot. They're kind of like regions... what? Oh... why, you... You kids, get off my lawn!

ef4• 13 years ago

Part of automating deployment is automating key management. Ignoring host key checks is the worst possible solution.

You're actually better off just baking a shared key into your image. So long as it's a key you generated yourself (not like this Digital Ocean scenario, where the key came from the cloud host), only someone who has already rooted one server can successfully MITM your SSH connections.

Whereas if you ignore host key checks entirely, anyone who gains control of one network hop between you and your servers can own you.

Spooky23• 13 years ago

Not managing keys is asking for trouble. Consider signing your SSH keys to validate that you've properly pruned outdated host entries.

See:

http://www.openbsd.org/papers/asiabsdcon2011_openssh_whats_n...

http://blog.habets.pp.se/2011/07/OpenSSH-certificates

http://en.community.dell.com/techcenter/b/techcenter/archive...

Spend an afternooon to figure this out.

helper• 13 years ago

The official ubuntu AMIs generate a new key pair on first boot. The AMIs provided by Amazon do the same.

If you're using AMIs from some other third party you should verify that they do the right thing.

scottlinux• 13 years ago

I suspect this kind of thing happens with other companies, but can only speculate.

Somewhat related: chicagovps gave me a 'fresh' gentoo vps, and the default provided root password was identical to the original one from several months ago. I assume it is one gentoo image with the same password (for all customers)?

druiid2• 13 years ago

One good thing to note is that any VM image using cloud-init (a package for debian/rhel systems) should automagically generate a new host_key set for any new system image. Basically if you build a system image for EC2 or any system that uses the EC2 data format (like Openstack) for host instantiation, then you should install cloud-init. It would prevent something like this.

agwa4• 13 years ago

SSH host keys are problematic on cloud servers, not just because of this problem, but also because if the cloud provider does the right thing and generates the SSH host key on the first boot, the key is generated when the system has very little entropy available. The primary sources of entropy on Linux are key/mouse input, disk latency, and network interrupts. There's obviously no keyboard/mouse on a server, and in an SSD environment like DigitalOcean, disk latency is quite uniform and thus useless as a source of entropy.

Linux distros mitigate the cold boot entropy problem by saving some state from the RNG on shutdown (on Debian, it's saved in /var/lib/urandom/random-seed) and using it to seed the RNG on the next boot. On physical servers this obviously isn't available on the first boot, and on cloud servers, the provider often bakes the same random-seed file into all their images, so everyone gets the same seed on first boot (fortunately this doesn't harm security any more than having no random-seed file at all, but it doesn't help either). What cloud providers should really do is generate (from a good source of randomness) a distinct random-seed file for every server that's created, but I haven't seen any providers do this.

niels_olson• 13 years ago

It's slow, and manual, but this gives me one more reason to like prgmr, at least for my little side projects. You generate the keys, and send the public key to Luke. Your only access to the control layer is via public_key auth from that key you generated. Presumably on a laptop, with gobs of entropy available.

agwa4• 13 years ago

I believe you're confusing SSH host keys with the keys used for user authentication. The private SSH host key has to reside on the server, so it's either being generated on the server or you're sending Luke a private key.

But I agree this sounds like a good way to handle user authentication.

lsc• 13 years ago

as much as I appreciate the plug, that key I ask you for? that's to get you into the admin interface on my side. I don't mess with the ssh keys within your image. (host keys are auto-generated on first boot, just like a real server, which has the 'generating keys in a low-entropy environment' problem described above. The image doesn't come with anything in the authorized_keys file; though actually, I could pretty easily adjust the creation script to put the key you sent me in the authorized_keys file of root... but I don't really know the 'right' way to do that; everyone has preferences, so I leave it up to the user.)

We've actually talked a lot about trying to write a xen driver to share a hardware entropy device, but it hasn't gone anywhere. (I mean, xen has 'vtpm' - a virtualized Trusted Platform Module. And nobody uses that, so why not a vrandom?)

druiid2• 13 years ago

True, but much less of an issue than just leaving the same version of keys in place. In case you're curious, here's how the cloud-init library generally used for ec2 and ec2 compatible clouds works: https://github.com/number5/cloud-init/blob/master/cloudinit/...

agwa4• 13 years ago

Absolutely - baking the same private key into images is much, much worse (and in my opinion a pretty embarrassing thing for a cloud provider to do by accident).

sweis• 13 years ago

Regarding entropy sources for guest virtual machines, the host can expose a random number generator device to the VM. This is an option in KVM: http://libvirt.org/formatdomain.html#elementsRng

Modern Intel x86 processors now have a hardware RNG built in, so even if the host boots without any devices, you have a source of entropy.

(Incidentally, KVM defaults to /dev/random which might create a DoS vulnerability if a guest exhausts entropy.)

agwa4• 13 years ago

Interesting. Do you know if any cloud providers do this? (DigitalOcean uses KVM so it would be a possibility.)

Exposing hardware RNGs to guest VMs would be great; entropy in guests is pretty lousy in general, not just at first boot (though obviously at first boot it's at its very worst).

karlkatzke3• 13 years ago

We've run into the /dev/random problem when doing heavy SSL traffic. We had to pipe in sources of entropy. It was a janky solution, but wasn't as janky as it might sound.

anonymousDan• 13 years ago

Are you referring to the hardware RNG in TPMs? I thought they were very low bandwidth in terms of entropy, would that be a problem on first boot?

mistercow2• 13 years ago

It seems like booting up would actually be a great time for generating a decent amount of entropy, what with all those devices starting starting up from a relatively unpredictable physical state (temperature etc.), hard drives spinning up, and so forth. I'm kind of surprised that the Linux kernel doesn't exploit that.

tripzilch2• 13 years ago

I'm kind of surprised that the Linux kernel doesn't exploit that.

I wouldn't say "surprised", more that I wonder why and if there's maybe a good reason for it--that seems more interesting than the notion of surprise to me, anyway :)

Especially considering pretty much everything I've heard about entropy-generation over the years mentions that getting entropy just after boot is a difficult problem.

I'm assuming there must be some truth to that, so your surprise is my curiosity for the explanation why that probably wouldn't work (yes I'm assuming it won't work, sorry, but I really think otherwise they'd be using it :) ). Does anyone know?

prg3182• 13 years ago

Even so, the boot sequence of a VM guest is generally straightforward with very little entropy.

inopinatus2• 13 years ago

I was interested in providing entropy as a service, but I couldn't validate the proposition with the obvious customer segment (i.e. cloud HVM operators, whether public or private).

Essentially, the trust bar for entropy is set so high that no-one who understood the product enough to want it, was willing to trust it enough to actually use it.

(It's not just for SSH keys; random values are required for e.g. SSL connection setup, too)

pyre• 13 years ago

Sounds like most cloud-providers could just undercut your business by enabling such solutions locally though. I'm not sure how stable the business would be unless you were going to guarantee more entropy than they could locally generate.

prg3182• 13 years ago

In my experience, 'haveged' [1] has worked out really well for generating entropy on headless servers. It could be scripted into a provider's base distribution to include the haveged binary and run haveged before generating the SSH host keys.

[1] http://freecode.com/projects/haveged

foxhop• 13 years ago

So you are the reason I started getting these error messages, I noticed the change on June 2, great work.

If you are still reviewing salt, I just wrote a post about salt-cloud and DigitalOcean that you should check out -

Create your own fleet of servers with Digital Ocean and salt-cloud:

http://russell.ballestrini.net/create-your-own-fleet-of-serv...

sehrope2• 13 years ago

Generating fresh keys aside, one thing I do with our AWS setup is whitelist the IPs that can connect to our SSH bastion host. This completely eliminates scripted port scans of the SSH server and makes the auth logs much more manageable.

If our IP address changes (eg. ISP assigns a new one for the cable modem) then we just update the whitelist (and remove the old address). It's very infrequent. I could probably count the number of times I've done it on one hand.

It might not be the most scalable setup but at our small size with everybody working from home it works great.

The only slight hitch is updating it when traveling but even that isn't much of a problem. It takes a minute or two from the AWS console and its good to go.

I recently took a look at digital ocean ($5 servers gives me ideas...) but didn't see a firewall option similar to the security group setup in AWS. If it does exist then I highly recommend it.

matthewbadeau• 13 years ago

This is a very good idea. It could be script-able using the AWS API[1] (though I haven't tried it yet).

[1] http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiRef...

[EDIT] The relevant example is 4th in the list.

werkshy• 13 years ago

With Digital Ocean you'd need to install a software firewall on the servers themselves, there is no API-configurable network-level firewall. I used 'ufw' which was quite easy to get started with (on Ubuntu), and replicated my AWS security group config pretty quickly. I added the ufw config to my host setup scripts so it happens automatically.

sehrope2• 13 years ago

The problem with the software one is you need a way to modify it when you can't access the instance. With AWS you do it from the admin console or APIs. If it's on the machine itself you'd have to know the IP to open up in advance or have someone at home base do it.

rwmj• 13 years ago

They should be using cloud-init or virt-sysprep[1] on new instances. In particular, it is vital that you give your new instances a unique random seed (which virt-sysprep can do). Also that you provide the virtio-rng to guests that support it.

[1] http://libguestfs.org/virt-sysprep.1.html

joeblau• 13 years ago

Great find. I came from a heavy security background and moved to SV where it seems like security is an after thought. I spent many long days and nights STIGing RHEL boxes so I can appreciate this find. Also thanks for letting me know about Digital Ocean, their VPS looks promising and I think I might start using it.

Nux• 13 years ago

This is not the last of the problems we'll have with "the cloud", but I guess it's part of what makes it so exciting. :-)

Many people, especially beginners, make the mistake of leaving the same SSH keys in a certain template or in a snapshot of a virtual machine that they later use as a template.

There are a few files that you really, really need to wipe out from a wannabe image template:

- /etc/ssh/* key* (for reasons explained in the parent article. stupid autoformatting, remove the space after the first asterisk)

- /var/lib/random-seed (the seed used to initialise the random number generator. this is the location on CentOS)

- /etc/udev/rules.d/70-persistent-net.rules (so that the VM's new NIC - with a new MAC - can use the same "eth0" name)

People who want to do this more exhaustively can have a look at libguestfs and it's program virt-sysprep which does all of the above and more!

http://libguestfs.org/virt-sysprep.1.html

rlpb3• 13 years ago

To avoid this kind of security problem, use providers that use official Ubuntu Cloud images only. If Canonical haven't certified the Ubuntu images you're using, then your provider could have done anything to them. You'll need some other way to determine their competence.

Cowboy images like this are exactly the reason trademarks exist. Commercial providers who don't get certification are in fact violating Ubuntu's trademark by telling you that you are getting Ubuntu, when in fact you are getting a modified image which is possibly compromised (such as in this case).

regularfry3• 13 years ago

How do I validate that my provider is actually providing an official Ubuntu Cloud image?

rlpb3• 13 years ago

Technically? I'm not sure that's possible. They own the hypervisor, so you have to trust them. This is why the Ubuntu trademark is so important.

stevekemp• 13 years ago

We ran into similar problems on the hosting side; another surprise can be the debian-sys-maint password configure by the Debian mysql-server package.

Identical Droplets in the DigitalOcean: Regenerate your Ubuntu SSH Host Keys now

Related Stories