Hetzner Servers Compromised

http://wiki.hetzner.de/index.php/Security_Issue/en

by martindale • 13 years ago

180 130 13 years ago

wiki.hetzner.de

anoother• 13 years ago

Full text of the email sent to cutomers:

Dear Client

At the end of last week, Hetzner technicians discovered a "backdoor" in one of our internal monitoring systems (Nagios).

An investigation was launched immediately and showed that the administration interface for dedicated root servers (Robot) had also been affected. Current findings would suggest that fragments of our client database had been copied externally.

As a result, we currently have to consider the client data stored in our Robot as compromised.

To our knowledge, the malicious program that we have discovered is as yet unknown and has never appeared before.

The malicious code used in the "backdoor" exclusively infects the RAM. First analysis suggests that the malicious code directly infiltrates running Apache and sshd processes. Here, the infection neither modifies the binaries of the service which has been compromised, nor does it restart the service which has been affected.

The standard techniques used for analysis such as the examination of checksum or tools such as "rkhunter" are therefore not able to track down the malicious code.

We have commissioned an external security company with a detailed analysis of the incident to support our in-house administrators. At this stage, analysis of the incident has not yet been completed.

The access passwords for your Robot client account are stored in our database as Hash (SHA256) with salt. As a precaution, we recommend that you change your client passwords in the Robot.

With credit cards, only the last three digits of the card number, the card type and the expiry date are saved in our systems. All other card data is saved solely by our payment service provider and referenced via a pseudo card number. Therefore, as far as we are aware, credit card data has not been compromised.

Hetzner technicians are permanently working on localising and preventing possible security vulnerabilities as well as ensuring that our systems and infrastructure are kept as safe as possible. Data security is a very high priority for us. To expedite clarification further, we have reported this incident to the data security authority concerned.

Furthermore, we are in contact with the Federal Criminal Police Office (BKA) in regard to this incident.

Naturally, we shall inform you of new developments immediately.

We very much regret this incident and thank you for your understanding and trust in us.

A special FAQs page has been set up at http://wiki.hetzner.de/index.php/Security_Issue/en to assist you with further enquiries.

Kind regards

Martin Hetzner

amarraja2• 13 years ago

Have all Hetzner customers received this mail? I currently have a couple of servers with them and have received nothing yet.

moepstar3• 13 years ago

AFAIK yes - it might take a while though, i've gotten mine quite a while after the first ones seemed to have gotten them..

sunsations• 13 years ago

I also got the mail.

PawelDecowski• 13 years ago

I just got mine now (almost 45 minutes after this submission to HN) so yours is probably still in the queue.

babuskov• 13 years ago

I did not get it yet. Perhaps they are only sending to those who might be affected?

mythz• 13 years ago

yes, received mine this morning as well.

amarraja2• 13 years ago

Thanks all, finally got my mail. I was worried my account was hijacked! Panic over

e12e4• 13 years ago

I can also confirm receiving the email (but I really need to tidy up my mail filters, I actually read about the hack here first).

Nice email, one improvement would be to host the wiki behind a proper https-site -- especially since we already know the attack appears somewhat sophisticated, and that the attackers gained control over servers on the Hetzner network. Now, the faq doesn't contain anything obviously bad (email your credit card info to... etc) -- but one small thing that could be improved.

Other than that I appreciate how Hetzner have handled this so far.

gommm• 13 years ago

Just got the email too. I'm quite happy with how clear they are about what happened and how they hash their passwords. Very different from Linode.

I just wish that they used bcrypt instead of a salted SHA256 but at least it's salted (and anyway in my case I never reuse passwords so no big deal).

darkarmani4• 13 years ago

> I just wish that they used bcrypt instead of a salted SHA256 but at least it's salted (and anyway in my case I never reuse passwords so no big deal).

If they fall under PCI:DSS, they might not be able to use bcrypt since it isn't an official recommended standard. (I am of course assuming that they mean PBKDF2 when they say salted SHA256.)

josephagoss• 13 years ago

I don't understand, are they saying the backdoor might be because of the Nagios software or because something has implanted itself into the Nagios software?

freehunter2• 13 years ago

I am not a Hetzner customer, but the comment about Nagios perked up my ears. I work with some clients who have Nagios running in their environment, and I'm wondering if there is an exploit in Nagios or if it's just a coincidence that this was where they noticed the infection?

Ecio78• 13 years ago

It makes me wonder if there's any connection with the recent Drupal Security problem (they cited a "third-party software installed on the Drupal.org server infrastructure" but they haven't - afaik - disclosed the software name yet)

staunch• 13 years ago

It's most likely that Nagios was simply used as a tool. It's easy enough to add a Nagios plugin that does something naughty and call it check_inode_usage or something.

..not that I'm a fan of Nagios' architecture though. I'm not.

scott_karana2• 13 years ago

I suspect it's simply out of date. Nagios does have a large surface area, but from what I've seen in the past, monitoring systems are very difficult for sysadmins to want to upgrade. :)

devicenull• 13 years ago

I'm kinda confused about why Nagios was even involved here. If you've got all your checks setup properly, the Nagios host doesn't have any write access to your monitored hosts. You can use NRPE to restrict what commands it runs. Combine that with a read only SNMP account, and even if your Nagios server is compromised it cannot access your server.

wyck• 13 years ago

Well it's hard to tell what Hetzner means, it could be someon just piggybacked onto Nagios, but there there is also this, http://www.cvedetails.com/vulnerability-list/vendor_id-1424/...

pdpi• 13 years ago

Huh, this doesn't make me happy. I haven't gotten an email yet.

larrys2• 13 years ago

"The standard techniques used for analysis such as the examination of checksum or tools such as "rkhunter" are therefore not able to track down the malicious code."

rkhunter is out of date.

http://rkhunter.sourceforge.net/

http://en.wikipedia.org/wiki/Rkhunter

We use it as well but I'm not sure it is an example of a serious security practice when given with only one other "checksum or tools such as" type statement.

tonetheman• 13 years ago

hmmm... what does Hetzner do?

moepstar3• 13 years ago

They're a German Server/Webspace ISP - see www.hetzner.de/en

th0br0• 13 years ago

Not just German actually, they (Hetzner Online AG) have also got a subsidiary in South Africa ... http://www.hetzner.co.za/

dspillett2• 13 years ago

They are a provider of hosted dedicated servers, colo and virtual machines (with or without server management services), based in Germany.

samspenc• 13 years ago

They're really inexpensive, which explains their niche popularity...

tbh3• 13 years ago

Considering https://twitter.com/omgtbh/status/337567604887658496 I can't say I'm surprised...

(tweet text reproduced here: "I asked Hetzner if they plan to support 2 factor auth & was told that they already do - they require a username and a password. Seriously.")

aw3c2• 13 years ago

You don't seriously think that a server admin said that, do you? It was probably some low rank customer service peon.

tbh3• 13 years ago

Of course not, but regardless of who in the company said it, the official support response was as quoted. Not exactly encouraging!

FooBarWidget13• 13 years ago

Sigh, another hosting provider hack. As if Linode and OVH are not enough. This is the reason why we use full disk encryption, where we enter the key manually during boot. This way we're protected against many types of hosting provider hacks.

pekk2• 13 years ago

What kind of effect does this have on your I/O performance?

FooBarWidget13• 13 years ago

Minor. It costs more CPU because of encryption/decryption. But our servers have so much CPU power, and our workload is mostly dependent on disk I/O throughput, so enabling encryption is almost free.

gkoz2• 13 years ago

What types of hacks does that protect you from exactly, considering you don't have physical access and can't even see the hardware?

FooBarWidget13• 13 years ago

It protects against hackers that compromise the provider's administration interface, and then use that interface to reboot the server to single user mode or a rescue system. Upon reboot, the attacker will encounter an unusable blob as filesystem because he doesn't have the encryption key.

It also protects against hosting provider employees that detach the hard disk and attach it onto another machine to copy data off it.

If the server is virtualized, then in theory it's still possible that the attacker hacks into the hypervisor, and then through modifications in the hypervisor's RAM gains access to the guest kernel, and thus the guest system. However the skill that is required for this is an a whole new level compared to "regular" hacks.

If the attacker has physical access then it's still possible that he uses electromagnetic emissions from the hardware or power supply patterns to obtain information about it to hack the system. But again, this requires skills on a whole new level.

jonknee2• 13 years ago

Do you have a very small number of servers? Requiring manual steps to boot is not very scalable--if there is an event that requires all your servers to reboot (power outage, mandatory upgrade, etc) you would be in quite a tough spot.

FooBarWidget13• 13 years ago

I much prefer having to repeatedly enter the password dozens of times than risking data leaking out - or worse, data being tampered with.

epochwolf3• 13 years ago

How does this help? The key is still stored in memory which I assume the hypervisor has access to.

FooBarWidget13• 13 years ago

Correct, but hacking into the hypervisor is harder than hacking an administration interface. In the end the only secure server is one in a vault at the bottom of the ocean, but there are ways to prevent certain attack vectors.

tbh3• 13 years ago

Hetzner provide a lot of physical machines too, I believe this is what the other poster was talking about.

mike_esspe2• 13 years ago

What if attacker infects unencrypted part used for booting? Do you have protection against that?

It's relatively easy on Linux, where /boot is usually not encrypted.

scott_karana2• 13 years ago

Agreed, but typically /boot isn't automounted when it's a separate partition, which means they would need a root-access compromise already.

FooBarWidget13• 13 years ago

The trick is putting all important data in the encrypted part. The unencrypted part must be reconstructable in an automated manner. If you are paranoid and you assume that all unplanned reboots are attacks, then you can reconstruct the unencrypted part every time. Tools like Chef make this relatively easy.

e12e4• 13 years ago

Well, maybe. Do you have servers with a secure boot environment?

Also, if the attackers are able to attack your servers in a similar fashion (gaining access without reboot) -- they have full access to the already decrypted disks (well virtual devices).

As for other comments on automating booting -- as eg cryptsetup allows entering the encryption key via ssh -- automating that shouldn't be too hard.

I still think that it mostly defends against someone stealing/cloning the physical disks though.

FooBarWidget13• 13 years ago

Even without secure boot environment, if you're paranoid enough you can throw away and reconstruct the boot environment on every unplanned boot.

Encrypted disks are not meant to protect against attackers that can gain access to the server without rebooting. But all your OS-level security mechanisms (user separation, AppArmor, file permissions, firewalls) become useless if the hosting provider's admin interface has an exploit and the attacker can reboot the machine into single user mode. In other words, the Linode scenario. That's what encrypted disks protect against.

X42• 13 years ago

Seriously, again??

I think such a fauxpas shouldn't be tolerated twice.

M4v3R• 13 years ago

Do you have link to their previous hack?

atesti• 13 years ago

In October 2011, unfortunately these links are in German only:

http://www.heise.de/security/meldung/Web-Hoster-Hetzner-geha...

http://www.heise.de/ix/meldung/Datenleck-bei-Hetzner-1356468...

http://www.heise.de/security/meldung/Passwortklau-bei-Hetzne...

There was unauthorized access to customer data, even passwords. Someone claims to have accessed it via an FTP-server where he found a root password to a management server

925dk4• 13 years ago

Here is the (english) mail they sent to customers back in October 2011 on the previous intrusion:

http://pastebin.com/uJhG5aLS

pekk2• 13 years ago

Several events with Linode, now Hetzner. These are relatively "premier," high-quality hosting companies, you can count on thousands and thousands of companies to pay even less attention.

Yet every time, the discussion is only about one specific company, without seeing any broader pattern.

When are we ever going to draw the conclusion that popular hosting companies (and, actually related, facilities like RubyGems) are especially attractive targets, and that the approach of waiting for an exploit and then shaming the targeted company is not an effective way of getting better security?

TylerE• 13 years ago

Apply Occam's Razor. The simplest speculation is an internal breach. All the procedure in the world won't save you from that.

gtaylor• 13 years ago

They are of good quality, but let's keep in mind that Hetzner tends to be solidly on the budget side. I am a Hetzner customer, and it stinks that this happened, but let's consider: I don't expect Bugatti quality when I'm paying for a Chevy Malibu.

I do appreciate them being forthcoming and sharing some security details that do make me feel pretty safe. I don't care if they have the last few digits of my CC number, and I've already reset my password. Stinks I had to even think about this, but again, Chevy Malibu.

hhw• 13 years ago

Assuming that the exploit really is unknown, what better security could they have done to prevent it, without severely impacting usability?

Compromises will happen sooner or later when running a large number of public facing services. With good policy, the breaches will be well contained, which seems to be the case here. What more can be expected out of a premier hosting company, let alone Hetzner which is very cheap budget provider?

saturdayplace• 13 years ago

Is it just the customer account details that apparently make hosting companies attractive targets. If that's the case, I'm wondering why we're not seeing more breaches from all over the e-commerce world. Why just hosting companies?

ahi• 13 years ago

Maybe the hosting companies are simply more likely to detect a breach. Not a happy thought I know.

jtchang• 13 years ago

The lesson to take from this is you can't rely on someone else providing security for you. You need to make sure you "layer" your security. If your database is compromised make sure you are salting your passwords or at least using a hash that is designed to be resistant against cracking.

This doesn't protect against someone who has root access to your box and manages to capture passwords in the clear as they are being transferred but it does limit the surface of a breach.

berkay• 13 years ago

Things seem to be getting worse there. A server we have there just went offline, and their management servers are not responding. I wonder it's part of the exploit or as a result of attempts to "fix" the problem

zapt02• 13 years ago

I had 5 minutes of downtime, but everything seems to be back in working order.

rebelde• 13 years ago

Something was happening with their routers, too. Using PingPlotter to one of my servers shows severe packet loss on the hos-bb2.juniper2.rz19.hetzner.de router.

I see it just cleared up for now. Still, I'm moving my traffic to other locations till this blows over.

PaulFreund2• 13 years ago

That might be because various parts of germany are flooded at the moment. One of Hetzners datacenters is located in Falkenstein near Regensburg and others might me affected too

illumen• 13 years ago

Their SSL certificate is fucked for their security page... oh dear.

https://wiki.hetzner.de/index.php/Security_Issue/en

Hope everyone has fun with their identity theft cleanups. They ask people for ID scans.

Bank details are compromised.

boo!

This was announced AFTER the German work day was done. Really nice for admins. Thanks.

StavrosK• 13 years ago

You would have preferred that they waited until tomorrow?

925dk4• 13 years ago

As part of the registration process with hetzner.de, you have to send them scans of personal documents (such as passport, drivers license or similar).

I asked them just now if these systems were compromised and they promptly replied:

"The system that stores scans of ids, credit cards and so on was not compromised. In addition to that, we delete that information after 21 days."

larrys2• 13 years ago

"you have to send them scans of personal documents (such as passport, drivers license or similar)."

What? Seriously?

"In addition to that, we delete that information after 21 days."

Hmm, do they delete the info that ends up on backup copies? How do you know they even actually delete it in 21 days? It's not like there is a third party even auditing which you can rely on. (Not that I'd ever do that for something like this anyway, I would just find another provider.)

925dk4• 13 years ago

Yes seriously, here is what they asked me on first order with them:

"Since you're a new customer with Hetzner, we ask you for a scan of your passport or ID card (authenticity check). It's only necessary for your first order with us.

Please send the scan by fax or as an email attachment."

When they say they delete it after 21 days, as they did in the mail I've just received, I trust them. I find their communication on this matter, as well as previous matters, open and serious.

Dylan168074• 13 years ago

If they're proper backups then they're offline and you don't have to worry about them getting hacked.

WA2• 13 years ago

Is this outside of Germany? I rent two servers and I never sent any ID.

This shit is annoying. I think it have been only 6-8 months since the managed server part of Hetzner (KonsoleH) got hacked. Now the VPS/root server part (Robot) got hacked. I understand that both incidents are completely different and it seems that they might've learned a thing or two from the KonsoleH-hack, but still. My address data and my bank data are very likely to be compromised.

But then, changing the hoster doesn't make any sense. My data is somewhere out there, can't get any worse I guess.

gokhan• 13 years ago

The info in the mail regarding the safety of credit card info contradicts with the linked FAQ.

FAQ: Bank details are encrypted (two-way) in the database. However, it cannot be excluded that the attacker/s have also been able to obtain access to the key.

Mail: With credit cards, only the last three digits of the card number, the card type and the expiry date are saved in our systems. All other card data is saved solely by our payment service provider and referenced via a pseudo card number. Therefore, as far as we are aware, credit card data has not been compromised.

antoko• 13 years ago

banking details is a much broader term than credit card number.

account numbers / routing numbers / company names / po boxes/ physical addresses / account history and other billing info.

I'd assume they're saying some of these might be compromised, but specifically that credit cards aren't.

perlgeek• 13 years ago

Hetzner probably has many customers who pay with debit card/recurring direct debit instead of credit card. It's quite common in Germany.

thejosh• 13 years ago

They also allow PayPal, which I recently changed to with Hetzner. 99% of my payments to websites are PayPal, for this very reason.

LogicX2• 13 years ago

Perhaps the email is different based on that particular customer's payment method, which is why one is talking about credit cards and the other about banking details?

muxxa• 13 years ago

I've got a login for robot.your-server.de and have changed my password but don't seem to have one for konsoleh.your-server.de.

Is the konsoleh login/account something that would have gotten created for me automatically? i.e. do I need to worry about it?

linx• 13 years ago

You need a konsoleH account for the registration of 'special' domains not covered by the robot. One doesn't have one by default.

I think webspace users get a konsoleH account, too.

WA2• 13 years ago

KonsoleH = managed servers

Robot = VPS/root servers

ChuckMcM• 13 years ago

One of the differences between servers in data centers and desktops is that servers don't reboot a lot. So the bad guys can build an in memory system that doesn't change anything on disk (which avoids the configuration management system from flagging it) and if it cloaks itself as a normally long lived process then active monitors might be fooled as well. It then runs, effectively undetected, until the server reboots which can be months (or even years).

Tools to protect against those threats are going to have start taking into account process activity and footprint.

raylu2• 13 years ago

Hetzner is very open, transparent, and exact in their communication about this. I'm not a customer but I really like that they didn't try to dress anything up and were forthcoming with information.

johnsea• 13 years ago

Doesn't surprise me. I'm a customer and the whole experience feels like this.

sgt• 13 years ago

I use the Robot to access a VNC console of my leased server. I can even do a CTRL-ALT-DELETE and boot the OS in single user mode from it, if I only have access to the Robot.

Doesn't this imply that my server is potentially compromised as well?

alexvr• 13 years ago

Am I an overly-cautious nubcake, or do any of you also refrain from clicking the link when it says, "We've been compromised. Come visit our site!"?

If I were a mean dude who hacked into some prominent site, the first thing I would do is submit a link to HN that points to a malicious "article" and laugh as malware gets thrust into thousands of computers.

ricardobeat• 13 years ago

Way too much mork. If you have a way of infecting computers by simply visiting a webpage, just tag it some kind of porn and post on reddit & other forums.

Hetzner Servers Compromised

Related Stories