Linode Manager Security Incident

http://status.linode.com/2012/03/manager-security-incident.html

by ErneX • 14 years ago

186 99 14 years ago

status.linode.com

Well handled.

mrschwabe• 14 years ago

Agreed. This is in sharp contrast to the Media Temple / Plesk debacle. Which affected how many thousands of sites? How long did it take them to admit it? Meanwhile, Linode reports full details of a hack affecting no more than 8 accounts the SAME day.

ErneX3• 14 years ago

It's good the fast acknowledgement, but I find it lacks in detail. I'd love to know exactly what kind of measures are they going to take so the chances of this happening again are truly reduced.

I share a Linode with a friend and at work we deal with 4 so this is a concern.

Lazare• 14 years ago

There's already quite a lot of discussion of this incident here: http://news.ycombinator.com/item?id=3654110

orofino2• 14 years ago

I'd like to understand the what actions will be taken to prevent similar attacks in the future. Also, what can I as a linode customer to prevent my host from being compromised in a similar fashion.

Implementation of two factor authentication for your customers and requiring it for a root password reset would go a ways to preventing similar attacks.

eli2• 14 years ago

Two-factor authentication for their support personnel you mean?

orofino2• 14 years ago

It should be required for their personnel and an option for me to enable for my account/specific Linodes.

brown9-2• 14 years ago

Also, is this customer service portal available via a public URL? Shouldn't some sort of VPN access be required to even get on the network hosting these things?

ukd12• 14 years ago

It would be interesting to get a report on what actually happened - I've not seen anything official from Linode yet

strags• 14 years ago

While the incident is unfortunate, I do give them credit for being up-front, honest, and relatively speedy in their response. Sad to say, I'm not sure a lot of other hosting providers would be as quick to admit culpability.

That said, there's no way that resetting the root password should be something a customer service rep can do. Particularly when Linode are very explicit about not being a service for newbs - and are in general unwilling to provide help setting up a Linux system.

glfomfn2• 14 years ago

I am failing to understand what exactly happened.

The user who was affected by the incident quoted an email from linode that stated "Our investigation has revealed a customer support interface was used to access your account.", based on that and all the information of that post you get the impression that through the 'interface' the attacker was able to change the vps root password.

Now a reply from linode comes and says "The portal does not have access to credit card information or Linode Manager user passwords". So if the portal doesn't have access to Linode Manager how the attacker gained ability to change the root passwords ?

Thy should give more details on the incident, i do have a certain trust in the ability of linode to have a secure environment & i can understand that things like that will happen at some point to everyone. However its one thing for someone to get access in your system because you had your roots password to 'password' and another if there was a bug that got exploited.(yea this is an extreme example)

citricsquid• 14 years ago

> So if the portal doesn't have access to Linode Manager

They didn't say that, they said it doesn't have access to the passwords. They have an interface to change details, they just can't read them. So they can reset your password to "hunter2" but they can't see if it's "hunter2".

glfomfn2• 14 years ago

You are right, my bad on that. Still this looks like a Public relations post by them than giving out facts. They should be explaining what the attacker could do by gaining access on that interface, the ability of the attacker to change the password has the same consequences.

The point is that exploited interface had a backdoor access to the virtual machines (to be able to change passwords or w/e)

dissident• 14 years ago

It sounds like Linode's user system is different from their server management system. They probably have some administration tools for resetting VPS passwords, but don't have access to sensitive user details.

keeran• 14 years ago

Likewise.

   Suspicious events prompted an immediate investigation and the compromised credentials used by this intruder were then restricted

Does that mean the credentials were gained outside of Linode and used to change the root passwords of the accounts for the purposes of the theft, or were those credentials used as part of an exploit in Linodes systems?

dangrossman• 14 years ago

I run a web app. I built an administrative interface for managing it. This interface includes the ability to log me in to any individual user's account (by appropriately initializing a session with that user's ID, the same as logging in normally would do), and to reset any individual user's password. This interface does not let me view any user's passwords; it's both technically impossible, as I don't store them in plain text or encrypted form, and unnecessary.

Unless you believe they're lying, then Linode has the same thing. Some interface where they can access their users' Linode Manager accounts, but that interface does not show Linode Manager passwords or credit card information.

amalag• 14 years ago

No mention of liability. So they are fixing it, but someone is out of $12k of bitcoins. Linode says, tough luck for trusting us with your valuables.

techiferous2• 14 years ago

Best thing to do would be for Linode to generously reimburse these people, but they have no obligation to do so.

As a policy, it would neither be good business nor appropriate for Linode to assume all of the risk in a situation like this.

Also, if Linode did put a policy in place to assume some of the risk (some sort of insurance policy) they open themselves up to scams (just get your friend to rob your bitcoins and cash in on Linode's good will insurance policy).

larrys2• 14 years ago

"Best thing to do would be for Linode to generously reimburse these people"

Creates a precedent and greater future liability.

plox• 14 years ago

The scam you mention would be difficult to pull off as there would have to be evidence that the account was compromised on Linode's end for the customer to make a claim.

kylebrown3• 14 years ago

One user (slush) lost $12k worth of bitcoins, but another (bitcoinica) lost upwards of [$40k-$50k - wrong].

EDIT: first report from bitcoinica was "over 10k btc". Most recent report is 43,554 BTC, which would be worth almost $200k if liquidated on MtGox at the moment.

bdonlan2• 14 years ago

If all of those bitcoins were liquidated at once, that would probably depress the market significantly.

chrissnell• 14 years ago

These bitcoin servers trusted their currency exchange to cloud servers--VPSes, really--and they want Linode to compensate them for the money lost? Insane. I don't claim to know anything about the bitcoin infrastructure hosted here but I'm going to go out on a limb here and say that there was no dedicated hardware firewall in front of it, no IDS, no WAF, nothing but some Linux instances running iptables.

The payment card industry wouldn't certify this hardware to process credit cards for a mom-and-pop online business, yet these guys use it like their bank? Come again?

nwmcsween3• 14 years ago

PCI-DSS is actually quite easy to get certified in. I would say the issue is Linode not protecting their customers but as you say it's completely optional and would incur higher costs.

Tichy2• 14 years ago

So you are saying that you can not use cloud hosting for anything serious? Even if they don't keep cash around, most web apps might lose a lot of value by being hacked. user data stolen, reputation lost and so on. So according to you, the cloud is only for worthless toy projects.

_Lemon_2• 14 years ago

If you put $12k of value on their service, why do they take on that risk? Why wouldn't the risk fall squarely on your shoulders?

Why would a hosting provider take on the liability of what you host on it? If the underlying filesystem had an error that Linode could avoid and you lost data, would you expect them to replace the value that was lost? Why is it not limited (at most) to the value of the VPS itself? (I can't imagine them compensating for hardware failure either).

tyree732• 14 years ago

Any business that would be running on linode could be worth potentially far more than that. What your saying here is that no one should put anything of value into a VPS, which I imagine is not something linode itself would say.

look_lookatme• 14 years ago

I'm surprised that a tool with this level of access isn't walled off behind a VPN.

rdl3• 14 years ago

I really hope the various bitcoin related incidents don't poison the well for online currencies in general. Yes, you need a greater level of security when dealing with transferrable, relatively anonymous or pseudonymous (and rapidly extractable) online assets that you don't need for book-entry accounting with an audit trail and reversible transactions (credit cards, ACH, etc.). Yes, this is beyond what even most banks currently use. No, it's not beyond current technological state of the art.

Gaming (i.e. casinos), at least some of them, do a reasonable job with some very similar security problems.

wmf2• 14 years ago

Wasn't the well pretty thoroughly poisoned by e-Gold and such? (We realize they're not the same as Bitcoin, but most don't.)

rdl3• 14 years ago

Yeah, kinda. Although E-Gold's problem was force majeure, not technical (did they ever get hacked? people stole passwords all the time, but since it was double entry, it was pretty easy to reverse).

osGold, etc. were probably better parallels. But yes, this whole field has had a long series of technical, business model, governance, and government related failures.

dspillett• 14 years ago

> I really hope the various bitcoin related incidents don't poison the well for online currencies in general.

It shouldn't do significantly. In the public's mind the well is some distance off, many not even knowing it exists, so there is plenty of time to get security better sorted before the average man on the street has his money invested.

Also, everyone knows that cash and other forms of investment are far from safe anyway. Hopefully people will eventually see online currency as being no less safe than other forms, and if the security is done right they'll see it as more safe (as it could be).

But this seems to me to be a general security issue with decentralised anything, not a bitcoin specific problem. If you remove central control, and take as full control yourself as possible, then you remove responsibility from other people and that is something you need to seriously think about. Providers like Linode will not be taking responsibility for financial loss in these cases and paying for hefty insurance policies to underwrite the possibility of said loss: they'll just add clauses to their contract disavowing themselves of responsibility if such clauses don't already exist.

One way to protect yourself is to spread the money around multiple places, then one hacked provider doesn't put all your resource at risk. Again this isn't bitcoin specific: if you have more then 50K to invest over here (in the UK) then you split it between multiple organisations as only 50K per registered organisation is guaranteed protected by national safety nets provided by government.

Back onto "poising the well" this could of course work the other way around: if the virtual currency is worth the effort of stealing then it might be seen as more valuable in the eyes of the public - much like a sign you have a good product is that knock-off copies start appearing.

shazow• 14 years ago

Perhaps people affected by the many Bitcoin Exchange password leaks were reusing their passwords on Linode?

_wp3r• 14 years ago

The breach originated on the Lionode customer service system. Which the attacker used to reset user passwords. Unlikely the incidents are directly related.

nwmcsween3• 14 years ago

Can someone with a legal background comment on the viability of the victims taking legal action?

keypusher• 14 years ago

I've got to wonder too. If you rent out an apartment and someone breaks in with a key they stole from the landlord and takes your TV, can you sue the landlord? On the other hand, if you rent a security deposit box from a bank and it gets broken into, is the bank liable?

mbreese• 14 years ago

I think the threshold would be, did Linode take "reasonable" precautions in protecting the servers in question. Just like the landlord or bank. So long as they take reasonable precautions, they can't be held liable.

You can expect for your landlord to keep their copy of the key in some sort of lockbox, but you aren't going to expect them to keep it in a pressure-sensitive safe, guarded by movie-style lasers and a German Shepard.

You expect a little more security out of your bank.

The real question is: why were services that act like bitcoin banks storing their coins on Linode in the first place.

mindstab• 14 years ago

It all comes down to insurance. No one generally pays out of their pocket, they pay with insurance money. The reason you pay so much at a bank for a tiny box, where as, space wise, you pay a fraction for your appartment, or essentially change for a storage locker, is that the bank expects you to put valualbles in that space and so pays lots of that money to inssurance so if they do get robbed, they are covered and can pay you. The other places... dont. And usually have clauses in their contract saying as much: "We aren't insured for valueables, so don't store them here, it may not be safe and we won't (and can't) pay you back if they get stolen.

Linode falls into the second category, cheap, uninsurred storage space. You want storage space insurred against digital robbery, a) good luck b) expect to pay a lot more than $20 or $30 a month

jostmey• 14 years ago

Hey, I'm one of those guys! But I wasn't stupid enough to leave a stockpile of bitcoins sitting around to steal.

olefoo• 14 years ago

I wonder about that. It seems to me that the bitcoin wallet shouldn't have been accessible after reboot, at least until someone came by to give the agent managing it the passphrase that would allow it to decrypt it's state. From the sound of things the wallet was just laying around unencrypted?

ceol• 14 years ago

https://bitcointalk.org/index.php?topic=66979.msg778666#msg7...

According to the admin of bitcoinica, he didn't have the wallet encrypted because the ruby gem he was using to process withdrawals didn't support it.

meow2• 14 years ago

May be its time for some cloud insurance (covering outages/security etc) ? I don't mind paying a little extra if it mitigates risk.

tptacek• 14 years ago

Near unlimited liability and virtually zero ability to model the likelihood of events. A hard product to design.

meow2• 14 years ago

Liability might be unlimited, but most of the insurance products for other categories cap it based on premium. So that may not be much of a problem. I agree with the difficulty of modelling the events though... they need more data for any kind of modelling.. I'm guessing amazon already has some kind of insurance to cover its risks under AWS service agreement..

videoappeal4• 14 years ago

All this talk about banks being safe yada yada and cloud hosting not safe for US50k. Real banking companies (with billions of dollars on hand) do use commodity cloud hosting including Linode, for even sensitive parts. Take for example Natwest online banking login. On initial login page they load a cookie via an image from www.advanced-web-analytics.com and then once you enter a customer number the next page loads a ...drum roll... javascript file from www.omni-traffic.com. Now who can tell me what one can do when you have control over the Javascript on a banking login page?

Ah crap. It looks they have been moved to Amazon EC2, ~8 months ago they were hosted on conventional Linode VPSs. Points still stands though.

tomg4• 14 years ago

In my experience working on a US financial website, a bank would never consider using a VPS like Linode to store actual banking and customer data. It's not even close to Level 1 PCI compliant.

videoappeal4• 14 years ago

Well tomg, when I researched this ~8 months ago there were at least 2 US financial websites that were using the same specialized analytic company that injected JS into banking login pages that were hosted on Linode VPSes.

videoappeal4• 14 years ago

Not sure about this PCI complaint stuff, but perhaps this is why major banking companies jumped from Linode to EC2? Much improvement? Although I must say I have friends working on banking websites in the UK that dont know the whole picture, its not unreasonable to assume that these things are fucked up.

VonLipwig• 14 years ago

The message seems to be...

Only 8 accounts were affected. Do not worry. Minor breach. Not much harm done.

It seems to me the truth is the attacker looked for bitcoin wallets and emptied them. The fact he could identify 8 accounts and access them suggests the attacker could have accessed far more accounts if they wished. I think this is the most worrying thing about the breach.

I don't really understand how bitcoin works but it seems that people with wallets need to set up multiple wallets on multiple providers and limit the amount of bit coins in each wallet to limit any losses from breaches like this.

If I was a linode customer I would be thinking about moving. This message, while fairly open, doesn't give me much confidence there aren't other security issues with the platform.

microtonal2• 14 years ago

Indeed, we are a Linode customer, and this message only helps a bit. Yes, I know now that we are not affected. But little other information is given: were the user accounts compromised by a vulnerability in Linode's VM management software? If so, was this vulnerability found and fixed? Or did the attacker compromise the account of one of Linode's employees?

epaik• 14 years ago

I believe in an earlier statement they did say that they fixed the vulnerability. No idea why they didn't explicitly state so in this new statement.

drostie• 14 years ago

Just imagine that BitCoin is like having cash in your wallet, because that's more or less its intended model. There are a lot of 'anti-counterfeiting' measures because computers are very good at copying, and you don't want people to be able to copy BitCoins the way they can copy music -- and when you ask "what is BitCoin?" people basically start to tell you about the anticounterfeiting technology, and the limits on printing uncontrollable amounts of money. But it's essentially stamped paper in your wallet in any other sense, worth whatever people using it on the Internet will pay for it, not backed by anything in particular but its usefulness.

Basically a lot of people were renting storage rooms in an apartment complex run by Linode, you get your own key to enter the door and retrieve and store things -- whatever. Some people left their wallets inside these buildings, with cash therein. Someone else used some unidentified systematic security flaw, but we don't yet know what it was. Maybe there is a ventilation system which is easily navigable once you know how to get in; or maybe all of the rooms have unlocked windows for no good reason; we haven't been told yet. (There are some suggestions that they stole a key from one of the janitors who cleans these rooms up.)

What we have been told is that some burglar stole eight wallets, and that "All activity by the intruder was limited to a total of eight customers, all of which had references to 'bitcoin'." That suggests that the burglar did indeed peek in the windows beforehand somehow, to find out that these 8 rooms had wallets inside. Otherwise, presumably they would say something like, "The intruder broke into many of our customers' accounts but didn't actually do anything in 99% of cases." In that sense I think the scary bit isn't that he accessed the 8 accounts, it is the fact that he identified them in the first place.

Amortizing the loss across many points of failure may be a good idea, but it wouldn't seem to solve the central problem. Suppose I put $20 in two accounts with 5% chance of compromise, rather than $40 in one account with 5% chance of compromise -- either way, I should expect to lose $2. What I've changed is that I am more likely to lose some of my money (9.75%), but I am less likely to lose all of my money (0.25%). This may appeal more to risk-averse people but it is not fundamentally changing the situation.

Perhaps a better approach is to keep a BitCoin wallet encrypted, since that's pretty simple to do in day-to-day life. This is something that you can't do with your wallet -- you cannot turn your wallet into a steel vault with two-foot-thick walls.

lambda2• 14 years ago

> In that sense I think the scary bit isn't that he accessed the 8 accounts, it is the fact that he identified them in the first place.

This isn't all that surprising. There are basically two reasons why you would have a Bitcoin wallet on a server: if you are mining using the CPU power of that machine, or if you need to send Bitcoins from an online application. For example, one of the people who mentioned having coins stolen was from a mining pool; you need some automated system to pay out the earnings to the people who have been doing mining, and so the wallet for that automated system was on the server, and was stolen. I suppose one further reason might be as a backup, but in that case, I dearly hope that it's an encrypted backup without the encryption keys in the sever.

Given these reasons for having your wallet on the server, it's not surprising that people found them. These require network-facing services, that are easy to trace back to the server in question. The mining pool is a public service; anyone can join, and find the address of the server. Furthermore, when you make payments, you announce them to the full Bitcoin network. Someone sniffing transactions can watch where transactions originate, and target that. If they already had a compromised customer service account on Linode, they probably watched the Bitcoin network for a while, made note of transactions originating from IP addresses in the Linode range, and then targeted those accounts.

One way to protect yourself from this would be to proxy your Bitcoin transactions through a host other than the one that has the wallet, obfuscating where the transactions are actually coming from. You could even go so far as to make all of your transactions via Tor, which would probably make it fairly difficult to find where your Bitcoin wallet actually lives.

> Perhaps a better approach is to keep a BitCoin wallet encrypted, since that's pretty simple to do in day-to-day life. This is something that you can't do with your wallet -- you cannot turn your wallet into a steel vault with two-foot-thick walls.

The problem is, if you need to make payouts from your wallet, then the machine that does that needs to be able to decrypt the wallet. That machine can then be compromised to be able to steal your keys. Encryption doesn't buy you all that much, unless you are just doing a backup and don't need the machine to be able to do online transactions at all.

Perhaps another solution would be to encrypt each key in your wallet separately using a k-out-of-n encryption scheme (where produce n keys, any k of which can decrypt the wallet). You can then distribute those keys to independent hosts, which hopefully should not all be subject to the same vulnerabilities. Then any time you do a transaction, k of those hosts will need to produce their key to decrypt the key in your wallet and perform the transaction. That way you would have to compromise several different, independent hosts in order to steal the wallet.

Of course, this would drastically increase the cost and complexity of the system; and you would need to ensure that whatever system that authorized payments was likewise distributed, which if you had, say, a web-facing service would be difficult.

The easiest thing to do to reduce the risk is to only leave enough value in the wallets that are on the servers for a couple days worth of transactions. Then you transfer Bitcoins from a more secure location once a day to keep the coffers full. This is not much different than a physical store; yes, you are at risk of being robbed, but if you only have one days worth of cash there, with the rest somewhere more secure, you reduce how much risk you have.

mdda2• 14 years ago

It wasn't clear from the memo, but how did the attacker know which 8 specific accounts had 'references to bitcoin' if they didn't access other accounts too?

calloc• 14 years ago

Are you able to name your VM's on Linode?

desade• 14 years ago

In Linode's FAQ, they mention that their virtualization management software was developed in-house. They seem quite proud of this, quipping: "The Linode Manager is custom software, written in house, and is not for sale (although others have tried to mimic it)."

It seems to me that this is a classic example of security failures following inevitably from a lack of peer review. Maybe Linode didn't consider its LM software to be peer-reviewable, but I bet the victims of the bitcoin thefts wish that someone else had tested the code (and human systems surrounding it) for vulnerabilities.

Is this not exactly what Bruce Schneier frequently points out? Anything that must withstand attacks to protect the valuables within should be tested by attacking it. A lot. My hunch is that the vulnerability exploited by this attacker would have been found and fixed already if the LM software were more open.

jarito• 14 years ago

I think you may be a little off here. The statements in the thread seem to indicate that the compromise was not based on a vulnerability in custom software, but compromised credentials. You can certainly argue that the management console should be protected by two-factor (and it should be), but their software doesn't seem to be at fault here.

I would be willing to bet that they have had the system tested by external security contractors and scanned with automated scanning tools. This seems to be a people problem and features problem not a vulnerability problem.

I guess we just don't know at this point. You may very well be correct. I guess if you want to use an open source provider, just make sure they are running OpenStack (openstack.org).

Linode Manager Security Incident

Related Stories