$5,000 Security Breach

http://blog.joemoreno.com/2014/04/5000-security-breach.html

by mantrax4 • 12 years ago

29 22 12 years ago

blog.joemoreno.com

'They' spin up spot instances which isn't subject to Billing Alerts. You'll need to cancel those spot instances, revoke your AWS credentials, and change your account password," he said.

This doesn't make sense at all. Amazon should let us if monthly bill > X send me a priority email and phone call. Why do they hide behind these dark patterns? I thought they were better than that.

theboss• 12 years ago

The fact of the matter is amazon is a giant company and cannot thoroughly think through each piece of logic in their system.

The reward for focusing on this before-hand is much lower than just writing a check for $5k to this person and then fixing later (lot of $5k checks from amazon today. Wheres mine?)

sandy23• 12 years ago

Is there anything you have done that might have compromised your account. Moreover bill could as well been much larger than $5000 i guess.

nilved• 12 years ago

The instance was spun up on April 2, but Heartbleed wasn't disclosed for almost a week later. I highly doubt anybody used the Heartbleed 0-day to access your account.

viseztrance• 12 years ago

According to Cloudflare (http://blog.cloudflare.com/answering-the-critical-question-c...), exploiting heartbleed may actually be very difficult. So yeah, it's very unlikely for that to have happened.

dmix• 12 years ago

Well, getting an SSL private key is difficult as they don't often get into memory and are quite long (difficult to get from 64k at a time). Whereas AWS credential keys are something that get into your servers RAM much more frequently and are shorter strings. So it could easily be remote memory exploitation. But more likely social engineering or some other easy path in.

colinbartlett• 12 years ago

Another reminder to be extra careful about checking in AWS credentials into version control. You never know when you might open source that repository and someone can easily yank it from Github.

friendstock• 12 years ago

Actually this is what happened to us... our AWS credentials had accidentally been put into Github for a hackathon project several months ago.

Coincidentally, the incident also occurred around the same time (April 1-2). We were hit with $13,000 worth of EC2 usage before we shut them down and changed our AWS key... We reported to Amazon, and they are working on a refund.

bequanna• 12 years ago

"'They' spin up spot instances which isn't subject to Billing Alerts.

Spot Instances aren't subject to AWS Billing Alerts? Is this common knowledge?

ceejayoz3• 12 years ago

I'm not sure it's true.

josephagoss• 12 years ago

Is it too pedantic to want the tech support to just say they were probably mining cryptocurrency instead of bitcoin? It's most likely the intruder was mining either litecoin or whatever coin is most profitable for the month. I know it's all very much the same but they were almost certainly not mining bitcoin.

A $5,000 AWS instance would mine about $1 worth of bitcoin and would not be worth the time logging into someones account.

matthuggins• 12 years ago

Why the cliffhanger? Why not just finish your thought? As is, this story provides nothing other than showing the Amazon provided some helpful customer service.

abhimir• 12 years ago

A similar thing happened to me when I made public a previously private repo on Amazon and forgot to scrub it for AWS key. Luckily I got an email from Amazon within 24 hours of the instances being started, and as such my bill was just $360, which in any case they waived off. You might want to check your Github repos.

jes51992• 12 years ago

I have 2-factor auth enabled on my AWS login - but am I right in thinking that if someone has my API keys that they don't need the 2nd factor?

prattbhatt2• 12 years ago

Yes, someone with your access and secret keys can spin up instances, create buckets, and do everything else that the stolen keys are authorized for.

ceejayoz3• 12 years ago

Which is why most things should be done with IAM keys specifically locked down to minimal privileges.

Havoc• 12 years ago

The fact that the consultant - unprompted - knows exactly what is going on suggests to me that Amazon needs to fix this. If one customer gets burned then the customer is an idiot. If lots of customers get burned then maybe its not the customers that are the problem.

Jare• 12 years ago

We had an almost identical event last Sunday in one of our dev accounts: multiple high end spot instances in multiple regions with a newly created security group pointing to a suspect IP.

We caught and corrected it quickly, but we still don't know how the keys leaked out - we have chalked it up to lower security practices since it's not a production account and is shared by more people (e.g. no 2-factor on it). We started to investigate, but then Heartbleed happened.

I wish there were more mechanism in AWS to prevent bills from mounting up, but the basic billing alarms worked in this case. I can't imagine how or why spot instances would be excluded from alerts, their cost certainly is included in the estimates that alerts are based on.

$5,000 Security Breach

Related Stories