My run-in with unauthorised Litecoin mining on AWS

http://vertis.io/2013/12/16/unauthorised-litecoin-mining.html

by vertis • 13 years ago

223 120 13 years ago

meritt• 13 years ago

Ugh, that sucks. Too late to help you now (but perhaps others) on your billing alerts points: check out http://cloudability.com -- alerts, analytics, prediction, suggestions, etc. Free for the most useful stuff.

sillysaurus2• 13 years ago

Is this illegal? Could he somehow go to some authority?

EDIT: Why is it unlikely the FBI will successfully investigate?

patio11• 13 years ago

Ask a simple question, get a simple answer: it is obviously and unambiguously illegal. You can certainly refer this to the FBI computer crimes folks and your local law enforcement. It is unlikely they will successfully investigate.

vertis12• 13 years ago

Yeah, I'll discuss this with Amazon when they get back to me, but I don't hold much hope for a positive outcome.

rplnt2• 13 years ago

What specifically is illegal about that? Someone gave you credentials to an online service. Now you use those credentials. What law did you brake? You probably broke the ToS/EULA/whatever, but that by itself is not illegal.

(I'm not saying otherwise, I'm asking)

stefan_kendall• 13 years ago

Possibly because it would cost the FBI orders of magnitude more than the damages to investigate.

The damages which amazon will likely refund.

The FBI does not target petty thieves.

richo• 13 years ago

Amusingly, I went to a meetup with some feds the other day.

Their statutory minimum to investigate is 5k, but they suggested it's worth getting in touch with them either way. If nothing else, it'll give them an excuse to say "cyber" a few more times.

jellicle• 13 years ago

Oh, they certainly do, if powerful enough people complain.

http://en.wikipedia.org/wiki/Aaron_Swartz

The question is whether the poster is such a person.

negamax• 13 years ago

I am not sure but FBI can only be involved on financial frauds over 1 million (I vaguely remember this and could be wrong)

tptacek2• 13 years ago

Pretty sure you're wrong about that.

billclerico• 13 years ago

statutory minimum is 5k. we work with federal law enforcement somewhat frequently at wepay to deter fraud, and it appears that the informal threshold to get their interest is about 100k of damages

meowface• 13 years ago

Illegal? Yes. Convictable? Yes.

But will any law enforcement department or agency actually look into it? Highly unlikely. There's so much major scale white collar crime/cybercrime going on daily that it's generally too hard to dedicate resources to smaller cases like this.

Now, if they got evidence the perpetrator was doing this with dozens or hundreds of AWS accounts, that would be another story.

dhughes• 13 years ago

Now I'm curious, how many litecoins would it have generated in two days?

vertis12• 13 years ago

Wish I knew.

dwiel• 13 years ago

Less than the cost of the ec2 instances, though that is still a wide window

gigq3• 13 years ago

One cc2.8xlarge instance will give you 85 kH/s so 20 instances would give 1700 kH/s which only nets about $18 at current market prices / difficulty. So over 2 days he would have made off with a whopping $36.

rkuykendall-com• 13 years ago

Writing a script that generated $38 every time somebody accidentally commits a AWS key sounds like an amazing source of extra cash, depending on your current income.

delinka• 13 years ago

or "...with unauthorized account usage on AWS." I get that the unauthorized use was mining, but the mining operation itself isn't unauthorized by Amazon nor by the creator of the currency.

vertis12• 13 years ago

Sure guess that could have been clearer. Unauthorized account usage used for mining litecoins.

trapexit• 13 years ago

You can (and should) set up an AWS CloudWatch alert on your account that will send you an email or SMS notification when your monthly bill exceeds a set threshold.

vertis12• 13 years ago

Yeah, I've done that now, but it's 'after the horse has bolted'

zwass• 13 years ago

I was surprised how incredibly difficult that is to set up. Eventually I dead-ended following the instructions when CloudWatch told me there were 0 metrics to choose from for monitoring...

sharpy2• 13 years ago

Hi,

There are a couple of things that are easy to overlook when using billing metrics.

1. All billing metrics are stored in us-east-1 even for usages in other regions.

2. If you are using consolidated billing, billing metrics will be published under the linked account, and will only be visible to that account.

Hope that helps.

debaserab2• 13 years ago

I wonder if the author is going to be on the hook for the bill for this.

If he originally received this note from amazon, it makes me also wonder if amazon knew about the fraud while it was happening. I sense that they probably monitor the launch of many of the XXL servers more closely than others.

vertis12• 13 years ago

I have a support ticket open with Amazon about this. I'm hoping not, but I have to accept a certain amount of responsibility for leaking the key in the first place.

res0nat0r• 13 years ago

I highly doubt you will have to pay this large bill, they most likely will reverse the charges and you should be free and clear, especially if you detail what you found and that you also have an email from them that your key was exposed.

I for sure would go to your billing report on the AWS console and setup alerts. You can have AWS notify you automatically if your bill goes above any threshold you want to define.

snowpalmer• 13 years ago

I had the exact same thing happen to me. 2 days of 20 instances running and then amazon called me. I shut it down and revoked permissions immediately.

They did refund the money.

vertis12• 13 years ago

Yeah, I'd say that anyone on Github that has somehow leaked keys is in for a nasty surprise (or has already received one)

icpmacdo• 13 years ago

I just checked my EC2 page and have found $45 worth of charges from instances that I shut down weeks ago. Would it have automatically re ordered some if the bidding price dropped below a threshold? When I looked at my management console there were no EC2 instances running. This has turned out to be a very expensive experiment in LTC mining.

ceejayoz2• 13 years ago

I wonder if the author is going to be on the hook for the bill for this.

Almost certainly not. I've seen AWS reverse charges even when it's unequivocally the user's fault rather than someone hacking in.

As an example, I launched two reserved instances outside of a VPC when I meant to launch inside a VPC. At the time, this couldn't be changed, and the docs were very clear on that. After filing a support ticket, they reversed the reservations... and gave me a $100 AWS credit. For my fuck-up.

kamjam2• 13 years ago

I can't vouch for AWS, since I've never used it, but if the same principles carry across from their online store then one thing I've always found with Amazon is that they have the most amazing customer service.

I had an issue with a 9 month on SD card splitting, not expecting much I emailed them and they immediately sent out a replacement and told me to return the old one. I've had a couple of things go wrong with orders (sometimes just delayed in post) and they've always been very quick and helpful. Not all companies, online ot brick & mortar, understand this well enough.

dignati• 13 years ago

They are quite certainly not monitoring their servers closely, I once had a application running on a small instance costing me more than 30,000$ in just a few weeks because it was doing expensive API calls to AWS. When a growth from <10$ per month to 30,000$ in about 2 weeks isn't concerning them why would 3000$?

umairsiddique• 13 years ago

Exactly same thing happened to me. 20 x xlarge instances raking up a total bill of $1800. I've opened a support case with them.

TwoBit• 13 years ago

I'm confused about this. Somebody got OPs account number and password because it was written down in a file he put on git?

vertis12• 13 years ago

It wasn't my password it was my access token and secret key. But it has almost the same effect.

sheetjs• 13 years ago

Audit code before open sourcing

It's important to remember that open-sourcing is generally one-way: once it's out there, it's impossible to completely eliminate all traces. Always audit code, and if there's even a remote possibility that you'll regret it you should check again

lancefisher• 13 years ago

I'd recommend blowing away your repository's history before open sourcing as well.

DanWaterworth• 13 years ago

If you decide to do this, using git replace so that you and other trusted people can see the history, but others can't is a particularly good idea.

acangiano4• 13 years ago

CPU mining of scrypt-based cryptocurrency is highly inefficient. Let's do some math.

A cc2.8xlarge is reported to mine at 85 kh/s, so 20 of them would give you 1700 kh/s. That's roughly equivalent to a couple of high-end AMD GPUs (say a couple of overclocked 290x). This hashing power gives you a little over 0.5 LTC per day. It mined for two days, so it gained a little over 1 LTC. Let's call it $40.

That's right, the idiot behind this cost the OP $3000+ for $40 profit. A smarter criminal would have spawn GPU instances on EC2.

vertis12• 13 years ago

sigh I was afraid it was THAT stupid.

fat0wl• 13 years ago

it almost always is, with the crypto-currency crew. unbridled greed is the driving force of the movement now

gigq3• 13 years ago

Makes you wonder why the criminal didn't launch g2.2xlarge instances which would get 185 kH/s per instance. In fact for awhile there litecoin mining was profitable at spot prices.

http://www.completefusion.com/profitable-litecoin-mining-on-...

acangiano4• 13 years ago

My guess would be ignorance. I don't think the criminal behind this knows what he is doing.

A smarter criminal would have opted for g2.2xlarge instances as well as mining for a currently more profitable coin. Granted he'd need to be careful not to leave a trail, he could essentially trade these coins for LTC and still obtain more litecoins.

benatkin• 13 years ago

I used it to mine dogecoin using those same instructions. I either had to use an older nvidia driver version or an older version of CUDAminer, though. So it wasn't that simple to set up once CUDAminer got the update that broke the newer NVidia software. This happened near the first of the month. http://doges.org/index.php?topic=43.0

rwg• 13 years ago

Criminals don't care how much money they're costing their victims, only how much money they'll make from their crime. This is no different than someone smashing your car's window ($100+) and destroying its dashboard ($1000+) to remove a stereo head unit that they'll flip for $50.

acangiano4• 13 years ago

Correct, but for the same crime and effort, (so just as easy) the criminal could have made a bigger profit. They took the stereo and left behind the laptop. Criminals however are not always the brightest bulbs.

muyuu• 13 years ago

Not only that, LTC has been one of the lesser profitable scrypt-coins to mine for a while.

See https://www.multipool.us/ for reference. Although it fluctuates a lot, LTC has been around 4th-5th in profitability among scrypt coins. Often 5th. And wait for Doge to feature in these pools :-)

acangiano4• 13 years ago

Actually, it's worse than that. Check http://coinwarz.com and http://coinchoose.com. Multipool is limited to the coins they mine. If you check out the more general list at the sites I mentioned, LTC is not even in the top 25.

brownbat• 13 years ago

That sounds about on par with copper theft, it's often ripped out of installed equipment like AC units or transformers that require total replacement.

yayitswei• 13 years ago

I was trying to launch a GPU instance the other day for (scrypt-based) dogecoin mining, but I couldn't find it in the list. Perhaps they are sold out?

withjive• 13 years ago

You need to make sure you have selected a compatible (cluster something, and green perhaps) AMI first.

Then the GPU instances will be available in the list. Ran into the same thing myself ;)

billjive• 13 years ago

How did Amazon detect your key in the wild? Or did they notice based on usage patterns/activity in your instances?

earless12• 13 years ago

All AWS keys I've seen start with 'AKIA'. I am assuming that they have bots that search Github and other search engines for access keys. At that point it is easy for them to tie them back to an account and notify the user.

vertis12• 13 years ago

They must only have started doing that recently. This project has been out in the wild for at least a year.

vertis12• 13 years ago

Email linked to my GitHub profile, so I would say by searching. But that's an assumption.

proksoup2• 13 years ago

I bet you 20 bucks they searched for your key after they investigated the sudden spike in your charges :D

devonbleak• 13 years ago

FYI all AWS keys start with AKIA - makes it easy to search for 'em.

awhitty• 13 years ago

Shoot, as someone who made the same mistake of leaving my AWS keys in an open source project, I think I narrowly dodged a bullet. I didn't realize this risk was so high. Thanks for this post!

earless12• 13 years ago

I think smarter usage of IAM roles would have also helped here. Keys created strictly for S3 access should not have the ability to launch new instances and so on. Limiting keys to their specific purpose is a good security practice even for dev environments.

sounds• 13 years ago

Role-based access control. :)

Though Amazon still has services that "don't support IAM" ... I'm looking squarely at Payments. That particular master key always makes me nervous.

miles9322• 13 years ago

AWS IAM supports the ability to permit or prevent specific types of instances from being started by a given key; if folks are worried about a key being used to start G2 or CG1 or any other specific instance, take a look at the instructions here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-polic...

colbyaley• 13 years ago

I suggest the OP check out Cloudability[1], which provides realtime cost management for AWS and other cloud providers. We help over 10,000 customers make sure this doesn't happen to them. (disclosure: I work there)

[1] https://cloudability.com/

lambda• 13 years ago

Another good habit to be in is never checking any kind of credentials into source control; even if it's some private personal project, just don't be tempted to check in your credentials to source control, because at some point you may find some portion of that that's useful that you import into a public project, accidentally preserving full history.

Sorry to the OP, hope that Amazon reverses those charges once you tell them what happened.

xur17• 13 years ago

What's the best way to handle api keys without storing them in a repository? A separate configuration file?

I've saved api keys to a repository, but only ones with no payment information attached and read only access. Mainly because I couldn't think of a better way at the time.

grannyg00se• 13 years ago

Usally in environment variables rather than a config file.

for example https://devcenter.heroku.com/articles/config-vars

yeukhon2• 13 years ago

We know we shouldn't be committing password or personal key. But shit happens and it does happen very frequently, even top notched people do.

What we need is not to say "don't it", because no shit we shouldn't be doing that. instead we need defense mechanism. It would be helpful and interesting if git or hg has a plugin that detects when some credentials is leaking through and warn users "hey you better check this shit out" before doing a real commit.

The other thing is "don't commit key into a private repository". Don't chef and puppet users usually do that? How are people backing up their keys?

dwaltrip• 13 years ago

For rails, the Figaro gem is really helpful for managing environment variables, which is a good place to store credentials (the config file for Figaro is then added to .gitignore)

miles9322• 13 years ago

Here's a handy GREP to find AccessKey/SecretKey pairs: grep -RP '(?<![A-Z0-9])[A-Z0-9]{20}(?![A-Z0-9])' * grep -RP '(?<![A-Za-z0-9/+=])[A-Za-z0-9/+=]{40}(?![A-Za-z0-9/+=])' *

source: http://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-...

steveklabnik• 13 years ago

This would be neat for a pre-commit hook.

badmadrad• 13 years ago

"Having a poke around confirmed what I had already guessed. The unauthorised user had been mining litecoin with the mining pool pool-x.eu."

Hmmm..you already guessed someone hacked your account to mine litecoin? Astroturfing much? That's the last thing I would have guessed. I would have thought someone was using it as some crazy web server or mail server to generate spam or phony websites for bogus ad clicks.

vertis12• 13 years ago

shrug maybe I was wrong to make that kind of assumption.

It was probably the type and quantity of instance that tipped me off a little, having read about people trying to mine with EC2 again.

omarchowdhury• 13 years ago

So are you liable?

tomphoolery• 13 years ago

I know there are a few code-quality bots on Github, but is there any service that you can install as a webhook which automatically checks for things like Amazon key pairs (which, IIRC, always start with "AKIA", at least the API keys anyway)?

tlrobinson• 13 years ago

I'm also curious if there's some utility or at least a list of regular expressions that can be used to scan for a number of credentials, not just AWS.

mnml_• 13 years ago

Amazon will refund you if you explain your situation.

davidjgraph• 13 years ago

This is rough luck, but getting specific servers hacked is more commonplace. In the AWS billing console [0] there is an "alert" option. It walks you through setting up the various types of alarms.

If you're hacked the most likely problem you'll get is a spike in data transfer costs. You can up the alarms to, for example, email you if the bandwidth usage goes above x (cost) over y time period.

I had a perl DOS bot get into a server, took about 2 hours to trigger the alarm. Shame I was fast asleep at the time, but the idea was there...

[0] https://console.aws.amazon.com/billing/home

judah• 13 years ago

The author suggested enabling billing alerts. For those running on Azure, billing alerts are currently in preview mode, and can be enabled via https://account.windowsazure.com/PreviewFeatures

smiro2000• 13 years ago

sorry dude :(

My run-in with unauthorised Litecoin mining on AWS

Related Stories