How I attacked myself using Google and I ramped up a $1000 bandwidth bill

http://www.behind-the-enemy-lines.com/2012/04/google-attack-how-i-self-attacked.html

by Panos • 14 years ago

775 142 14 years ago

ltcoleman2• 14 years ago

This was an extremely interesting article. I hope Google rectifies this type of behavior. This is actually pretty scary.

typpo• 14 years ago

This isn't Google's fault. I could get a couple machines and do the same thing if I had a list of 250 gigs of files. And I wouldn't be limited to once every hour.

bri3d• 14 years ago

But those machines wouldn't be Google's, and you would probably either be committing a crime (building an exploited botnet) or using someone's money (be it your employer, university, or yourself) to run them.

This is pretty clearly a different league of attack: rather than attacking systems or spending money, you'd be exploiting a Google feature to use Google's resource for free to incur a giant amount of data transfer.

igorsyl• 14 years ago

Try http://cloudability.com to keep track of your cloud costs. Note: I am not affiliated with them.

ltcoleman2• 14 years ago

Thanks for the link!!

TwoBit• 14 years ago

That such a simple thing on your part could result in such an extreme and expensive result implies something is wrong with the design of the system you're using.

ma2rten• 14 years ago

I was going to post exactly the same thing. It's very lenient of him to blame himself instead of Google for this.

Yarnage3• 14 years ago

I'm pretty surprised Google didn't have the client download the images instead. Wouldn't that be a better solution or am I missing something here?

Pretty interesting though and if this becomes a big enough story you can bet Google will be changing something; the last thing they need is someone using Google Docs to DOS websites.

recursive• 14 years ago

Perhaps the server needs to know the image dimensions for page layout.

Yarnage3• 14 years ago

They use quite a bit of HTML 5 so they should be able to do this client-side I would imagine. But I dunno.

bibinou• 14 years ago

Google Docs is in HTTPS so they need to proxy the assets like Github does : https://github.com/blog/743-sidejack-prevention-phase-3-ssl-...

Yarnage3• 14 years ago

Interesting. I never thought of that. It seems almost silly that you're bringing insecure content over a secure channel like that but then again it is only images.

jonknee4• 14 years ago

The function lets you specify height/width, passing it through makes some sense. It's also usually (though not in this case) to not hot link--a document can be viewed by thousands of people. There are some privacy issues too, viewing a Google Doc could have your browser making requests to unknown servers that could return whatever they want.

timwang• 14 years ago

very interesting findings, good read.

gojomo2• 14 years ago

But, why re-downloading every hour?

Does merely having the spreadsheet passively open in a browser trigger that, or was some other process re-loading the spreadsheet every hour? (If the former, I wouldn't be as forgiving of Google. I understand the desire not to cache possibly-private data, but proper URL design and conditional GETs should be able to prevent the entire download on an automatic hourly schedule. And even if the latter – the author had chosen to reload the spreadsheet each hour – I'd want Google's design to allow browser-side caching to work for such embedded and/or generated images.)

ardillamorris• 14 years ago

There must me something else to it because if the spreadsheet is sitting passively open there would be no need for Google to go an pre-fetch again.

jonknee4• 14 years ago

About an hour is the standard time for external data in Google Spreadsheet to be refreshed. I've come across this with JSON data.

gojomo2• 14 years ago

Even if noone has it open in a browser? Or they do have it open, but haven't interacted with it?

In either case, this seems odd, unless the URL is especially noted as 'volatile', and/or there are other parts of the spreadsheet that might trigger conditional calculations/notifications based on that URL's contents. (And don't S3 resources have last-modified-dates or etags for conditional GETs?)

spullara• 14 years ago

I've seen this behavior with google gadgets as well, even with no user interaction or even having the page open.

RobertKohr• 14 years ago

Can you limit bandwidth with AWS?

Also, why would the spreadsheet be calling these images every hour. Did you have the spreadsheet open? Does google do this call even when no one is viewing the spreadsheet?

justincormack2• 14 years ago

You can put a robots.txt in the bucket.

Panos5• 14 years ago

Which will be ignored by Feedfetcher :-)

Plus you cannot put a robots.txt at s3.amazonaws.com so if the url is accessed through the https://s3.amazonaws.com/.... url, the robots.txt will not work.

ceejayoz2• 14 years ago

No, you can't. It'd have to be at the root of http://s3.amazonaws.com/.

This is mentioned specifically in the article, in fact.

simonbrown• 14 years ago

The article states that (1) you can't and (2) the bot ignores it.

eli2• 14 years ago

According to the article, that would not have helped; feedfetcher is meant to be manually triggered and thus does not obey robots.txt

ceejayoz2• 14 years ago

> Can you limit bandwidth with AWS?

Not on S3, no.

SagelyGuru• 14 years ago

Scary indeed. Many thanks for the warning. I was contemplating starting to use Amazon Cloud and some Google tools but definitely will not touch any of it now. Who knows how many other traps like this are laying in wait for the unweary?

All this automation is very well but this illustrates the dangers of running stuff on other's machines of unknown complexity and out of your own control, while having to pay for whatever may happen. Not for me, thanks.

dasil003• 14 years ago

I don't think the OA would want you to take this as FUD. These are incredibly powerful tools that give you more control than the alternatives (AWS anyway), and yes unintended consequences are possible, but this really is a freak occurrence.

K2h• 14 years ago

I loved your reference to the huge Russion bomb Tsar Bomba. I for one had never heard of it, and it made a great metaphor.

[1] http://en.wikipedia.org/wiki/Tsar_Bomba

DanBC• 14 years ago

Check out the Nuclear Effects Calculator; You can see what effect various historical bombs would have. (They include Tsar Bomba.)

(http://nuclearsecrecy.com/blog/2012/02/03/presenting-nukemap...)

(http://news.ycombinator.com/item?id=3624714)

tocomment• 14 years ago

This really underscores Amazon's glaring omission of a billing cutoff on Amazon web services. How hard would it be for them to let me say, cut off my services at $100/month?

This is the main reason I'd never use AWS to host anything public.

eli2• 14 years ago

I guess that would be a nice feature to offer as an option, but most people would not want their web service cut off if they get a spike in traffic.

adgar• 14 years ago

> most people would not want their web service cut off if they get a spike in traffic.

Only if they are sure enough about their finances and their web service to be confident that the spike will be profitable.

scott_w• 14 years ago

That's where the "option" part comes in. People who aren't worried about the cost simply don't turn it on.

Those of us who can't afford an extra £1000/month (or more if it's a dedicated attack) would love to be able to just turn it off when a threshold is hit.

Users may lose access, but at least you don't go bankrupt in the process.

tybris• 14 years ago

End of the article: "PS: Amazon was nice enough to refund the bandwidth charges, as they considered this activity accidental and not intentional. Thanks TK!"

pavel_lishin• 14 years ago

Would they have refunded him if someone did this to him maliciously, as he points out is possible at the end of his post?

jonknee4• 14 years ago

Or perhaps an email saying "We're past the limit you set, approve the overage in the next X hours by clicking this link". That way you don't cut off accidentally during a spike you want, but know about it.

rramsden• 14 years ago

quite sad that Amazon still hasn't implemented bill capping yet. A lot of users want this... its been on their TODO list since 2006 https://forums.aws.amazon.com/thread.jspa?threadID=58127&...

dredmorbius3• 14 years ago

The argument in favor of bill-capping: it puts the onus on Amazon (or whatever other service provider) to have the tools to identify what's causing cap-busting behavior, as they're going to eat the costs.

I killed my EVDO wireless service based on a similar experience with my cell provider, after requesting multiple times that they provide me capped service. It simply wasn't worth the downside cost risk.

raverbashing• 14 years ago

A cutoff is good, but a triggered alarm would go a long way.

For example in Linode you can configure trigger conditions for CPU (triggers in x% cpu for y time) and Network usage (there are other conditions as well)

Very useful for flagging suspicious activity (or just plain accidents)

guelo• 14 years ago

Amazon did send him a warning email.

mryan2• 14 years ago

You can set this up with CloudWatch - add your EC2 costs as a metric, then you can create alarms based on your desired threshold (although I'm not sure how frequently this data is updated).

dredmorbius3• 14 years ago

It's not that simple.

First, the problem isn't inherent to virtual hosting services, you could just as easily get hit by this on a bare-metal site, though the interplay of S3 and Google Docs is an added dimension.

Cutting off all services opens the door for a class of DoS attacks. Simply direct enough traffic at a single account's assets, and you'll knock them offline for a given billing cycle. If the attack is cheap to launch (botnet, URL referral network, etc.) it's a cheap attack. Different entities would have different cut-off and degradation policies.

Better would be to identify the parameters of a specific anomalous traffic pattern, but this can be hard.

A more general solution is to set asset (server side) and client (remote side) caps in tiers. You'd want generous (but not unlimited) rates for legitimate crawlers, your own infrastructure, and major clients. The rest of the Net generally gets a lower service level. Such rules are not trivial to set up, and assistance through AWS or other cloud hosting providers would be very useful.

mikeryan• 14 years ago

Cutting off all services opens the door for a class of DoS attacks. Simply direct enough traffic at a single account's assets, and you'll knock them offline for a given billing cycle.

I dont' see this as a real issue (the potential DOS is there, but not a real problem with having a bandwidth metric kill switch). I'd assume this would be a configurable setting and would have to be enabled by choice. In the author's case I'm sure he'd prefer to have is service cut off prior to running up a 1k bill. If one of my test instances started bleeding bandwidth I'd prefer for it to just get killed then to rack up an ungainly bill. If its your production service then don't configure a cutoff.

nknight2• 14 years ago

> you'll knock them offline for a given billing cycle

Er... Because caps can never be raised mid-cycle?

seanlinmt• 14 years ago

Or if you have the connections, contact one of the Vetted and get the attackers at the origin. Well that's according to http://www.wired.com/politics/security/magazine/15-09/ff_est...

jeffbarr• 14 years ago

We are already testing a billing charges monitor that works through CloudWatch. Here's some more info:

http://blog.bitnami.org/2011/12/monitor-your-estimated-aws-c...

sgrytoyr• 14 years ago

Looks great! Any chance that this will become available to accounts without Premium support in the future?

richardlblair• 14 years ago

It's really awesome that Amazon was reasonable and refunded the charges because they were accidental. I mean, technically it was still your fault, so it would have been easy for them to be jerks about it.

sohn• 14 years ago

They wouldn't be jerks if they asked for the charges; you still generated the traffic and they had to pay for it.

EvilTerran2• 14 years ago

Just as "legal" does not mean "ethical", "contractually permitted" does not mean "not a jerk move".

gscott• 14 years ago

It's possible Google and Amazon have a peering arrangement so they may have paid little to nothing for the traffic.

rachelbythebay• 14 years ago

Feedfetcher strikes again, huh? I never did find out why they were so interested in one of my images.

http://rachelbythebay.com/w/2011/10/27/wtfgoog/

RandallBrown2• 14 years ago

If I wanted to launch an attack on something like Instagram all I would need to do is put a bunch of images (hosted on instagram) into a Google Spreadsheet? Then the google crawler will come through and download them all once an hour?

BryanB55• 14 years ago

That's what I'm wondering too, I don't see how this would work on a normal website since a normal website probably wouldn't be whitelisted by Google like AWS is... What am I missing here?

RandallBrown2• 14 years ago

I don't think you'd be able to take down a website with this strategy. The OPs site never went down, it just cost him a lot of money. Even pretty small webservers should be able to serve up static image files pretty easily.

But you could do a denial of service by making the service too expensive and all of your work would be hidden behind the anonymity of the google bot.

OzzyB• 14 years ago

+1 For Amazon for kindly reimbursing the overage charge.

-1 For Google for creating what is the biggest threat to content providers by enabling easy-to-use DDOS attacks across the entire interwebs.

</hyperbole>

Seriously, is this what we have to look forward to when Google Spreadsheets, and God knows what else, become ever-more popular?

Think about all the additional onerus costs that would be incurred by content providers as more and more Google Spreadsheet users hotlink images, mp3s, videos...

This has to be a bad design decision by Google, there's no need to redownload assets by-the-hour, on-the-hour, regardless of whether the user's spreadsheet is open or not.

Is it time to go back to the days of putting your web assets behind $HTTP_REFERER?

hm8• 14 years ago

Like the author notes, I don't think the problem was with Google. It was no fault of theirs. The reason: It's a fine line between maintaining privacy policies and managing such events. If google were storing/caching these links, there would have been an outcry from those worried about user privacy and stuff.

About the by-the-hour downloads, well there again is a trade-off between providing data quickly and doing a lazy evaluation.

I think, as the author notes, it was just an unfortunate event that was a consequence of good design decisions gone bad circumstantially and wreaking havoc for the author.

It was certainly nice of Amazon to have made that refund. The resources (read bandwidth) were used after all.

ricardobeat• 14 years ago

I don't understand the privacy concern - a publicly accessible URL doesn't offer any privacy. It's the same as a transparent proxy.

mooism2• 14 years ago

Google are presumably storing/caching these photos for an hour before expiring and redownloading them.

I don't see why Google don't redownload them using If-Modified-Since, and reusing the original downloaded photos in the case that the origin server says they've not changed.

tripzilch3• 14 years ago

> If google were storing/caching these links, there would have been an outcry from those worried about user privacy and stuff.

Where would we be without Google taking such a noble and strongly principled stance on user privacy, applied equally and consistently, striving to avoid legal responsibility, even when it costs unrelated parties thousands of dollars.

sp332• 14 years ago

It's not a DDOS becuase it's not distributed and there was no denial of service. And it doesn't work "across the entire interwebs" because the Google bots are rate-limited against most websites. There are some whitelisted sites, like S3, that are not rate-limited. S3 did not go down in this "attack". In fact the app didn't even go down. So the decision not to rate-limit against S3 was sound, since there was no DDOS.

eridius• 14 years ago

Are you saying Google isn't distributed?

In any case, it may not be a denial-of-service, but if you can find someone with an S3 bucket with large files you could maliciously cause them to rack up a huge bandwidth bill using this mechanism. I guess you could say it's a DDOM (Distributed Denial of Money).

goblin892• 14 years ago

Ironically, it might be DoS in the sense that it drives app maintenance costs so high that the owner could decide to, naturally, deny the service (shut down).

Otherwise, maybe it's the new type of flood attack, cost-of-service (CoS), applicable against those who use ‘invincible’ cloud infrastructure such as Amazon's.

rapind• 14 years ago

Come on, really? This is clearly an edge case they simply didn't see and will probably solve once the right people know about it. The sky may be falling, but it's not because of Google Spreadsheets...

whackberry• 14 years ago

> -1 For Google for creating what is the biggest threat to content providers by enabling easy-to-use DDOS attacks across the entire interwebs.

Wanna help make the web better? Stop using Google and use a competitor like DuckDuckGo.com. Google has too much power, too much control.

damoncali• 14 years ago

Reminds me of the email cannon we built on Gmail, not exactly on purpose. We needed a full Gmail account to do some testing on our company's email backup system. At the time that meant 8GB of email. And you couldn't just send a bunch of huge attachments, as it had to be like regular email. So we took a gmail account and signed it up for dozens of active linux related email lists (because they were easy to find).

The result was an email account that got an email message every second or so in a variety of languages.

We quickly realized that you could have some fun by forwarding that address to someone else's email account.

Fast forward a few months, and Gmail smartly requires a confirmation before allowing you to forward.

creamyhorror• 14 years ago

Once in middle school in the late '90s I created an email loop between two free email accounts that forwarded an extra copy of the email every iteration. It was an experiment out of curiosity and mischief. I don't think the server actually went down when I started the loop (at least, I can't remember if it did) - since the mailbox would have failed up and messages fail to be delivered.

I think I was hoping to produce an "email cannon" similar to yours - an email spammer to be directed at whoever I liked.

joelhaasnoot• 14 years ago

Ah yes, the days of computer labs in high school and figuring out that outlook lets you copy and paste emails in the Outbox. Just get a big enough collection and keep copying and pasting. Easy recipe for getting hundreds - thousands of emails.

rorrr• 14 years ago

Kudos to Amazon for refunding the guy. I don't think any other hosting company would just take the bullet for the customer.

anthemcg• 14 years ago

Quite an interesting article. I didn't even think of it as a potential DDoS tool until he mentioned it. It makes sense but it seems like Google should have a built a fail-safe for something like that situation.

lubujackson2• 14 years ago

Here's the big money question, since I haven't used S3 - do you have any throttling control? Can you simply block the feedfetcher bot, or block repeated hits like this at all? It seems like S3 is a nightmare $$$ hole if there aren't some really robust tools to manage this sort of problem.

drivebyacct22• 14 years ago

>"But how come did Google download the images again and again?"

"But how come did" indeed.

drivebyacct22• 14 years ago

Communication skills matter, sorry.

arunoda2• 14 years ago

But you get the idea what he trying to say no? We all here are not speaking English as the mother language. Please excuse :)

vladd• 14 years ago

I cannot help notice that Hetzner offers 5000 GB/month AND a full dedicated server, for 39 EUR (51 USD) [1], so his traffic would have cost him at that rate a total of 100 USD if he were to use a dedicated server instead of Amazon.

(before mentioning Amazon's scalability, consider that Hacker News is ran on a single dedicated server, and the moral of the story seems to be how not to scale especially when you don't want to)

[1] http://www.hetzner.de/en/hosting/produkte_rootserver/x3

robryan• 14 years ago

Of course if you want to do away with any kind of built in redundancy and take over the sysadmin duties involved you can get the raw space and transfer cheaper.

krelian2• 14 years ago

I have a VPS on buyvm which includes 2TB/month for $6. Each extra TB is another $2.50. This is not a special case, you can other providers with very good pricing for data transfer. I don't understand how the difference in bandwidth pricing can be so large. I'm also certain that Amazon doesn't pay as much for bandwidth as the guy from buyvm.

negamax• 14 years ago

For me biggest reason to use AWS is ease of use and APIs. I have used dedicated servers before but with AWS I don't even think about servers. Application is somewhere sitting in the cloud and it will scale, if required or I will get alert if something is not right.

Also, majority of people will not be needing more than a single EC2 instance which costs ~$20 a month, lesser for Linux. Amazon reviews and lowers prices periodically.

devs1010• 14 years ago

"What I find fascinating in this setting is that Google becomes such a powerful weapon due to a series of perfectly legitimate design decisions."

I call in to question that these are "perfectly legitimate design decisions", basically, if google thinks that the data is private or too sensitive to cache, then it shouldn't be this easy to have it automatically keep hitting a site like this. Google should have realized the potential for abuse here. I'm guessing it truly is an oversight on their part as I can't imagine they would want to waste all this bandwidth either, however its something they should figure out a solution for.

cabalamat• 14 years ago

I have a VPS. When i read stories like this, I think there isn't any point in going over to GAE or AWS. But lots of people do use these services, so what am I missing?

stretchwithme• 14 years ago

kudos to Amazon for running time backward and letting you pull your foot out of the way.

I can't help but think that if benign decisions lead to disasters like this in the cloud, how much destruction could robots wreak in the future due to similar benign choices?

Panos5• 14 years ago

That was exactly what I was thinking after reflecting on this issue...

mquander• 14 years ago

"To err is human, but to really foul things up requires a computer." - Bill Vaughan

Freaky2• 14 years ago

Amusing thing about that page - it's full of '\$100', and uses Javascript to strip the \'s out, replacing them with empty <span> elements. Not sure I really want to know why...

Panos5• 14 years ago

Mathjax :-)

Freaky2• 14 years ago

Ah, that's a good excuse. Thanks :)

jyothi• 14 years ago

Sometimes a 509 Bandwidth limit exceeded helps!

AWS can actually do a setting for max bandwidth per hour or so & alert early if there is suspicious activity.

otterley• 14 years ago

I'm curious to know whether the objects were stored with Cache-Control: or Expires: headers. Does having such headers make a difference?

Clearly the client's not presenting If-Modified-Since: pragmas as I believe S3 honors those.

arunoda2• 14 years ago

This is the same kind of attack. I've demonstrated here. But for users of a popular analytic service. http://news.ycombinator.com/item?id=3873774

_k• 14 years ago

He's getting 100+ requests per second.

You could rate limit the IPs. The question is how many IPs is Feedfetcher using ?

matthieupiguet• 14 years ago

Seeing how popular the story is, amazon could not have made a better $1000 PR campaign than being classy and reimbursing him!

nfonrose• 14 years ago

You can use http://cloudcost.teevity.com to protect yourself from such situations (disclaimer : I'm the founder and CEO)

How I attacked myself using Google and I ramped up a $1000 bandwidth bill

Related Stories