Hundreds of exposed Amazon cloud backups found leaking sensitive data

https://techcrunch.com/2019/08/09/aws-ebs-cloud-backups-leak

by ewood • 7 years ago

181 56 7 years ago

d2mw4• 7 years ago

Public EBS snapshots are great, and thankfully a design other clouds didn't copy. I've found all kinds of stuff in there, including a 900GB Oracle backup of a publicly traded manufacturer's accounting system. It doesn't require much imagination to understand how this kind of data could be profited from, given relatively low effort

It seems unlikely a lot of people didn't already know about this, it's hard to miss even if you only spend a few days with the EC2 API, and it's also quite surprising AWS have yet to correct the design. 90% chance it is mostly a UI problem -- there are no warning labels around snapshotting in the EC2 UI

rolltiide• 7 years ago

How do you scour for EBS snapshots and open browsable S3 buckets?

d2mw4• 7 years ago

For EBS, step 1 is reading the docs, step 2 is cutpasting a documentation example.

For S3 I'm not sure how people are building their lists. AFAIK the API provides no enumeration. So this is possibly something coming from web crawl data (e.g. common crawl)

flavor8• 7 years ago

All public EBS snapshots appear in the EC2 > Snapshots section in the AWS UI. Toggle the dropdown in the top right of the table to "Public" and you'll see them. Sort by size and you'll get some interesting looking ones at the top.

It reminds me a bit of old time cdroms.

dragonwriter3• 7 years ago

and it's also quite surprising AWS have yet to correct the design. 90% chance it is mostly a UI problem -- there are no warning labels around snapshotting in the EC2 UI

Snapshots are private by default, you have to actively make them public (impossible if encrypted) or share them (which also requires sharing the associated keys if encrypted.)

Now, AWS hasn't wrapped the extra layer of “by default, reject any setting or policy allowing public or cross-account access unless separate additional default switches have been toggled off” thing to EBS that they have to S3. But people still expose stuff via S3, so that's hardly a panacea. At some point, one has to conclude that customers are responsible, in many cases for giving too many(or just the wrong) people admin access to their accounts.

d2mw4• 7 years ago

What I mean is https://i.ibb.co/P6N35qv/Screenshot-from-2019-08-10-10-27-25... does not make it clear whatsoever that 'public' really means public. Before we get into blaming the customer, there should be a bright red warning label in that dialog. Consider that English may not be the first language of many users reaching that screen

I think they have it for some stuff elsewhere, but it doesn't seem unreasonable to make public snapshots a per-account permission that defaults to disabled, and requires an interactive UI checkbox to enable. Out of the millions of AWS accounts, public snapshots are legitimately useful to maybe 1000 tops

slenk• 7 years ago

Well, for S3 buckets, Amazon has made it very clear when it is public. It also used to be pretty clear.

For EBS - nothing is public by default, so customers have to willingly decide to click buttons to make it public.

By default, if I create a snapshot, it is NOT public...

cfstras• 7 years ago

Oh god, how much I hate articles that don‘t list their sources. Where are the slides from?

The talk description is here: https://www.defcon.org/html/defcon-27/dc-27-speakers.html#Mo...

snazz2• 7 years ago

It’s still true that most security issues are caused by human ineptitude, not clever vulnerability-hunting or burning sophisticated zero-days.

eli3• 7 years ago

I would replace "human ineptitude" with "flawed system design that makes it very easy to make very bad mistakes"

snazz2• 7 years ago

That's a valid way to categorize all memory errors in C.

dragonwriter3• 7 years ago

But is the flawed system design here the automated system at AWS or the human-in-the-loop systems by which companies are providing admin access to IT resources, including AWS accounts?

notabee• 7 years ago

"a common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools" - Douglas Adams

ryacko• 7 years ago

Once I mentioned on a mailing list Chrome’s reaction to mouse driver bug on my computer, it would buffer JavaScript events, and process it even for pages on a different domain.

Later I told the EFF I had a suspicion that iOS didn’t rate limit input events to the lock screen, independently a research found out about it a month later.

Even if there are zero days, I don’t think finding them is a particularly noteworthy or rewarding task.

joncrane2• 7 years ago

I've been working almost exclusively in the AWS space for about 10 years now. Clients anywhere from tiny little three-person consultancies to Fortune 100. Commercial, govcloud, dozens of clients.

Never once have I ever found a use case for making public EBS snapshots.

Who on Earth is thinking that it is a good idea to take an EBS snapshot and make it public?

Note, several of those engagements did involve multiple accounts, and the need to share / copy AMIs and/or snapshots between accounts. But never making them public.

d2mw4• 7 years ago

Who on Earth is thinking that it is a good idea to take an EBS snapshot and make it public?

Non-marketplace AMIs are built on public EBS snapshots, but that's something they should still fix. Marketplace AMIs already handle private snapshots

dboreham• 7 years ago

Laziness in attempting to share data with someone in another org?

"Nope, can't access it" ...

"Nope, still can't access it"...

"My manager is harassing me to get access now"...

"Look, just make it public then change it back after I get it copied"...

heyoni• 7 years ago

goes home completely forgetting to change it back...

lowdose2• 7 years ago

The guy that produces that last line definitely wears a suit.

privateSFacct• 7 years ago

This 100%. I don't do much AWS stuff - but this issue exists outside of AWS as well (inside a corp).

It's why I hate all the password rotation, double VPN stuff - people work around it too much.

All the old staff start having the tech person track their passwords or write them near the computer, all the young staff use whatever new .io domain does X super easily (but less securely) or stick things on thumb driver (no keypad) or use their personal google drive etc.

joncrane2• 7 years ago

Yeah, that's pretty believable. Guess some places have a laxer culture and no automated scans checking for things like that.

Bombthecat• 7 years ago

Too real...

fromthestart• 7 years ago

Who on Earth is thinking that it is a good idea to take an EBS snapshot and make it public?

Maybe they're trying to reproduce functionality of docker? It would actually be extremely useful for research involving modeling/AI because you could trivially reproduce the results by bundling the exact code and data.

Edit: actually maybe I'm confusing EBS snapshots with AMIs...

t34543• 7 years ago

Aren’t public EBS snapshots the underlying mechanism for public AMIs? I’ve ran into complex permissions in a golden image deploy model where the same AMI is used across multiple accounts.

There needs to be controls like S3 where you can explicitly block public data.

AWS IAM kind of sucks.

spydum• 7 years ago

I’m convinced the last few years of ramped up concern about AWS “blast radius” is an admission by AWS professionals that NOBODY gets IAM right.

Exuma• 7 years ago

Is it a default setting that it's public or do you have to go out of the way to do that?

based2• 7 years ago

to synchronize backups to the onprem datacentre. https://angus.readthedocs.io/en/2014/amazon/using-ebs-snapsh...

jedberg• 7 years ago

The creator of the first Ubuntu distros for EC2 wrote about the dangers of public EBS snapshots 10 years ago:

https://alestic.com/2009/09/ec2-public-ebs-danger/

He just got notified by AWS a couple days ago about the public snapshot he mentioned in the article.

But at least AWS is trying to make things better here by proactively checking for public EBS snapshots and notifying people.

AaronFriel• 7 years ago

I just checked an EC2 console and I can see 19,356 snapshots created by other users.

I am so confused.

It would be trivial to make finding a snapshot require knowing a unique ID like an AMI.

And, why do I need to be able to search for 1000s of customers' public snapshots in the EC2 console? What conceivable purpose does that serve except being a giant opsec fail?

jcims2• 7 years ago

That’s per region, too.

judge20202• 7 years ago

looks like some bigger images were of wiki[pedia|media] backups so I guess easy-to-find/use public data?

kpeekhn• 7 years ago

AWS Trusted Advisor has warned of this since 2017:

https://aws.amazon.com/about-aws/whats-new/2017/06/aws-trust...

jcims2• 7 years ago

FWIW same thing is possible with RDS db snapshots and dbcluster snapshots.

chovy• 7 years ago

https://fullstacknews.com/t/devops

chimen• 7 years ago

Gotta appreciate the hijack of the back button on techcrunch. Bounce rate too big?

andrewstuart• 7 years ago

I had a simple glance in the console and there are like 20,000 exposed ebs snapshots - available for anyone to copy and examine - I think that's only for a single region too - switch regions to see more.

Amazon should make an emergency decision to make all these private.

Sure it will break stuff but I'd be disappointed if Amazon left what is in effect a security hole open for the sake of backwards compatibility.

They should also give me a single click link when I sign in to show me all of my public ebs snapshots and throw it hard in my face when I sign in to the console so I simply cannot avoid seeing them all.

I have multiple AWS accounts and I just signed in to try to see if I have any public EBS snapshots and then I realised I would need to search every single region in every single account and then select every snapshot one by one to find out. That's a huge problem. I need a single click to show me every exposed snapshot across every region in my account.

UPDATE:

I can't say for sure if this is 100% right but I think if you sign in to your AWS account, then click on each of these links, you will find if you have public snapshots.

Maybe someone else could confirm if this is correct?

https://us-east-1.console.aws.amazon.com/ec2/v2/home?region=...

https://us-east-2.console.aws.amazon.com/ec2/v2/home?region=...

https://us-west-1.console.aws.amazon.com/ec2/v2/home?region=...

https://us-west-2.console.aws.amazon.com/ec2/v2/home?region=...

https://ca-central-1.console.aws.amazon.com/ec2/v2/home?regi...

https://eu-central-1.console.aws.amazon.com/ec2/v2/home?regi...

https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=...

https://eu-west-2.console.aws.amazon.com/ec2/v2/home?region=...

https://eu-west-3.console.aws.amazon.com/ec2/v2/home?region=...

https://eu-north-1.console.aws.amazon.com/ec2/v2/home?region...

https://ap-east-1.console.aws.amazon.com/ec2/v2/home?region=...