Code Spaces data and backups deleted by hackers

http://www.codespaces.com/?hackernews

by leejacobson • 12 years ago

119 78 12 years ago

pjc50• 12 years ago

That's extraordinary .. but it does drive home the distinction between "backup" and "online second copy of your data". Proper backups should be offline when not in use.

__david__2• 12 years ago

Proper backups should be offline when not in use.

I'm not sure I'd go that far, but I'd say that backups on the same service as the data aren't a good idea. It would smart of use a different cloud provider for the backups and (depending on the size of the operation), even a disk sitting in someone's house somewhere, just in case.

twistedpair3• 12 years ago

That's why I have a Mini on the bookshelf that SSH's into an EC2 instance (IP restricted) during a 1 hour window each night when the router is set to open and copies that day's PGP'd backups over. Then it goes back to sleep to wake on schedule the following night. Not the same as a tape in a vault, but it's my last resort.

gnu8• 12 years ago

Why is it possible to destroy an entire enterprise by compromising an Amazon account? Where the fuck is their 2FA? What about a cooling off period before committing changes like deleting all of your storage? Amazon's infrastructure seems to be built without essential safeguards.

lelf• 12 years ago

Amazon's infrastructure seems to be built without essential safeguards

Their amazon's infrastructure

theatrus2• 12 years ago

Amazon AWS has 2FA, clearly not used (unless it actually was an inside job).

As for the cooling off period, I'm not sure. Perhaps you can get the contents back, but it may be cost prohibitive.

TkTech2• 12 years ago

Amazon has extremely tight security, including 2fa, fine-grained IAM permissions, instance security groups, VPC, and more.

The fact that Code Space's didn't bother to use them is their own problem, not a failing on Amazon's side.

Additionally, storing all of your backups with the same service as your production environment was outright moronic.

stevekemp• 12 years ago

Amazon does have a remarkably fine-grained control mechanism - but you need to use it.

For example I never publish my Route53 (DNS hosting service) keys, but even if they were leaked the account is only setup on the Amazon side to work from a single source IP.

You can restrict permissions significantly, so again in my case I've got a user configured who can only add/delete DNS records - but cannot create a new zone, or delete other zones. Not ideal, since "remove all records" is almost the same as "delete zone" in practice, but I'm not worried that unrelated zones on that account will be broken if I do lose my keys.

aaroncampbell• 12 years ago

Saw this earlier today. That's rough. Obviously they had some problems with their architecture (backups shouldn’t be able to be deleted like that), but it's still pretty messed up. I hope they catch the guy. I won't help Code Spaces, but whoever it was deserves to be caught.

informatimago• 12 years ago

s/I/It/

qworty• 12 years ago

It hope they catch the guy?

bagosm• 12 years ago

All I can say is this:

If you can delete it with a single control panel, it doesn't count as an offsite back. Fire the devops

dijit• 12 years ago

DevOps means different things to different people.

most people use the term to mean 'a coder who can configure apache'

this whole incident reeks of a lack of OpSec and general Ops knowledge.

Lack of offsite backups is one thing, but if you're using cloud infrastructure you never place all eggs in one basket.

your instances have to be disposable and reproducible, at the first hint of a break in my default action is to torch and rebuild.

robryk• 12 years ago

your instances have to be disposable and reproducible, at the first hint of a break in my default action is to torch and rebuild.

Or rather shutdown, preserve, start replacements. IMHO your automation should be able to deal with the old preserved instances being left alone.

maxlybbert• 12 years ago

Not to be too flippant, but the company's closing shop. So, yeah, the DevOps are fired, along with everybody else.

As for the rest of us: AWS is a great one-stop shop. Unfortunately, using just AWS puts you in the "all the eggs in one basket" scenario that we were warned against as children.

judk• 12 years ago

Two-factor authentication is a second basket.

Sending a copy to Glacier is a second basket.

Does Amazon not have 30-day undelete for bulk storage? Seems crazy.

toomuchtodo4• 12 years ago

DevOps here.

There are some things you aren't going to expect (compromise of your AWS console). This could have been solved by having MFA enabled, as well as having the app push backups in realtime, versioned with delete protection, to S3 buckets under the control of another account (write access, but no delete access).

Show of hands how many people here are doing it this way.

mentat• 12 years ago

Seriously, if your root account and all full admin accounts aren't using MFA you're just asking for it. Also if you're not using purpose specific access keys, you're just asking for it. If the first thing you do isn't calling AWS support, wow...

Globz• 12 years ago

How the hell would you let the faith of your entire enterprise data in the hand of Amazon? Offsite/offline backup are mandatory especially when you deal with this kind of data.

twistedpair3• 12 years ago

Real companies don't. I know a major video provider in Cambridge that replicates everything to S3 and Google Cloud storage for this very reason. Now, neither has ever gone down for them, but if you've got to stay up, you have a resilient architecture.

invertedohm• 12 years ago

Leaving aside the obviously deficient sysadmin work here: the timeline of the story doesn't add up. I can only hope this explanation is not accurate.

You find notes in your AWS control panel saying you should contact some Hotmail address. OK. So the first thing you do is reach out to that address and take the time to communicate intricate extortion details? Only after that you think maybe it's a good idea to start changing passwords, and right then the other party takes action and deletes all the things?

If that's what actually happened then I'm afraid something like this was bound to happen sooner or later.

Kequc• 12 years ago

I feel that a lot of people here are being unnecessarily harsh. It was all a bit of a silly mistake in hindsight but Code Spaces was a very new service I'm not even certain it had secured funding yet.

The timeline looks to me like email address shows up. Check email address. Email address contains extortion details. Try to change passwords. Hacker gets in again and again while deleting stuff. Cannot get rid of hacker. Do not have money. Within 12 hours everything is gone.

tylerlh• 12 years ago

They have existed for a number of years[0]. Even still, being a new service is no excuse for such poor handling of OpSec.

[0] https://twitter.com/CodeSpaces/status/265757401368637440

wasamasa• 12 years ago

Does anyone know of noteworthy projects that were hosted there?

TkTech2• 12 years ago

Not really, archive.org shows Oracle, Macy's, and some other random corps using it for private hosting as of a few months ago.

eterm• 12 years ago

Without naming names I know of at least one company who were using it for their main svn repo. I am hoping they have come out of this relatively unscathed. I am not with them anymore and last I heard they were switching to git so perhaps they have managed to dodge this bullet.

kevingadd• 12 years ago

Couple projects I've contracted on were using codespaces for their private SVN repos. I used git-svn, so that's a mercy...

mkal_tsr• 12 years ago

They ran a code hosting service and they had no offsite backups?

Wow, just wow.

spacemanmatt• 12 years ago

I am in awe, myself.

nevster• 12 years ago

From their homepage "We offer Rock Solid, Secure .. hosting". Perhaps not so rock solid...

Arnor• 12 years ago

From the Amazon RDS documentation:

When the backup retention changes to a non-zero value, the first backup occurs immediately. Changing the backup retention period to 0 turns off automatic backups for the DB instance, and deletes all existing automated backups for the instance.

http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overvi...

jacquesm9• 12 years ago

and deletes all existing automated backups for the instance.

That could do with some sort of gracetime, both against shoot-in-your-foot scenarios and bad guys.

Rapzid• 12 years ago

Unfortunately one of the major backup related pain points with RDS currently is you have no way to interact with the automated backups aside from restoring an instance to a point in time. They are also deleted when the instance is deleted. Manual snapshots aren't. Manual snapshots can be used to launch and new instance, or they can be moved to a different region. You MUST do your own backups of the databases if you want to get them "offsite" or off account. Like I said, PITA right now and from what I here a common request on RDS is being able to get at the automated backups and snapshots so you can move them offsite.

codeddesign• 12 years ago

while I feel bad for thoer customers, I have no remorse for the creators. Stupid and mediocre managment of their code...not to mention...how does a code hosting service not understand the difference between a backup and an off-site backup? how could they delete an account and not notice (or bother to check) the other created accounts? No notification on account access creation? So mamy mediocre mistakes...

randunel• 12 years ago

Am I the only one feeling cheated when people post referral / tracking links?

taspeotis3• 12 years ago

The ?hackernews in the URL is to work around HN's duplicate URL detection. It could have been ?lulz and had the same effect.

opendais3• 12 years ago

Seems to be down so cache: http://webcache.googleusercontent.com/search?q=cache:qpjW4k2...

"We are experiencing massive demand on our support capacity, we are going to get to everyone it will just take time. Code Spaces : Is Down!

Dear Customers,

On Tuesday the 17th of June 2014 we received a well orchestrated DDOS against our servers, this happens quite often and we normally overcome them in a way that is transparent to the Code Spaces community. On this occasion however the DDOS was just the start.

An unauthorised person who at this point who is still unknown (All we can say is that we have no reason to think its anyone who is or was employed with Code Spaces) had gained access to our Amazon EC2 control panel and had left a number of messages for us to contact them using a hotmail address

Reaching out to the address started a chain of events that revolved arount the person trying to extort a large fee in order to resolve the DDOS.

Upon realisation that somebody had access to our control panel we started to investigate how access had been gained and what access that person had to the data in our systems, it became clear that so far no machine access had been achieved due to the intruder not having our Private Keys.

At this point we took action to take control back of our panel by changing passwords, however the intruder had prepared for this and had already created a number of backup logins to the panel and upon seeing us make the attempted recovery of the account he proceeded to randomly delete artifacts from the panel. We finally managed to get our panel access back but not before he had removed all EBS snapshots, S3 buckets, all AMI's, some EBS instances and several machine instances.

In summary, most of our data, backups, machine configurations and offsite backups were either partially or completely deleted.

This took place over a 12 hour period which I have condensed into this very brief explanation, which I will elaborate on more once we have managed our customers needs.

Data Status

All svn repositories that had the following url structure have been deleted from our live EBS's and all backups and snapshots have been deleted: https://[ACCOUNT].codesapces.com/svn/[REPONAME]

All Svn repositoies using the following url format are still available for export but all backups and snapshots have been deleted: https://svn.codespaces.com/[ACCOUNT]/[REPONAME]

All Git repositories are available for export but all backups and snapshots have been deleted

All Code Spaces machines have been deleted except some old svn nodes and one git node.

All EBS volumes containing database files have been deleted as have all snapshots and backups.

Code Spaces Status

Code Spaces will not be able to operate beyond this point, the cost of resolving this issue to date and the expected cost of refunding customers who have been left without the service they paid for will put Code Spaces in a irreversible position both financially and in terms of on going credibility.

As such at this point in time we have no alternative but to cease trading and concentrate on supporting our affected customers in exporting any remaining data they have left with us.

All that we can say at this point is how sorry we are to both our customers and to the people who make a living at Code Spaces for the chain of events that lead us here.

In order to get any remaining data exported please email us at support[at]codespaces.com with your account url and we will endeavour to process the request as soon as possible.

On behalf of everyone at Code Spaces, please accept our sincere apologies for the inconvenience this has caused to you, and ask for your understanding during this time! We hope that one day we will be able to and reinstate the service and credibility that Code Spaces once had!"

lugg• 12 years ago

Thanks for the cache just getting to this.

Edited because someone didn't like my original tone. Was a bit rushed to be honest.

Few things seem off about this:

- Offsite backups were also deleted, I don't think they had offsite backups, or at least backups you could legitimately say were "off site."

- EC2 has two factor auth, why you wouldn't use this for your business I don't know. [1]

- Corresponding with extortionist is a really dumb move. It would be better time spent locking things down - contacting amazon directly to get an account lock / getting your ducks in a row.

[1] http://aws.amazon.com/iam/details/mfa/

jacquesm9• 12 years ago

There is something about this whole story that feels weird, I can't name it but it is as if this isn't the whole story.

jacquesm9• 12 years ago

Is it possible to 'lock' your amazon control panel to a specific set of IP addresses?

In the payment world it is a fairly common feature to use a block-by-default strategy for such crucial controls.

Hosting your project management and your sources with other companies always did feel strange to me. I can see how it works well for open source project and git (after all, every repo is a complete copy) but to host the master of a subversion repo 'in the cloud' and to have your project management in the cloud feels uneasy to me.

If this or something like it would happen to github and all the github issues would be lost that would be a fairly major disaster.

You never know how solid the infrastructure and solutions chosen behind a nice looking web front are until it goes down, and this one went down hard.

Condolences to the users of codespaces.com, they are the ones who lost most in all this.

From codespaces backup page:

" Real Time Backups Backup

All your Source Code is backed up in real time, so that in the unlikely event of a system break down your data is safe.

Not only do we Back Up your data we also give you access to our backup up data via the Code Spaces Admin console so you can keep your own copies.

Whenever you make a change we make a backup."

So much for that I guess, if it is spinning and online it is not a backup.

diwo• 12 years ago

Amazon's web services and their control panels support multi-factor authentication methods and customisable permissions models. For details see: http://aws.amazon.com/iam/

toomuchtodo4• 12 years ago

There is no reason people shouldn't be using multi-factor auth with their AWS master account. Please do so!

(ops guy)

toomuchtodo4• 12 years ago

So much for that I guess, if it is spinning and online it is not a backup.

True, although with S3, you can make backups very difficult to remove:

http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMFADelet...

"If a bucket's versioning configuration is MFA Delete enabled, the bucket owner must include the x-amz-mfa request header in requests to permanently delete an object version or change the versioning state of the bucket. The header's value is the concatenation of your authentication device's serial number, a space, and the authentication code displayed on it. If you do not include this request header, the request fails."

jacquesm9• 12 years ago

Yes, so all it takes then if for amazon to either be buggy or to fail for some reason. You can't really outsource responsibility for stuff like backups. They should be under your control and yours alone, and they should not live in the same DC or with the same provider where you store the rest of your data.

And when you've made your backup you take it offline so that no matter what you can get back in business.

franklyjeremy• 12 years ago

Either that or the "hacker" pressed the "Close Account" button in the EC2 panel.

taspeotis3• 12 years ago

Another point for distributed version control systems. If the server hosting my repository exploded I'd have the entire repo on my computer.

Why hosted subversion is a thing, I don't know. It's a horrendous experience once you introduce any latency between the client and server.

rahimnathwani• 12 years ago

offsite backups != offline backups

twistedpair3• 12 years ago

Reminds me of LiveJournal's dead harddrive induced closure. When will startups learn how it's done?

gaadd33• 12 years ago

When did that happen? I was under the impression that LiveJournal was still around however it was mostly popular in Russia. Any links to that?

csixty4• 12 years ago

Definitely still around. My wife uses it daily and is part of several communities.

kjs3• 12 years ago

Apparently they won't. I can't tell you the number of times I hear "everyone who's important in the startup world says 'shut up and ship' and we'll think about security & BC/DR later", where "later" usually ends up being "after a breach". If they survive.

msantos• 12 years ago

When will these cloud people learn that storing a so called backup in the same place or under the same domain or root account as the original data is a copy not an actual backup.

_cipher_• 12 years ago

Cloud kicks ass. Totally secure along with the high-level fellowship of paladins^Wadmins.

Code Spaces data and backups deleted by hackers

Related Stories