Ask HN: Is it necessary to backup S3?

https://news.ycombinator.com/item?id=821650

by dawie • 17 years ago

26 29 17 years ago

news.ycombinator.com

I have a consumer Saas Web Application that uses S3 to store user's documents. Do you think it's necessary to backup the S3 data to Rackspace Cloud?

jacquesm5• 17 years ago

How would you feel if one day that data was lost ?

There are already documented cases of Amazon losing data in S3.

The Amazon S3 SLA does not cover data loss at all, only service unavailable situations, which they will cover by crediting you.

http://aws.amazon.com/s3-sla/

http://www.datacenterknowledge.com/archives/2007/10/09/amazo...

And even if all this was not the case you have a responsibility to your customers you can not outsource that responsibility.

garnaat• 17 years ago

I have never seen a documented case of S3 losing data. Please provide references.

I have recommended to some customers that they back up S3 data to the S3 service running in another region. The new export service also provides a way to get physical copies of your data but, depending on how much data you have, it might not be practical.

jacquesm5• 17 years ago

http://developer.amazonwebservices.com/connect/thread.jspa?t...

Do you need more or do you think that is enough like that ?

We've been here before by the way:

http://news.ycombinator.com/item?id=528541

brk• 17 years ago

You can put all the marketing spin on it that you want, but S3 is still just another webapp. Architected by humans, operated by humans.

If you think the humans at Amazon are somehow above making mistakes or bad decisions, then there is no need to backup your data.

If you don't fully trust the humans at Amazon, and what to be able to have access to your data, on your terms, at any time, you should back it up...

PanMan• 17 years ago

Sofar I have never heard about S3 actually losing data (apart from users erasing it). They store everything 3 times. But I would have a look at the cost/MB. Documents are usually small and important (thus, worth to backup). I have read that SmugMug, for example, stores it's files just on S3, with no backup.

jacquesm5• 17 years ago

http://www.google.com/search?q=amazon+s3+dataloss

mechanical_fish• 17 years ago

apart from users erasing it

Very very important caveat, there. If you have three copies of the data, but all of them are in the same S3 account, a prankster who steals your S3 creds can delete all of them in about ten seconds.

Or, if you make a typing mistake, you can do that to yourself. Boy oh boy, will that be an unhappy day.

Diversify, diversify, diversify.

sbhat7• 17 years ago

Very true. One instance I can think of is using the firefox addon S3Fox for managing S3. My co-worker came close to deleting a prod bucket with 80K+ images.

quellhorst• 17 years ago

If you are hacked, someone could get your S3 credentials and delete your content. You should backup.

idlewords2• 17 years ago

It would be wonderful if S3 offered a write-only, no-overwrite mode so you could upload stuff from servers without storing credentials.

zachbeane2• 17 years ago

You can actually kind of do that. There's a system intended to allow random users credential-free upload access to your buckets via POST, but you can enforce policy control like "the target has to have xyz/ as a prefix" and "the size must be fewer than 100000 bytes". Here are the S3 docs for it: http://docs.amazonwebservices.com/AmazonS3/2006-03-01/index....

dagw• 17 years ago

At the end of the day it's an economics question. How much might you lose in a worst case scenario if you failed to recover some vital data? How much extra will it cost to back up your data to a second place? If the answers to those questions are "everything" and "not a lot" then there is no reason not to backup your backups.

As a general rule important data should be backed up to (at least) two separate places, and in this scenario I'd consider S3 to be one place.

LogicHoleFlaw• 17 years ago

AND PRACTICE YOUR RESTORES!

jacquesm5• 17 years ago

That can not be said often enough. We should make a list of IT disaster stories from people that did not take that piece of advice to heart.

oscardelben• 17 years ago

You can refer to this discussion for a tool for s3 backup

http://news.ycombinator.com/item?id=577717

chaosmachine• 17 years ago

It's necessary, because you never know when you'll accidentally corrupt all your data. Even if Amazon never fails, your application still can.

byoung2• 17 years ago

At $0.15/GB it's a no-brainer for the piece of mind. I once heard of a case (not with Amazon) where data was replicated in 3 locations, and data got corrupted in one location and was copied to the others, resulting in corrupted data in all 3.

windsurfer• 17 years ago

Standard response from anyone: Backup everything, just in case

http://tvtropes.org/pmwiki/pmwiki.php/Main/CrazyPrepared

mgorsuch2• 17 years ago

You should certainly backup your data. While the engineers at Amazon are certainly among the smartest, they are not infallible. Things do fail from time to time.

I lost 1TB of data several months ago due to some backend issues with EBS and S3. Fortunately for me, it was just a backup of a backup of a backup. ;-)

mgorsuch2• 17 years ago

It's also worth pointing out that this might have been more of a failure of EBS (elastic block storage, an abstraction over S3) than with native S3 buckets.

Ultimately, my EBS device become unusable by any operating system, and Amazon support stated that the data was lost due to several backend systems failing.

kierank• 17 years ago

Yes, Sometimes the software tools you use can mess up. I lost many production EC2 images because of this unfixed bug in elasticfox:

http://sourceforge.net/tracker/index.php?func=detail&aid...

(basically if you have similarly named images, they'll all get deleted with the "Delete AMI parts and deregister" feature)

aolnerd• 17 years ago

keep multiple snapshots so that you can recover files after you delete them yourself.

#!/bin/sh

AWS_ACCESS_KEY_ID=<>

AWS_SECRET_ACCESS_KEY=<>

BUCKET=<>

DIR=<>

S3SYNC=/usr/bin/s3sync.rb

DAYS_TO_KEEP_BACKUPS=30

# colons in path confuse s3sync

DATE=$(date +%F_%H-%M-%S)

NEWEST=$(/bin/ls -r $DIR | grep -v current | head -1)

# copy the newest copy of files into a new directory, creating hardlinks instead of duplicate files

cp -al $DIR/$NEWEST/ $DIR/$DATE/

# sync the s3 bucket against the new directory

$S3SYNC --recursive --make-dirs --delete --no-md5 -v $BUCKET: $DIR/$DATE/ 2>&1 | grep -v 'Could not change owner'

# update the "current" symlink link

test -e $DIR/current && rm $DIR/current

ln -sf $DIR/$DATE $DIR/current

# remove snapshots older than DAYS_TO_KEEP_BACKUPS days

find $DIR -maxdepth 1 -type d -mtime +$DAYS_TO_KEEP_BACKUPS -print0 | xargs -0 --no-run-if-empty rm -rv

lolido• 17 years ago

fyi, you should take a look at cloudloop (www.cloudloop.com). it has a nice command line interface that lets you sync data across providers (s3, rackspace, nirvanix, azure, etc).

dnsworks• 17 years ago

Remember that Amazon is not infallible. In fact, Amazon has had at least two S3 outages that resulted in permanent data-loss.

Would you put all of your eggs in a bookstore who just recently decided to become an egg storage vendor?

Ask HN: Is it necessary to backup S3?

Related Stories