Ask HN: Help me fix your backups

http://spreadsheets.google.com/viewform?formkey=dFpPNl9FYTFlQVFCQXowWVdhRHF2TUE6MQ&ifq

by drewcrawford • 16 years ago

46 40 16 years ago

spreadsheets.google.com

pwim• 16 years ago

As a software development company, all of our code is in a git repository that has a remote on a VPS. Our documents are primarily Google docs. Important docs are checked into git.

Nothing we have locally that is important is not also at some remote location.

drewcrawford3• 16 years ago

Thanks, that is great feedback. Maybe so much is web-based now that backups aren't really relevant.

Is there any pain associated with restoring your dev environment? How long would it take you to get your text editor reinstalled, color scheme set back up, system preferences back, etc.?

dlsspy• 16 years ago

Almost everything I care about is managed via git. That includes editor and shell configs and ~/bin -- restoring my own dev environment is so easy, I just randomly do it on different computers.

Every time I make a change worth saving and want to share it, I push it to gerrit, which means there are now two copies (one on my laptop and one "in the cloud" on my gerrit server).

Every time a change is reviewed, verified, and blessed as part of our code base, the reviewer or verifier pushes a button to submit the change, it's automatically sent up to github, github fires webhooks that automatically replicate the data down to a machine in the office, another copy on our build master (also in the office), and then, shortly after, another copy on every build slave.

Internally, files are stored via nfs or smb onto a solaris box that takes snapshots every 15m. Those snapshots are not stored externally.

The biggest part of our data, though, is made up of vm images. Big files that change a lot. Restoring them hurts, but anything that loads up IO on that box makes vmware think the NFS server isn't responding and it unmounts and remounts it (it even does this when nothing's using it -- though the NFS server itself works fine for other clients during this time).

I wouldn't mind a backup procedure there, but it'd have to not break that.

(sorry to not use your form... the further down I got, the less it applied to me)

surki• 16 years ago

I assume you are using git to manage your home directory. I use gibak[1] for the same(built on top of git). Out of curiosity, what is your setup like?

[1] http://eigenclass.org/hiki/gibak-backup-system-introduction

cperciva3• 16 years ago

  How is your data backed up?
  ...
  [X] I use Tarsnap
  ...

Where's the "I am Tarsnap" option?

Somewhat related: Are the responses to this going to be made public somewhere? I'd love to see what they look like (for obvious reasons) and was thinking just last week that maybe I should post a survey to HN and Reddit.

drewcrawford3• 16 years ago

Lol. I knew it was you as soon as I saw "I use tarsnap" as the only one checked.

I love Tarsnap. I use you for a lot of very-critical-yet-small files.

My personal pain point is that I'd like to back up a few TB of data but can't afford to--it would be nice to back up an entire workstation. I'm interested in seeing if that's a sentiment that others share, or if it's just me.

cperciva3• 16 years ago

My personal pain point is that I'd like to back up a few TB of data but can't afford to--it would be nice to back up an entire workstation.

Why do you have a few TB of data on a workstation? What sort of work are you doing?

uggedal• 16 years ago

Personally I use Puppet to handle configuration of all my systems. All code is on git/hg on codebasehq.com. This means that I don't need to backup entire systems, only databases and other user created content (which I use tarsnap for).

d_r• 16 years ago

Thanks for the recc. Just spent some time reading the Tarsnap website. It sounds to be a very competitive solution, both in terms of operational transparency (detailed, no-gimmick website) and the layer of security.

Hope it's not a stupid question, but how do you deal with the "bus factor" (a.k.a. "truck factor")? In other words, if the service relies on encrypted S3 backups and the founder exits for whatever reason, is there a contingency plan?

(http://en.wikipedia.org/wiki/Bus_factor)

cperciva3• 16 years ago

how do you deal with the "bus factor"?

I don't. If I get hit by a bus, Tarsnap will almost certainly cease to exist.

That said, the service runs itself quite happily on its own (I've gone for months without touching any "live" code) so the odds are very good that Tarsnap users would have plenty of time to download their data between "email goes out announcing bus hit and account balances get refunded" and "service shuts down".

I intend to solve the bus-factor issue once Tarsnap gets larger, but right now it's not financially feasible to bring someone else in.

mstevens• 16 years ago

Just to note, I've had very good results with tarsnap, thanks cperciva!

I'm hoping that the results are made public they'll be anonymised at least somewhat.

Skyline• 16 years ago

Are you going to post the survey results on HN?

I bet a lot of people would find them interesting.

lovskogen4• 16 years ago

Dropbox solved everything backup related for me, and I love it.

edanm2• 16 years ago

Absolutely agree.

Only problem: getting the 100gb option is too expensive for me (for now), which is a shame. Also, I'd love it if they had a much larger size as well, so I can quietly back up everything without worrying about space.

lovskogen4• 16 years ago

Currentlig using 60% of my 51GB. I've got all workrelated files there (I'm a webdesigner), my music and a folder of collected inspiration. No videos.

sreitshamer4• 16 years ago

On the Mac, Dropbox doesn't do so well with restoring metadata: http://www.haystacksoftware.com/blog/2010/06/the-importance-... http://www.haystacksoftware.com/arq/dropbox-backup-bouncer-t...

(Disclaimer -- I write backup software: http://haystacksoftware.com/arq/)

lovskogen4• 16 years ago

What does this mean for regular people?

huhtenberg2• 16 years ago

Arq looks nice. I wasn't able to digest the (S3) pricing table, but all in all it leaves a favourable impression of a well-designed and well thought out program.

jhg2• 16 years ago

Including the needs for backing up confidential data? Hm.

pierrefar• 16 years ago

One thing to point out: backups in the cloud are useless if you don't have internet access when you want access or to restore data.

If you want to fix my backups, make the data readily accessible when there isn't a net connection between me and the service.

One time this usually bites is if you just (re-)installed a new OS and you need the network driver, which is only available online. I always keep a backup of the drivers on my external hard disk for this situation.

Another service I worry about is Gmail and other Google apps. With email you can easily download the several GB of email onto your hard disk, thus keeping a backup for when you or the service are offline.

Dropbox solves this nicely: it's a local folder that's synched to the cloud. If Dropbox is down or I'm offline, I still have access to my data.

larsberg2• 16 years ago

"The biggest problem with backups is"

Missing option is impact on the computer while I'm using it. All of the transparent services (especially Time Machine) that I have experience with are abysmally bad at turning themselves off when the machine is being actively used.

altano• 16 years ago

That's my favorite aspect of Windows Home Server's backup. It does it once a day whenever you're asleep. It'll even wake up your machine, back it up, and them put it back to sleep. And it's smart enough to only do this if you're laptop is plugged into AC and not running on battery. And if any machine isn't backed up in a while, you get a Health warning in your system tray. And this is all pretty configurable.

How's that for low impact.

larsberg2• 16 years ago

That's awesome! I'm very jealous.

apowell• 16 years ago

I got about halfway through the survey before realizing it was about workstation backup, not server backup.

Time Machine + Dropbox makes workstation backup a solved problem.

Slicehost Backup + Tarsnap makes server backup almost a solved problem. And I say almost because I don't like having no choice but to store my full bootable backups with the same provider that hosts my primary server. If I could store bootable weekly images off-site, I'd be golden.

jhg2• 16 years ago

> * Required

> Looks like you have a question or two that still needs to be filled out.

If I want to skip a question, I should be able to do so.

lincolnq• 16 years ago

Agreed. If you give me a survey and mark all the questions "required", if I find anything I don't want to answer, you're out of luck on the whole survey because I will just click back.

dcreemer• 16 years ago

Since I've not seen it mentioned yet: I use CrashPlan to back up all of my personal systems, both to a home server and to an remote peer. It's a bit RAM heavy, but other than that nicely stays out of the way & does it's job. The killer feature is trivial peer-to-peer connectivity, making off site backups easy. I'm not affiliated with the company -- just a happy customer.

huhtenberg2• 16 years ago

What is this "trivial peer-to-peer connectivity" exactly?

Ask HN: Help me fix your backups

Related Stories