Why Tarsnap doesn't use Glacier

http://www.daemonology.net/blog/2012-09-04-why-tarsnap-doesnt-use-glacier.html

by cperciva • 14 years ago

210 55 14 years ago

michaelt3• 14 years ago

I'm confused about the reason a block can't exist in both S3 and Glacier at the same time, if the deduplication code decides that block is needed in a new archive.

Why couldn't you simply have a rule that each file is either S3 or Glacier, and S3 lists-of-blocks can only reference other S3 blocks, while Glacier lists of blocks can only reference other Glacier blocks?

In the worst case, where every block was in both archives, this would only increase costs by 10% if Glacier costs a tenth what S3 costs.

cperciva8• 14 years ago

What happens when you decide that you don't want that new archive to be in S3 any more and tell Tarsnap to migrate it over to Glacier?

michaelt3• 14 years ago

To S3 it's like you just deleted the file; to Glacier it's like you just created the file.

cperciva8• 14 years ago

Another reason, which I decided deserved to be written up as a separate blog post, is the surprising nature of the "data retrieval" component of Glacier's pricing model: http://news.ycombinator.com/item?id=4475319

tedunangst3• 14 years ago

I was thinking about one time, single shot backups, but encrypted wtih tarsnap. No dedup. After all, storage is cheap and this is supposed to be zuper worst case scenario stuff, so maybe I don't want to go to the trouble of reconstructing the backup. I want to know it's there. But my interest is only theoretical.

aristidb• 14 years ago

If you have a command line tool to access Glacier, I think what you can do is make an archive (with tar), encrypt it with gpg, and just use the command line tool to upload the encrypted archive. So you don't need tarsnap for that, but if you want fancy deduplicating incremental encrypted backup, that's where tarsnap shines.

tedunangst3• 14 years ago

Yes, but I'm already using tarsnap. Running it with "--backup-to-glacier" is easier than rolling my own. I don't want two passwords, two super duper secret keys, two online services I have to pay, ....

cperciva8• 14 years ago

Sounds like setting up a Glaciated Tarsnap machine would do what you want, then? Or maybe I'm misunderstanding what you're saying.

tedunangst3• 14 years ago

More or less. But I may also want to use the same machine key to perform smaller incremental backups (smaller data set), and I definitely don't want to have to print out two different keys to store in my safe. Just thinking here.

ChuckMcM3• 14 years ago

While I think this is a completely reasonable thing (not storing tarsnap data in Glacier), I'm not sure if the reasoning holds beyond, "For the way we designed tarsnap it doesn't make sense."

When you look at how GFS is implemented or Bigtable, or Blekko's NoSQL data store, there are layers, with the meta data as a pretty separable layer from the data. In all of the cases for these large stores that separation was put in to facilitate putting the meta data into lower latency storage than the 'bulk' data which facilitates fast access.

As cperciva relates, tarsnap spends a lot of time de-duplicating data (or writing only one copy of a block with identical contents) which is great for reducing your overall storage footprint which lowers your costs. But if the cost of storage is much much smaller than the retrieval cost, then your design methodology would be different.

So I would not be surprised if there was a way to build a function equivalent product to tarsnap that had lower storage costs if the bulk data was in glacier and the index data was in S3, or if the index data was designed such that a document recovery was exactly two retrievals (a catalog, and then the data). And of course such a system would not de-duplicate as that would result in potentially more retrievals.

It seems that if the price delta ratio between Glacier and S3 was higher than the de-dupe ratio, then Glacier would 'win' with no de-dupe. Else de-dupe would still win. Thoughts?

pdovy• 14 years ago

Based on the article it seems like the motivation for the deduplication is as much bandwidth conservation as reduction of storage costs. What you're proposing would make sense if storage costs are your only concern, but it shuts out bandwidth limited users.

ChuckMcM3• 14 years ago

An excellent point. So capturing the bandwidth savings and the storage cost savings suggests an interesting mid-tier between longer term storage and shorter term.

JoachimSchipper• 14 years ago

This would indeed be true for a sufficiently large price advantage of Glacier over S3, but note that tarsnap typically saves ~99.99% by de-duplicating my data (daily backups, usually only a few e-mail and code files have changed.) I imagine that the typical usage pattern may be a bit less extreme, but the price differential would have to be huge to be worth it.

wmf• 14 years ago

When calculating dedupe ratios you should be careful what baseline you use. Performing full backups every day is generally not realistic. I would compare dedupe-based systems against classical incremental backups.

ghshephard5• 14 years ago

I don't know if only sending the changes is considered "De-Duping" (though it is related) - That's more equivalent to doing differential (or block level differential) backups.

De-Duping is if you add the same file (or portions of a file) in 10 different directories - the data is only stored once in backups.

Dropbox takes it a step further - if 1,000 people store the xcode DMG on their dropbox store, only one copy is stored in Dropbox.

jerf• 14 years ago

"But if the cost of storage is much much smaller than the retrieval cost, then your design methodology would be different."

If you read between the lines a bit on how the second-to-last paragraph would manifest in code, I think that's what he's already getting at. A Glaiciated machine would probably not get global de-duping, but merely a machine-level de-duping, if even that. (The retrieval cost structure is quite bizarre.)

caladri• 14 years ago

Well, it's possible to make the deduplication work with Glacier, though. Colin's right that you want to phase versions of the whole tar into slower and cheaper storage, but the technical problem of whether blocks are used in newer versions doesn't actually seem to be too much of a problem. You can compute it in a batch for extant tars, and track it for new ones. (And you can keep blocks-of-references always in S3 rather than Glacier, say.) The problem he identifies as most troublesome is what if you see a case where you would want to deduplicate, but the similar block is in Glacier, and so would be a bottleneck for the whole deduplication process. In this case, you could always treat it as a non-match, right? And optionally have some way to track it and then when it gets moved to Glacier either determine whether it was actually a match all along, or just have some duplicate data.

Of course, I don't know how tarsnap names its blocks or stores them, so I don't know how feasible it is to have two blocks with the same name because they had the same hash but there was a byte-by-byte mismatch, or if that's even a problem.

I mean, blocks that have been moved to Glacier because there are no references to them from indices on S3 can be assumed to be less likely to show up in new archives. It's a trade-off, but my experience with deduplication is that it's often not much of a trade-off to get rid of old things even though the magical thinker in me is tempted to think "but what if that chunk happens to show up again somewhere else!?"

Locke1689• 14 years ago

I think the most important part is separating the de-dup metadata from the data. I assume the de-dup is done via either digest or HMAC -- if you keep the digests (and the blocks of digests, et al.) and pointers to the data location in Glacier in S3, couldn't you do the de-dup using S3 but actual data storage using Glacier?

ghshephard5• 14 years ago

This is exactly what I've been wondering - obviously cperciva is 10x (100x?) smarter in this area than I ever will be, but I kept thinking, "Just keep the digest in online storage, and the actual data in (offline?) Glacier. You don't need to access Glacier during backups to calculate digests, because you already have the digests online."

I read through his posting carefully, to see if he would capture that possibility - but didn't really see it there.

cperciva8• 14 years ago

The deduplication is done on the client side; and yes, it is done by comparing HMACs of blocks.

cperciva8• 14 years ago

I'm not sure if the reasoning holds beyond, "For the way we designed tarsnap it doesn't make sense."

I agree -- that's exactly what I was saying. The title of the blog post was "Why Tarsnap doesn't use Glacier", not "Why you shouldn't use Glacier".

I think Glacier is a great service, just not a particularly good fit to Tarsnap.

Spooky23• 14 years ago

Glacier is simply meant to solve a different problem than S3.

I will be using Glacier for storage of litigation hold data -- potentially many terabytes of stuff that is being held because a plaintiff asked to hold "everything" and a judge agreed.

So I'll pay to store this stuff for a few years, and in the off-chance that I need to retrieve it, the other party will pay at least half. (A powerful incentive for them to not ask for the data to be retrieved in the first place!)

Basically, you want the data that your sending there to be contiguous -- no fancy de-duping or incrementalism. If you are going to use it for backup, just tar up everything and send it up in blocks that make sense from a recovery cost POV.

danielweber• 14 years ago

I like articles like this.

1. Explains what the new whiz-bang technology is.

2. Explains what Tarsnap is.

3. Provides technical explanation that lists both the upsides of the new technology along with the downsides.

cperciva8• 14 years ago

Thanks! I like blog posts like this too -- they appeal to my didactic instincts.

mapgrep• 14 years ago

Agreed. Too often we hear of new tech via "this is the best thing ever" commentary, followed a few months later by the inevitable "actually that thing sucks" commentary.

Much better is "here are the trade offs, based on actual usage" commentary like this. I wish there was more stuff like this during the first wave of NoSQL mania.

nchuhoai• 14 years ago

Does anyone know of products that already incoorperate Glacier. Basically, I am looking for a solution to back up my photo library. It's one big file with a "Write Once, Read hopefully never" Access Policy. I guess I could go and push it to Glacier by myself, but I'd rather have a library/product do it for me

ghshephard5• 14 years ago

Glacier was rolled out on August 21st - I don't think anyone has rolled out code to support it in a consumer friendly fashion - but drop a line to http://www.haystacksoftware.com/arq/ and encourage them to give you the ability to target Glacier for your Archives, and I bet you'll see something fairly soon.

Yrlec• 14 years ago

My startup Degoo (http://degoo.com) is working on it. We're building a P2P Backup system were we will be using Glacier as a fall-back whenever we don't have enough reliable peers. We're planning on launching our private beta in the coming months.

ocharles• 14 years ago

Great write up, I've been curious about it, but I'm not particularly bothered. I think what I'm going to end up doing is to basically tar ~/media, pipe it through GPG, and pipe it through some upload-to-glacier script. Maybe automate that once a month or something and then have tarsnap for much smaller/rapidly changing stuff. As it stands now I have 100gb in ~/media that's costing me about $1/day, so it'd be nice to reduce that.

RexRollman2• 14 years ago

Cperciva,

Could a similar system to Tarsnap exist for Windows? I'm not asking you to implement it, I'm just curious.

magic_haze• 14 years ago

Tarsnap runs without any problems under cygwin/windows right now. It is very easy to set up: the Tarsnap downloads page lists all the dependencies - you just need to make sure to include them in the Cygwin setup and then run his makefile. Frankly, his setup instructions are far more easier to follow than certain other gui-based installs that look shiny but induce far more stress trying to guess how to opt-out of the complementary crapware (e.g., Skype, Adobe Reader, utorrent...)

Also, as he notes in his install page, it _definitely_ is worth checking out the source code: one of the cleanest C source I've ever seen, and very educational.

RexRollman2• 14 years ago

Thank you for the info!!

unreal37• 14 years ago

I think the best use-case for Glacier is large files that you would absolutely hate to lose. The thing that people cry over when their laptops get stolen.

For home use, all the family photos and videos. An archive of your emails (outlook.pst) because a lot of important data is stored in there. All your taxes and accounting data from years past. Bulky stuff that takes up space on your home system, but isn't used daily.

In business, many companies use "Iron Mountain" to archive their paperwork. Old invoices, reports, things that were important in the past and may someday be needed. That's what Glacier is for.

Glacier is archiving, not backup. You might want to take advantage of cheap storage by keeping your backups there, but that's a different issue.

fusiongyro3• 14 years ago

I don't get the impression Glacier is right for any home user. The pricing scheme is just too weird, and there are weird delays and time limits involved. I believe you have 24 hours to retrieve your data before incurring more fees when you do a retrieval from Glacier--but if I store a backup of 150 GB of photos and music and fetch the whole thing, I'm not going to be able to get that from Amazon to my home computer in one day. Avoiding that problem will necessitate other expenditures.

In order to avoid the retrieval costs you'll have to limit retrievals to a small fraction of what you have stored. In my example, if I lose all my photos and music I'm going to want to restore the whole thing. Ignoring the transfer problem above, you're looking at paying for transferring 95% of the archive, 142.5 GB. I find Glacier's pricing model so difficult to comprehend I couldn't even guess at what that would cost, but Colin's math shows that what looks like it should cost $0.02 (retrieving 2 GB once at $0.01 per GB) winds up costing $3.60 (peak rate, percentage of archive fetched, etc.), so I wouldn't hold out a lot of hope for our use case of fetching the entire 150 GB archive.

As you say, this is certainly something businesses that need to archive lots of stuff may be able to use (if they can navigate the pricing structure) I just don't see a home user getting anything out of it but frustration and a confusing bill.

warp• 14 years ago

Indeed the pricing is difficult to estimate. I followed along with [1] in a spreadsheet to get a feel for it for my usage.

In my case I am looking at making a full backup of a 1TB usb disk. Doing a full restore on a 20Mbit/s (= 9 GB/hour) would probably take about 5 days. In that case I'll be paying $65 in retrieval fees for the full restore. (I'm hoping that includes bandwidth costs, that isn't clear to me from the FAQ).

$65 for a full restore seems reasonable for something I expect to never need to do.

[1] http://aws.amazon.com/glacier/faqs/#How_will_I_be_charged_wh...

michaelt3• 14 years ago

Assume I store 150 gigabytes of family photos. I pay $1.50 a month ($18 a year) for storage. Assume I've uploaded the files in 150 * 1 gigabyte archives.

I decide to retrieve it all. The 5% (6 gigabyte) retrieval allowance is negligible. The data transfer out fee at $0.120 per gigabyte will cost $18.

If I retrieve 150 * 1 gigabyte chunks every hour retrieval will take 1 hour; the peak hourly retrieval will be 150 gigabytes; the data rate will be 341 Mbps; and the retrieval fee will be 150720$0.01 = $1,080

If I retrieve 7 * 1 gigabyte chunks every hour retrieval will take 150/7~=22 hours; the data rate will be 15 Mbps; the peak hourly retrieval will be 7 gigabytes; and the retrieval fee will be 7 * 720 * $0.01 = $50.40

If I retrieve 1 * 1 gigabyte chunks every hour retrieval will take ~7 days; the data rate will be 2.2 Mbps; the peak hourly retrieval will be 1 gigabytes; and the retrieval fee will be 1 * 720 * $0.01 = $7.20

If I share an account with 20 other people with the same amount of data stored, the 5% allowance would be enough for my entire download; I could retrieve as quickly as I liked without incurring a retrieval fee. I would still pay the $18 data transfer out fee.

TLDR: Retrieval isn't as cheap as storage, but if you lost all your family photos, you'd probably be willing to pay it.

genwin2• 14 years ago

For home use, it seems easier to put or more extra external drives (encrypted with, say, TrueCrypt) at some trusted locations, like the houses of friends.

cynicalkane• 14 years ago

I'd bet that this use case is pretty common: I have only a small amount of data (a few hundred MB in my case) that I deem day-to-day critical, but hundreds of GB that I don't want to lose. My backup solution is effective but primitive: parchives on two offline hard drives, one of which is at my parents' house seven hours away. Backing up takes little effort but it's easy to forget, especially at the "remote site".

It's really annoying that there's lots of easy backup solutions for online data, but nothing for cheap backup of large amounts of data that can afford to be offline for a while. Glacier is the perfect solution but I dread having to figure out whatever I have to do--divide things into a 180-part archive and download it over a month?--to get data back. The first glaciated backup solution on the market that I think I can trust will get my dollars almost immediately.

j_s• 14 years ago

Some people might not be aware that CrashPlan supports backing up between your own PCs (eg. to your parents) for free:

http://www.crashplan.com/consumer/crashplan.html

genwin2• 14 years ago

Your remote site method is my method too. I'm not worried about city-wide catastrophes, so I keep the spare drives closer for easier updating.

veyron• 14 years ago

I've got Terabytes of data that need to be archived and possibly recalled later (trading logs and market data preserved in case of an audit). I'm not at the scale of Glacier. If TarSnap had a readonly vault (files in said vault would never change) then it would be able to distinguish the files and split between the Glacier and S3 offerings.

ghshephard5• 14 years ago

Drop a note to http://www.haystacksoftware.com/arq/ - I'll wager they will have an option to create "Glaciered Archives" of folders within a few months. The developer there has already expressed some interest, and suggested that Arq would lend itself to pointing at a folder and storing it off to a Glacier Archive (with all the caveats around 4 Hour delay, 24 Hour of Availability, Costs associated with restores, etc...)

sreitshamer• 14 years ago

[I'm the developer] I'm hoping storing the metadata in S3 and the actual file data in Glacier would work well. I'm still looking into it.

[edited for clarity]

Why Tarsnap doesn't use Glacier

Related Stories