Ice Box Pro: The best of Dropbox and Amazon Glacier combined

http://www.iceboxpro.com/

by patrickod • 14 years ago

157 128 14 years ago

This sounds great, but since this is using your own AWS account, there is a very serious poison pill; Glacier only lets you retrieve 0.17% of your data for free per day. Beyond that, you get charged based on your peak hour of retrieval.

How much? $7.20 per gigabyte for your highest hour (minus a negligible free allowance if you're using it in this manner).

i.e. the cost to restore m gigabytes over n hours is:

$7.20 * m / n

I'm sure Ice Box Pro will have warnings in place, so nobody will get a $350 bill by accident by restoring a 50 gig account in an hour.

But for disaster recovery, the time will come to decide between a large bill or a very slow retrieval.

Pricing details are here (though quite difficult to follow): http://aws.amazon.com/glacier/faqs/#How_much_data_can_I_retr...

amalag4• 14 years ago

If they built something to warn you when you hit the 0.17% (or whatever the magic number is) it would be great, or just retrieve according to the limit for you.

reitzensteinm4• 14 years ago

To retrieve all your files at the free rate would take 588 days. Glacier is not a product geared towards consumers, and the free tier is basically negligible for this use case.

You're right that an upfront retrieval speed vs cost interface is the correct way to handle it, preferably on first use.

josephlord2• 14 years ago

I haven't looked at the details but is it cheaper to add a massive amount of dummy data for the duration you need to download all your real content?

reitzensteinm4• 14 years ago

No - there is a 3 month minimum for data. You have to add 20 times as much data as you have to download for free over a month, 600 for a day, or 14400 for an hour. Multiplied by 3c a gig.

biturd2• 14 years ago

That seems too costly for even large businesses that don't care. When you lose data, you want it back fast. In this case, the faster you take it back the more it costs.

I wonder why they don't use S3. Is it not reliable enough? The charging infrastructure at least is simpler to figure out.

StavrosK6• 14 years ago

Sure, it's too costly for large businesses, but it's not too costly for me, I just want to restore my 100 GB photo/video collection if all other backups fail, I don't care if it takes two days as long as it's all there.

Spooky23• 14 years ago

There is a huge market for data that needs to be retained indefinitely, or for some long period of time. Oftentimes this is related to compliance. The frequency that the data is accessed is incredibly small -- or even zero.

For example, in several states, all records relating to a minor in state custody, an adoption, or receiving certain nbenefits must be kept for 26 years after the minor turns 18. Today, states are either storing this data on tape, or paying some government contractor to do it for them -- at an expense several times that of Glacier.

Another example is litigation holds. One former employer was forced to hold around a petabyte of data 7 years for a complex civil suit, because... A judge said so. In that case, the high cost of retrieval may be a benefit, because the plaintiff would be footing the bill.

rolleiflex• 14 years ago

I love the idea, but besides the retrieval fees involved that you're not disclosing, I wonder if people will grasp the fact that the files they are putting into the icebox folder are not reachable with opening the same folder and looking into it—instead you need to go to a web site to find them. This is rather disorienting for a non-technical user as the way of putting files in gives a false sense of immediacy to a future retrieval availability.

forensic• 14 years ago

There's this nefarious sense of: Give me your data so I can sell it back to you!

smeagol10• 14 years ago

hey guys,

i built this with a friend and didn't expect it to get posted to HN already. we just got it approved and i accidentally hit our Like button before we were even ready. my friends immediately commented on the story so we just said, what the hey :-)

in any case, it just "launched" today, was only meant for friends to test out, and yes i agree we need to write more copy.

to address some concerns:

- this is really only meant for developers at the moment; i.e. those who actually know their way around AWS. if and when we decide to take payments and provide a service, you can expect a lot more documentation and support. we have no desire to dupe normal folks. we built this for ourselves and really only expected friends to try it out first.

- you don't need to go to your AWS console to get your files. once a file archives, you can click a button to download it back to your Dropbox

- putting in warnings about the quota is a great idea! we'll do that ASAP.

keep the feedback coming, we really appreciate it and wanna build something useful for everyone.

silverlake• 14 years ago

Perhaps you could offer a service where everyone saves their files under your corporate account. When 1 person needs to restore, it will mostly fit within the free download % of the total. So if 10 people store 1 TB each, you can download 500GB/month for free. It's like insurance. We all pay a small amount to cover when one person has a problem.

smeagol10• 14 years ago

we hope to get to that very soon!

most likely, we'll just keep a small data center. something to get around the bandwidth limitation and availability. we would definitely need to charge then in order to cover those costs.

dj2stein9• 14 years ago

What's not very clear to me is if my Glacier files are a subset of my Dropbox files, or a separate datastore.

I only have 2GB of Dropbox space because I never upgraded. But I have about 100GB of archive data I'd probably throw into Amazon Glacier if it were convenient.

Do I need to open a 100GB Dropbox account before using IceBox?

hamburglar• 14 years ago

What's not very clear to me is whether they have any idea what they're doing when it comes to security, or how to respond properly to a legitimate security hole. It's difficult to trust someone who builds a service like this and leaves securing personal keys as "something we'll get around to."

akmiller• 14 years ago

I'd like to see someone provide a service in a similar vein to this but just for photos.

What I'm thinking is a service that uses Glacier to store my photo library and some type of front end service (like dropbox) that keeps a low res version of that photo along with some meta data about it.

My reasoning for this is that we all build up pretty significant photo libraries (mines already over 60GB) and I'm always trying to make sure I have them backed up. I currently use a paid plan at Dropbox so I can put them all up there but it's kind of a waste since I hardly ever pull many of them down again. Every once in awhile I browse through them looking for certain pictures that I might need to get a copy of (which is why I'd need the low res copies easy to access/browse) and then be able to choose which ones to pull down from glacier. The other good thing about a service like that is the need is not typically immediate.

Maybe I'll look into building this since it's something I'd love to have for myself!

amalag4• 14 years ago

That does sound like a good idea! Sounds perfect for S3, can just store the low-res/thumbnails in S3 for browsing via a web page, then retrieve an occasional photo if necessary through glacier.

jmathai7• 14 years ago

Have a look at The OpenPhoto Project.

Hosted version: https://openphoto.me

Project page: http://theopenphotoproject.org

Source: https://github.com/photo

smeagol10• 14 years ago

soon... :-)

akmiller, can we talk offline? i'd love to hear more about your ideas. (we actually have photos in mind.)

just email anything@iceboxpro.com

matthew-wegner2• 14 years ago

I use Backblaze--$50/year--primarily for photos. My photo drive sits at around 900GB currently. It's entirely in Backblaze, in case it ever explodes and local backups fail.

I let it handle other drives, too, although I exclude my media/music/download/etc drive for simplicity.

StavrosK6• 14 years ago

Is it encrypted on the client side? That's pretty much my only requirement now, since I have separated my two use cases (backups and syncing). I use encfs on Dropbox for encrypted syncing, and it works great, but I need backups encrypted on the client, and I haven't found anything great and cheap yet.

budgi3• 14 years ago

Another good option for photo backup is to sign up for a flickr pro account. $45 for 2 years unlimited storage.

savrajsingh• 14 years ago

yes, I think photos / videos that you create are the remaining challenge, haven't found a great solution for backing it all up at a low cost.

jmathai7• 14 years ago

Have a look at The OpenPhoto Project

Hosted version: https://openphoto.me

Project page: http://theopenphotoproject.org

Source: https://github.com/photo

digitaltoad• 14 years ago

I have actually been toying with building out something like this as well. My wife is a photographer and building something around dropbox and glacier only made sense.

jmathai7• 14 years ago

A year ahead of you :). Pull requests!

Hosted version: https://openphoto.me

Project page: http://theopenphotoproject.org

Source: https://github.com/photo

dkokelley3• 14 years ago

I'd be interested in seeing a service like this but for professional photographers. A friend of a friend is a wedding photographer. The work flow for something like this is to have multiple shooters each taking ~25GB worth of photos for one shoot. Then they process the worthy photos, making 3 large files per photo (RAW, .psd, and a final .png). During the turnaround time between wedding and delivery, it would be ideal to keep a copy of each of these files somewhere, but at least keeping the RAW somewhere safe is a minimum requirement. We were trying to figure out a feasible backup solution for the volume of data that such a shoot would require. The time to upload this data is longer than the delivery time for the photos, given the current available upstream bandwidth.

We came to the conclusion that putting files on BluRay disks or hard drives and keeping them physically separate was the best solution for the cost. The problem is that disks take a long time to burn, and external hard drives aren't the best archival medium (not to mention they don't always travel well).

Has anyone here solved the problem of data backups where bandwidth limits make pure online/cloud storage infeasible?

jmathai7• 14 years ago

Have a look at The OpenPhoto Project. It's an OpenPhoto photo platform you can sign up to use or install yourself. Works with S3, Dropbox, Box.net, local filesystem or anything else you want to write a little code for.

Hosted version: https://openphoto.me

Project page: http://theopenphotoproject.org

Source: https://github.com/photo

jmathai7• 14 years ago

Pull requests welcome :)

The OpenPhoto Project is what you're looking for. Open source, installable and offered as a hosted service.

Hosted version: https://openphoto.me

Project page: http://theopenphotoproject.org

Source: https://github.com/photo

RobGR• 14 years ago

Have you heard of https://openphoto.me/ ? It is open source and might be extendable into what you want.

sjm• 14 years ago

I had this _exact_ idea when Glacier came out, and even registered a domain with a splash page. Still thinking about building it but lacking time. I was looking at specifically storing RAW files for photographers and generating low-res versions to show in some sort of library.

dagw• 14 years ago

Had exactly the same idea myself. Whoever makes it work first is probably onto a winner (I probably won't pursue it myself due to lack of time, but I'd love to use such a service).

erenemre5• 14 years ago

Trying it now (800mb is uploading).

This tool is great for me because the data I'm backing up is something I'll need to access maybe twice a year, or less. But I still need to have this data backed up somewhere.

I love how it's as simple as Dropbox. Set it up once, the rest is just drag and drop.

madrona• 14 years ago

I'm not sure you can use competitors' logos like that.

smeagol10• 14 years ago

all Dropbox apps require approval. someone from Dropbox actually signed up, reviewed IceBox, and approved it.

we have no intentions of competing with Dropbox. (like that's even possible.) if we get a complaint, we're more than happy to comply.

we love both Dropbox and Amazon Glacier so we decided to pair them. i'm almost certain Dropbox will eventually add a similar feature soon--we may just be beating them to the punch.

randomchars4• 14 years ago

You might want to make that logo a link to Dropbox since it's required in their guidelines.

https://www.dropbox.com/developers/reference/branding

randomchars4• 14 years ago

They're not competitiors, they're augmenting Dropbox.

From the Dropbox brading guide ( https://www.dropbox.com/developers/reference/branding ):

> you can use the Dropbox logo in marketing material to show your app's Dropbox integration, but it should never be modified and should always refer to our service. If you are using the logo on a website, it should link to https://www.dropbox.com.

So they're allowed to use it, but it must link to the Dropbox website. It doesn't so it's actually violating the guidelines.

smoody• 14 years ago

Is the data stored on Glacier encrypted?

smeagol10• 14 years ago

yes.

http://aws.amazon.com/glacier/faqs/

StavrosK6• 14 years ago

They're encrypted, but nothing's secure if it leaves your computer in plaintext.

RKearney4• 14 years ago

FYI Anyone can see your name, email, and AWS ID/Access Key.

The games on Stripe's CTF were more secure than this site...

EDIT: Looks like it was just patched. Still managed to get a few dozen AWS keys though.

smeagol10• 14 years ago

sorry about that. like i said before, we accidentally "launched" today. be nice?

if anyone's concerned about your AWS key, just destroy your IAM user and create a new one. that's what it was designed for.

dfc4• 14 years ago

You should think about sending an email to all of your users. I realize its a tough call pointing out a problem so early but it might also be a good way to garner user's trust.

RKearney4• 14 years ago

Many of these keys aren't IAM keys, or they were made with full access to the users AWS account.

smeagol10• 14 years ago

in case you have issues with your AWS keys. RKearny's email:

ryan@ryankearney.com

https://secure.gravatar.com/avatar/f7d7b021fb488fe6a67ddb286...

dfc4• 14 years ago

Your edit/addendum would have been way cooler if it read:

"Looks like it was just patched. Still managed to get a few dozen AWS keys though. I sent emails to the affected parties."

EDIT: effected-> affected (I'm not sure what's up with the snarky comment. Do most people think its inappropriate/snarky to urge proper notification?)

StavrosK6• 14 years ago

Grammar mistakes aren't cool.

matthewowen• 14 years ago

That's a moot point, surely? Everyone who used the app should receive an email from the people behind it very, very quickly - they've all been compromised.

The concern isn't just that RKearney has the keys - it is that anyone could have the keys for anyone on the site. Sending an email to the people whose keys he snagged would help them - but the people whose keys he didn't are still vulnerable too.

erenemre5• 14 years ago

Update: I misunderstood, I'm sorry. I wasn't trying to attack but trying to show my concern because I thought he saved some id/keys for himself. Please ignore my comment below.

"Still managed to get a few dozen AWS keys though"

Good for you! What a nice person you are. Please abuse more small projects like this. Even if they say they say it was "was only meant for friends to test out".

Oh I see, you just found a security hole and trying to get some reputation? Cute. Please do it by abusing the small power you found and hurting innocent users. That's really, really nice of you.

"Still managed to get a few dozen AWS keys though"

Wow. Just wow.

You sir, just ruined my night. Thank you.

Ps. I am really concerned about your company and its users. If you can do something like this, I wonder what else you could do (or doing) at your current company. I hope, I'm assuming wrong.

edit: "the" » "your". last paragraph.

amalag4• 14 years ago

Why are you upset with him, at least he informed people. You should be more upset with an insecure app. A less talkative person would just keep it and use the AWS accounts for bitmining.

zacharyvoase2• 14 years ago

It's massively important that people bring security issues into the open. Talking about these things creates pressure on developers to build secure web apps. Not talking about them because people like you get 'upset' about "hurting 2 nerds' inspiration" means we get poor information security policies all over the Internet.

As it stands we don't know whether hostile agents (silently) got copies of the AWS keys of every user on the site. That should be incredibly concerning for you, but apparently it's not.

RKearney4• 14 years ago

Perhaps you're misunderstanding my course of action.

1. I didn't disclose how to do it, merely that it was possible.

2. By "get" I in no way mean harvested. I just manually incremented the ID in the URL by hand in my web browser to see how many users could be affected.

3. Since I never saved any of the information (just viewed the pages) I no longer have it since the flaw was patched.

Nothing malicious was done.

adgar24• 14 years ago

You're so right! Writing secure web apps is for pussies.

jrvarela56• 14 years ago

I'm sorry man, but this kind of mistake is unacceptable even for a prototype. This is basic authentication and access control stuff. You'd at least expect this to be implemented if you're giving them your email, needless to say amazon token.

The problem with this is that people become more skeptical in general about new products/companies, which is bad for the community as a whole.

The issue here is how do you deal with this kind of thing? Assuming it is in fact bad for others who are trying to get people to test their prototypes, how do we help avoid basic pitfalls like this one? This could be a service...

dfc4• 14 years ago

For an open source alternative (and my tool of choice) check out git-annex with s3[1]. joehy has a todo for git-annex and glacier[2] and someone has already submitted the beginings of a patch. When i first heard of Glacier I was excited about the possibilty of using it for a backend to git-annex, but then I read cpercival's discussion of glacier[3,4].

[1] http://git-annex.branchable.com/tips/using_Amazon_S3/

[2] http://git-annex.branchable.com/todo/special_remote_for_amaz...

[3] http://www.daemonology.net/blog/2012-09-04-why-tarsnap-doesn...

[4] http://www.daemonology.net/blog/2012-09-04-thoughts-on-glaci...

EDIT: Removed comment about competition.

StavrosK6• 14 years ago

This isn't a Dropbox competitor, it uses Dropbox. It's just an addon.

callmevlad• 14 years ago

If I understand correctly, Icebox is not a direct competitor to Dropbox. If anything, it enhances the Dropbox experience, so you could argue that it could help Dropbox get more users/customers.

rlpb2• 14 years ago

I've written glacier integration with git-annex and it's fully functional. Details in [2]. The Glacier retrieval pricing uncertainty just means that I store less in Glacier - but for critical data where the paid retrieval cost is worth it, I happily "git annex copy --to=glacier".

biturd2• 14 years ago

Anything that uses Dropbox as the middleman scares me a bit. Dropbox scares me a bit. It silently ignores a lot of meta data , restores files with dates that aren't what the original was ( last I checked ) and has corrupted a few Mac OS X sparse bundles beyond repair.

There's a great app called Arc that handles all of Mac OS X meta data perfectly, works with the relative ease of Dropbox, and backs up to S3 at great savings. De-duplication and other "smarts" are all there.

It's sort of like a Time Machine that you can point to S3.

encoderer• 14 years ago

This is a little OT, but I was thinking about putting together a weekend (read: week) project that would offer a cloud service (details aren't important) but require a user to add an AWS key. The idea is that it would use the free micro instance Amazon gives a user.

The benefit I see is that customers would have total visibility into their own data, and, of course, it's very cheap to run a system like that.

I'd love to hear any opinions anybody might have about it.

jmathai7• 14 years ago

That's a subset of what we're trying to do with The OpenPhoto Project. http://theopenphotoproject.org

alinspired• 14 years ago

Great Idea, although to realy archive terabytes with a relatively small dropbox account you'd have to do it in 10-100 steps.. Recovering it via dropbox would be even more challenging.

oron• 14 years ago

I actually thought about it some time ago and even started working on a prototype, got dropjar.com for that, but after seeing how low companies like backblaze sell unlimited backup for ... I figured it's a hard sell.

Good luck with icebox.

ikken• 14 years ago

Don't give your data to this site until they fix this:

http://blog.ryankearney.com/2012/10/never-give-your-informat...

randomchars4• 14 years ago

It's already fixed.

hnriot• 14 years ago

And even when it's fixed after the way they responded, don't trust them with your data or anything at all.

almost2• 14 years ago

Wow, I've been working on almost the exact same thing! Except I'm charging a yearly fee instead of using the users own AWS account, this makes it easier for users without an AWS account and also means you don't have to worry about the crazy Glacier fee structure.

Try it out if you want, 1GB accounts are free at the moment:

https://www.tidy.io/

randomchars4• 14 years ago

Some feedback about the landing page:

It doesn't show any pricing. I don't want to sign up for something just to see the pricing.

almost2• 14 years ago

Thanks for the feedback!

The landing page not being finished was why I'd been waiting to launch, wish I'd got it done sooner now!

EDIT: Until I sort out the landing page here's the current pricing: 50GB is $25/year, 100GB is $40/year and 500GB is $189

damncabbage• 14 years ago

The security problem that was found (now fixed) is less of an issue than the responses they gave afterwards: http://news.ycombinator.com/item?id=4619652

alanh• 14 years ago

My Dropbox isn’t big enough for the file I would want to send to Glacier.

Does Amazon provide a fairly decent interface? Or, failing that, are there clients that can work with AWS Glacier already, the way some FTP clients can speak to AWS S3?

Ice Box Pro: The best of Dropbox and Amazon Glacier combined

Related Stories