"It's the (Backblaze) restores that fail."

http://www.seankilleen.com/2014/01/its-backblaze-restores-that-fail.html

by SeanKilleen • 13 years ago

31 54 13 years ago

SeanKilleen20• 13 years ago

I wanted to post this here to bring attention to what happens when trusting a cloud backup provider with your data, as well as to spur Backblaze to hopefully pay more attention to this part of their business model.

Again, while I say it in the post, I think it's important to state again that the CEO has been great in allowing a dialog via Twitter. This is not a post meant to diss Backblaze as much as it is to share concerns and put the thought out there for consideration.

As the CEO Mr. Budman notes in the Storify link, they no doubt do hundreds of successful restores every day. But when I have two working systems fail for Backblaze restores but nothing else, it's still a cause of concern for me personally.

voltagex_• 13 years ago

I wonder whether it's a header or an index on the compressed file, or the data itself that's damaged. Still not something you want from your backups.

SeanKilleen20• 13 years ago

Correct. They appear to basically zip the data using 7-zip and then transfer it, but I'm not sure what sort of hashing/checking they do to make sure the file makes it to me successfully.

Very possible that the next time I download the 6GB file it will work. But given how long it takes to do that, 1) I'd like to know it will work and 2) tougher for me to consider utilizing the free option of doing this for all ~400 GB of data.

howellnick3• 13 years ago

Coming from working in IT support, it's frustrating to me that you've made a big public ordeal over something that, firstly was fixed by rebooting your PC, and secondly you haven't tried downloading the file a second time. I'm not necessarily defending Backblaze here, but posting this when it doesn't work quite right the first time seems a bit harsh. I feel like we need more information before we all grab our pitchforks.

goldenkey• 13 years ago

Please don't singly use cloud backups. Your first line should be a local backup solution like a Time Capsule or other NAS device. Blind trust in a black box backup is not prudent.

SeanKilleen20• 13 years ago

Precisely the point I'm making. It was hubris and laziness and I'm surprised that I let myself go that long. In a follow-up article I discuss an overview of what I'll be doing going forward: http://www.seankilleen.com/2014/01/my-newest-backup-system-p...

Retric• 13 years ago

There is nothing wrong with trusting online backup providers provided your data is backed up on two or more of them and there not both using the same back-end (aka Amazon s3).

sahaskatta• 13 years ago

For this very reason I've always backed-up to both an encrypted external HDD and another encrypted copy in the cloud.

forgottenpass• 13 years ago

Whenever a cloud service failure is posted to HN, there are always posts basically telling the OP that expecting the service you paid for is a silly idea.

Why isn't the SaaS industry in general roundly criticized as the fly by night hacks they always turn out to be when someone gets burnt?

SeanKilleen20• 13 years ago

Haha, I appreciate the defense. At the same time, I understand the economical aspect of the service, and there is a measure of "you get what you pay for."

To be fair, I wouldn't call Backblaze hacks by any means, but I understand your point about the business responsibilities of companies and not giving them a pass just because they're a price-friendly SaaS.

crazygringo• 13 years ago

If you're going to only have one backup (which is the case for most people, realistically), your first choice should be cloud, not local.

As far as I know, the chances of something physically happening to your local backup (the hardware fails, someone knocks it off the shelf, it gets lost in a move, there's a fire) is vastly greater than the chance that your online provider loses your data.

Yes, obviously it's better to have both, but the fact that cloud is both off-site and stored redundantly off-site makes cloud the obvious first choice for most people (since most people are not setting up RAID arrays at separate locations).

acdha• 13 years ago

the chances of something physically happening to your local backup (the hardware fails, someone knocks it off the shelf, it gets lost in a move, there's a fire) is vastly greater than the chance that your online provider loses your data.

This is the key part: almost all of the failure modes affecting local storage also endanger local backups. Cloud backups are intrinsically less risky in that regard, although a local provider might still be affected by a massive natural disaster.

The remaining class of errors, however, are nasty ones like bit rot, human error or malice – all of which are tricky to prevent.

erkkie• 13 years ago

The more general advice would be to test your backups, whether they're online or offline (ideally both). Now I should go and practice what I praise ...

Zhenya2• 13 years ago

Wired or wireless connection? I would always do bit sensitive data over a wired connection.

This goes without saying.

SeanKilleen20• 13 years ago

Wired connection during both attempts (even on the laptop). But you're right and it does go without saying.

Dylan168074• 13 years ago

No it doesn't. Especially when it comes to HTTPS.

mrchucklepants• 13 years ago

This is a little disconcerting. I have 3 computers backed up with Backblaze. I had a successful 350 GB restore last week. I hope that wasn't an anomaly.

SeanKilleen20• 13 years ago

From what I gather, my situation may be the anomaly. Still concerning though. I'm still processing a 340 GB restore in the hopes that this one will go better. We'll see.

Your success has me hopeful!

bydo• 13 years ago

Your situation is an anomaly, or this specific incident?

You _have_ actually tested restoring from your only backup solution at a service you've been evangelizing for years, right?

magic52272• 13 years ago

FYI I left Backblaze for Crashplan for the same reasons.

I believe Blackblaze also removes the backup after some number of hours, so if you don't download the backup by then, you're in big trouble.

Crashplan also allows external and network drives to be backed up.

SeanKilleen20• 13 years ago

Could you elaborate? Same reasons in that you had slow or broken restorations?

Also curious: have you had to restore with CrashPlan, and if so, how has that process differed?

magic52272• 13 years ago

I had many issues with Backblaze over several years.

I found 1-2 times a year, when initiating a backup, the entire archive I had disappeared, as in, my computer would start to send all of my files all over again. This was very disconcerting.

I never liked seeing all files on an network drive disappear just because it was unplugged.

The final straw was when I attempted to restore a large archive, and saw the download speed was so slow, it would not finish before this restore was removed on Backblaze's servers.

Crashplan, has had zero issues over 2-3 years now. I've restored large files without issue, they store versions which I don't even need/want, and the desktop software works very well. Highly recommended.

durandal1• 13 years ago

Their selling point is "Unlimited data backup for your Mac or PC, just $5/month." If you're in tech that should tell you it's not for storing "mission critical data" (paraphrasing the blogpost).

For $5/month you can also not expect round the clock, seven days a week personal attention.

Don't get me wrong, Backblaze seems like a kickass service for an off-site backup of my personal data.

SeanKilleen20• 13 years ago

A good point: "mission critical" is an overstatement and I should phrase that differently. "Critical" to me means personal value (i.e. family photos), and not necessarily in terms of timeliness (i.e. business related).

And to clarify, I don't expect personal attention. Contacting the CEO was because I thought there might be a legitimate problem with a server and support hadn't responded, and writing this blog came after I'd attempted to follow up with support but hadn't heard back for over 8 hours.

I've stated this elsewhere on the thread but I'm by no means out to shame Backblaze. I think there are some UX points worth revisiting, and I may be looking at another provider, but I'm not trying to damn the company. I think it's a discussion worth having on how they might do things better (and also how I might do things better).

hobs• 13 years ago

I also lost a drive recently and found that while uploading to them was a breeze, the download was coming in at 14KB/s. I thought I was the one having a problem, but maybe not.

SeanKilleen20• 13 years ago

Yep, you weren't. Glad to know I wasn't the only one. Speeds varied for me, though the transfer did take much longer than the estimates. And that makes sense, given that I'm sure it's a ton of bandwidth to deal with all of those multiple restores. All the same, setting some expectations in those situations would be helpful, too.

free652• 13 years ago

My current backup strategy:

MacBook & Mini -> Time Machine on Mini (second hd), everything is using FileVault

Pictures -> Bitcasa & local copies

Personal Documents -> TrueCrypt Volume on Dropbox with local copies, coding projects are on dropbox also, obviously I have code in SVN/GIT repositories.

I have 2 local copies, one in the safe deposit box and rotate them about every 3 months. This is pictures + personal files on a FileVault volume.

Currently I don't backup my windows gaming pc aside from game saves on Bitcasa.

kyle_t• 13 years ago

I had the exact same situation three years ago with Carbonite. Had 80GB of data that I was never able to be restore. Their only means of amending the situation was offering a year for free. Never again. I had a local backup but it was unfortunately 3+ months out of date. Now I keep everything locally backed up with a NAS and a mix of dropbox/google drive.

manishsharan• 13 years ago

Backing up your data is not enough; you must also occasionally restore your data ( to a temp location )just to make sure your backup media or its governing database are not silently screwing up your backed up data.

SeanKilleen20• 13 years ago

Agree 100%. In the future I plan to do this with both my local and cloud personal backups. (I would always do this with any business-related backup; I don't know why I my standards be lower for my personal data.)

xbryanx• 13 years ago

Does anyone have restore experience with Tarsnap? Tarsnap seems like a great service, but I've never had to do a restore.

sc68cal• 13 years ago

I've pulled individual files from my tarsnap archives, and did a full restore of a server into a VirtualBox VM to verify that everything worked OK and can confirm that tarsnap works as advertised.

SeanKilleen20• 13 years ago

A question, because I'm unfamiliar with the way HN works in this regard. My article was #2 on the front page for a brief spell.

Now I look, and it's nowhere to be found.

Is this by design? I don't need it to be on the top page and this isn't about the page views, I'm just not sure how it would disappear. Did a moderator somewhere frown upon it?

SeanKilleen20• 13 years ago

Haha, nevermind, I'm an idiot. Just hadn't considered that the sheer amount of content would knock it off rather quickly (which makes sense, given that this is not nearly worth the focus that some other topics are.)

SeanKilleen20• 13 years ago

One other note: I think it's important to say that I'm a HUGE fan of what Backblaze does in terms of open-sourcing their storage box designs and how they accomplish things. I think that's a noble thing to do, and have always thought highly of them for that.

thaumaturgy2• 13 years ago

There doesn't seem to be any attempt to confirm that this is actually a problem on Backblaze's end -- it's just as likely to be a problem with Sean's internet connection. One of the problems with downloading large zip archives is that a few scrambled bits or dropped packets can scratch the whole archive.

Backblaze offers the ability to get your data in the form of a flash drive or external drive (https://help.backblaze.com/entries/20224632-how-can-i-restor...). It's not free, but if your data is important, it's a pretty reasonable cost-of-hardware-and-handling price.

SeanKilleen20• 13 years ago

Hey there! Thanks for commenting. I think I commented on this aspect in a few other places on the thread, but while I glossed over the IT troubleshooting steps in my blog post, I've been in IT for 10+ years in the roles of support, sysadmin, and developer. I _of course_ did some troubleshooting. Two different PCs with internet connections that worked completely reliably both had different issues, and they were both in known stable states at the time. On each, I'd tested the internet connection, speed, etc. Nothing else was wrong with my internet connection at the time, at least in terms of things like loading web pages, downloads, and streaming HD video. In the first case, the speed seemed to drop to zero and then timeout. In the second, case the file took a long time to download and then came out corrupted anyway. I'm not saying these things don't happen; but I'm saying that, if it's a company's bread and butter, it would make more sense to put as many safeguards in place to make sure that after 12+ hours, a corrupted file doesn't happen.

Also, I'm aware that a few dropped packets can be an issue. This is why I used Backblaze's advertised downloader application -- I got the impression that it would provide better stability, checking, etc. to ensure that the zip file did not get corrupted.

I also mention the external HDDs in my post, but my point is that I pay them so that I can restore my files conveniently, not so that I can be upsold to a $189 drive. Normally, I would drop the cash, but after just spending several hundred dollars on a new motherboard, I figured I would use their download mechanism, which is offered. I think I'm allowed to have a reasonable expectation that it would work.

With all that said, this is for sure not some "grab your pitchforks!" moment. I'm doing this partly to call attention to my own stupidity at putting my faith in one cloud backup provider, and also to highlight that Backblaze's process for restoring could likely be a little bit smoother, given that this is when users would like things to be absolutely foolproof. I remain a fan of Backblaze's CEO who was responsive when support wasn't, and of the company itself which has done great things like open-sourcing its hardware.

I've since been in contact with support again and I will be sure to update my post once the matters are resolved.

thaumaturgy2• 13 years ago

The part that doesn't make sense right now is that Backblaze has been around for several years, and if they had some kind of infrastructural issue with transmitting large amounts of data, I'd expect this not to be the first such case to come across my news feed. (BB has been featured on HN a handful of times -- no reports that I know of so far with retrieving data from them.)

With all due respect to your expertise, in at least 9 out of 10 cases where I've encountered something similar to this, it's been a problem on the client's side of the link -- bad traffic shaping by the ISP, faulty modem, glitchy router or switch -- and those often don't show up in speed tests or media streaming or web browsing.

I have an idea that would satisfy my curiosity, if you're willing: how about renting a VPS on DigitalOcean (who charge in small increments), with enough storage for zipped file size + unzipped archive size, and then see if you can download the archive from Backblaze to the VPS and unzip it on the VPS? See e.g. http://victorwyee.com/shell/getting-backblaze-restores-via-c...

If it works, you can rsync over ssh. If it doesn't work, then I'll buy that Backblaze has got a problem.

"This is why I used Backblaze's advertised downloader application -- I got the impression that it would provide better stability, checking, etc. to ensure that the zip file did not get corrupted."

I strongly suspect that their downloader just implements http resume in some slightly brain-damaged way.

"...my point is that I pay them so that I can restore my files conveniently, not so that I can be upsold to a $189 drive."

Ah.

You're paying them $5/month to store your data. That's it. It's a bit of a nit to pick, but it's important: for $5/month, they're not guaranteeing that you can download your data. So providing alternatives to downloading is not an "upsell" (especially at those prices).

"It's the (Backblaze) restores that fail."

Related Stories