Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

https://www.backblaze.com/blog/cloud-storage-durability/

by ingve • 8 years ago

319 184 8 years ago

garettmd• 8 years ago

This was an interesting read, both the points made about durability, as well as the in-depth math. However, what stood out to me most was the line:

Because at these probability levels, it’s far more likely that:

- An armed conflict takes out data center(s). - Earthquakes / floods / pests / or other events known as “Acts of God” destroy multiple data centers. - There’s a prolonged billing problem and your account data is deleted.

The point that once you get to a certain point of durability (at least as far as hardware/software is concerned) you're chasing diminishing returns for any improvement. But the risks that are still there (and have been big issues for people lately) are billing issues. I think it's an important point that the operational procedures (even in non-technical areas like billing and support) are critical factors in data "durability"

schoen2• 8 years ago

I wonder if there's a general term in engineering for the case where a particular risk has been reduced well below the likelihood of more serious but exotic risks. I've heard about this most in cryptography where we can sometimes say that the risk of, say, a system failing due to an unexpected hash collision is drastically less than the risk of the system failing due to cosmic radiation or various natural disasters. At that point it doesn't seem important or worthwhile to consider this risk, because it's dwarfed by the others.

This seems like a form of the same argument, and I wonder where else this arises and how people describe it.

matthewmcg• 8 years ago

I think in other contexts, you might say that these signals are "below the noise floor."

matt4077• 8 years ago

It’s not a perfect match, but Rumsfeldian “unknown unknowns” come to mind.

Specifically: every X-nines durability design will be compromised by some failure mode you didn’t think of.

pixl97• 8 years ago

https://en.wikipedia.org/wiki/Black_swan_theory

klodolph3• 8 years ago

I've posted the math here before but if we assume that an asteroid hits the earth every 65 million years and wipes out the dominant life forms, then this fact alone puts your yearly durability at a maximum of ~8 nines.

The point about billing is better, though.

My other concern is that a software bug, operator error, or malicious operator deletes your data.

GauntletWizard• 8 years ago

The event that wiped out the Dinosaurs was 65 million years ago, but the Mesozoic era lasted for ~200 million years. Your point stands, though.

tjoff• 8 years ago

That's why no sane entity would use one earth. Use two and you can quickly recover.

toolslive4• 8 years ago

Isn't the expected life span of the company more limiting? Plenty of cloud storage companies go out of business (typically, they run out of money). You can apply Gott's law to this. It's pretty grim

thaumasiotes• 8 years ago

if we assume that an asteroid hits the earth every 65 million years and wipes out the dominant life forms, then this fact alone puts your yearly durability at a maximum of ~8 nines

I don't think this is a useful definition of your yearly durability. If your data center is down for maintenance during a period in which it is guaranteed that nobody wants to access it, that doesn't reduce your availability at all -- if your only failure is an asteroid that kills all of your customers, it would be more accurate to say you have 100.000000% availability than 99.999999%.

TheAceOfHearts• 8 years ago

For a consumer the cheapest and easiest way to backup important documents or files is to encrypt it and store it across multiple storage providers, e.g. Dropbox and Google Drive.

They usually give you a reasonable amount of free storage, and it's unlikely all accounts would be terminated or locked at the same time.

And of course, you should always have your local backups as well.

e12e6• 8 years ago

Assume you want access to files over the next 20 years. What are the odds Google will have bought Dropbox in that time; and what are the odds an automated system monitor at Google ad words will have disabled your Google account in that time span?

Replace with Amazon and/or crashplan as appropriate..

eloff2• 8 years ago

This was an interesting read from a technical point of view, but also well written and refreshingly transparent.

I found the discussion about why it doesn't matter when you start talking about 11 nines of reliability to be hilariously true.

At the end of the day we're still flawed humans living in a hostile universe, and no matter how foolproof we make the technology, there are some weaknesses that just can't be eliminated.

kraftman• 8 years ago

It's 'foolproof', like dog proof but for fools.

eloff2• 8 years ago

Hah, good catch, I know that but somehow made the error anyway. I corrected it.

brianwski8• 8 years ago

Disclaimer: I'm the author of the blog post. :-)

well written and refreshingly transparent

Thank you! In the interests of full transparency, the blog post was a collaborative affair and was proof read and edited for clarity by several people at Backblaze.

discussion about why it doesn't matter

One of the philosophies Backblaze uses is to build a reliable component out of several inexpensive and unrelated components. So combine 20 cheap drives into a ultra reliable vault. We have two or three inexpensive network connections into each datacenter instead of buying one REALLY expensive connection for 8x the price. Etc.

Personally, I recommend customers do the same. Instead of storing two copies of your data in two regions in Amazon for 2x the price, store your data in one region of Amazon and put one copy in Backblaze B2 for 1.25x the price. We believe this will result in higher availability and higher durability that two copies in Amazon because Amazon S3 and Backblaze B2 don't share datacenters in common (that we know of), don't share network links, don't share the software stack, etc. For bonus points, use different credit cards to pay for each, and have different IT people's credentials (alert email address) on each. That way if one IT person leaves your company and you don't get an alert that the credit card has expired, hopefully your other copy will be Ok.

k__• 8 years ago

BB and S3 both have eleven 9 durability, how much does using both increase this?

ujjain• 8 years ago

It still wouldn't upload my 1TB in back-ups in an entire month. Amazon Drive back-up completed in 3 days.

Their pricing is amazing, but saving money on a back-up solution that doesn't seem as good as the other cloud storage providers is a dangerous game.

GuB-42• 8 years ago

Not that dangerous if you do your own maths. I don't really trust cloud backup. As they said, the 11 9s doesn't matter. You are more likely to encounter a billing problem (as they said) but also get hacked, have a problem with your internet connection, many things can go wrong.

That's why you need another solution if you are serious about your data, maybe a set of external hard drives (local backup). This way, you have redundancy and little correlation in failure, which greatly improves your general durability. That local storage may be paid with the money you save by getting an "inferior" cloud backup provider.

DenisM• 8 years ago

How do you mean? Is upload too slow?

anderspitman2• 8 years ago

I've found the upload speeds to B2 to be fantastic. Especially if running multiple connections.

3pt14159• 8 years ago

I use B2 and I have no problems uploading large files quickly. Is this for the consumer option?

fencepost3• 8 years ago

Where are you geographically? Was your Amazon upload to a datacenter physically much nearer to you than California? 30MBit/s sustained for 3 days isn't unreasonable for a business connection, but seems high compared to most of what I see available at least in the US.

brianwski8• 8 years ago

Disclaimer: I work at Backblaze and live in California.

30MBit/s sustained for 3 days isn't unreasonable for a business

We (Backblaze) are seeing more and more consumer internet connections in the USA with 20 Mbit/sec upstreams, I thought they were available most everywhere if you were willing to upgrade your internet package "just a bit". 30 Mbits is a little unusual for "consumer", but not unheard of. Of course, there is a "selection bias" when you look at online backup users. :-)

kalleboo2• 8 years ago

When I've gone a clean backup on Backblaze it's taken just over 24 hours to backup about 650 GB. And I'm not even in the US, so my data has to cross the pacific.

Backblaze is actually faster than Apple Time Machine is on my LAN which slightly bothers me. It also has lower CPU usage.

I originally chose Backblaze after benchmarking the other offers available at the time (Carbonite, Crashplan etc) and Backblaze was by far the fastest.

londons_explore4• 8 years ago

This analysis is simplistic.

Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc.

When calculating your chance of failure, you have to include that, or your result is bogus.

Eg. Model A of drive has a failure rate of 1% per year, but when failed the symptom is failure of the drive to spin up from cold, however if already spinning it will keep working as normal.

3 years later, the datacenter goes down due to a grid power outage and a dispute with diesel suppliers so the generators go down. It's a controlled shutdown, so you believe no data is lost.

2 days later when grid power is back on, you boot everything back up, only to find out that 3% of drives have failed.

Not a problem. Our 17 out of 20 redundancy can recover up to 15% failure!

However, each customers data is split into files around 8MB, which are in turn split into the 20 redundancy chunks. Each customer stores say 1TB with you. That means each customer has ~100k files.

The chances that you only have 16 good drives for a file is about (0.97^16 * 0.03^4)2019*18 = 0.3%

Yet your customer has 100k files! The chance they can recover all their data is only (1-0.003)^100000... Which means every customer suffers data loss :-(

ChuckMcM4• 8 years ago

I was going to post essentially the same thing, so here is an upvote :-)

While I always find storage analysis interesting (I spent 5 years at NetApp where it was sort of a religion :-)) some of the assumptions that Brian was tossing out are not good ones to make. (like the lack of correlation, or that Drive Savers will exist as a company 10 years from now).

Still it does help you to understand the they take data availability seriously which is the underlying message.

brianwski8• 8 years ago

Disclaimer: I'm the author of the blog post. :-)

some of the assumptions that Brian was tossing out are not good ones to make.

We COMPLETELY welcome other analysis and listing other assumptions. Internally, we argued endlessly about why this or that wasn't totally accurate, and finally decided to publish the math WITH all of our assumptions exposed so you could be the judge. If Amazon wants to publish their assumptions for S3 for comparison, we're all ears.

that Drive Savers will exist as a company 10 years from now

Absolutely true, this calculation is only good RIGHT NOW. For example, one of the things that came up internally was "well, when drives get more dense the rebuild time rises, so this calculation will no longer be accurate in two years". But at the same time, we have some additional tricks and optimizations to make which we have not done yet to cut the 6 day average drive rebuild time down to 3 days. Also, drives last us about 5 years, so your data will be migrated to totally new drives 5 years from now. Those drives will absolutely have a different drive failure rate (maybe higher, maybe lower) so the calculation will no longer have the same result 5 years from now.

thefifthsetpin• 8 years ago

I think that your point is what they were trying to address by saying that anything beyond eight nines is impractical. Their examples of correlated failures included earthquakes and floods which might have huge reach rather than just impacting a single rack, but I think it's the same general idea.

Johnny5554• 8 years ago

That could be a power surge taking out a whole rack

This failure mode, at least, is already accounted for by sharding data across cabinets:

Each file is stored as 20 shards: 17 data shards and 3 parity shards. Because those shards are distributed across 20 storage pods in 20 cabinets, the Vault is resilient to the failure of a storage pod, or even a power loss to an entire cabinet.

https://www.backblaze.com/blog/vault-cloud-storage-architect...

However, they don't seem to offer multi-datacenter (or multi-region) redundancy so are still susceptible to a datacenter fire/failure.

In comparison, AWS S3 distributes data across 3 AZ's (datacenters), and you can further replicate across regions if you choose. Though you pay for that added redundancy in 3 - 4X higher cost.

gm-conspiracy2• 8 years ago

I thought AZs were in the same physical location, just separate networks, no?

catwell• 8 years ago

Multi-region redundancy was on the roadmap for their B2 offering 10 months ago, not sure if it has shipped yet.

https://news.ycombinator.com/item?id=15125643

ChuckMcM4• 8 years ago

A better example would be the ceramic bearing fiasco that NetApp experienced with Seagate. Seagate had switched to a floating ceramic bearing on one family of their fiber channel drives. In those drives one or more of the bearings would shatter and start spreading ceramic dust across the disk surface. This happened between 3 and 4 years of run time and the disk would rapidly fail after that happened. People that bought a filer with several hundred drives started seeing large numbers fail suddenly. Up to the point they failed, not any indication. But it was run time hours related and thus correlated to all drives that started running at the same time. If you had put those drives in different data centers on different racks on different servers it wouldn't have mattered if you started 20 at the same time to be your 'tome', when they started failing it was possible to lose them all in a week.

ChrisLomont2• 8 years ago

The chance they can recover all their data is only (1-0.003)^100000... Which means every customer suffers data loss :-(

You just made the same mistake you're criticizing. You assumed the 100K files were uniformly and independently spread. They're also likely clustered, and perhaps not even at the same data center. Given the variety of drives Backblaze uses, the drives are also not likely to all be the same model, so your failure method is also unlikely.

You also only looked at the case there are exactly 16 good drives. The proper failure estimate is 1 - (odds of 17 good + odds of 18 + odds of 19 + odds of 20). I'm not sure where you got the 20 x 19 x 18 part either. Did you mean 30 choose 16 or something like that? Using the proper 1-... method I get 0.00267, not 0.03.

londons_explore4• 8 years ago

You just made the same mistake you're criticizing

Yes. Even if you take it into account, the vast majority of customers will see data loss, assuming random (but not even) shard distribution.

I rounded and approximated to avoid explaining too many probability rules... As you can see, it gives the same result at the end.

httpz• 8 years ago

Also, there are only 3 hard drive manufacturers left. If one of them have a bug that affects across their product line, that can take out 1/3 of all hard drives.

kolpa• 8 years ago

It's vanishingly unlikely that a bug would affect all their drives (across all recent models, and only after burn-in) simultaneously, unless the drives are managed by a remote server with a SPOF.

jrockway2• 8 years ago

I'm not sure how anyone, in their disaster recovery plans, ever expect anything less than 100% failure of a data center. The scenario is: a tornado hits the data center.

Your power outage causing 3% of drives to fail is just a subset of that.

viraptor• 8 years ago

If it's your data center, you can plan the location to prevent most of this. There are locations where the natural hazards can be completely managed. (No tornados, fires, tsunamis, earthquakes, ...) So the power outage is the most likely thing to happen.

arminiusreturns• 8 years ago

The first Datacenter I ever brought up to T2 was tornado proof (former military base).

stubish• 8 years ago

Its just risk analysis. The cost involved in splitting things over multiple data centers vs. the chance of your single DC getting wiped out. Declaring bankruptcy if it happens will be the best business decision in many cases, or insurance if the risk is higher or owners can't afford the loss or have liability.

Confusion2• 8 years ago

  Which means every customer suffers data loss

They suffer partial backup loss.

The customer only suffers data loss if they lost their 'master' copy of the data as well during the outage. Iff they don't have a secondary backup solution.

mmt7• 8 years ago

Are they positioning their "B2" product solely as a backup solution, rather than something closer to an S3 competitor?

If it's the latter, "data loss" seems an appropriate characterization.

cjhanks• 8 years ago

Nice analysis.

mmt7• 8 years ago

Correlated failures are common in drives.

This is why, when I was building DIY arrays for startups (around the same time Backblaze published their first pod design [1]), I went through the extra effort of sourcing disks from as many different vendors as possible.

Although it was somewhat more time consuming and limited how good a price I could get and how fast the delivery could be, it meant that, for any given disk drive size, it meant I could build an array as large as 12 where no 2 drives were identical in model and manufacturing batch [2].

Of course, it's still a vanishingly rare risk, and "nobody" cares about hardware any more. It does help to remember, at least once in a while, that, on some level, cloud computing really is "someone else's servers" and to hope that someone else still maintains this expertise.

[1] though I used SuperMicro SAS expander backplane chasses for performance reasons

[2] and firmware from the factory, although this is somewhat irrelevant, as one can explicitly load specific firmware versions, and, IIRC, the advantages of consistent firmware across drives, behind a hardware RAID card, outweighed the disadvantages

jquast• 8 years ago

I went through the extra effort of sourcing disks from as many different vendors as possible.

This is very good advice!

If you already built your array, consider advice: "replace a bad disk with a different brand, whenever possible".

Over time, you naturally migrate away from the bad vendors/models/batches. After following this practice, it seems ridiculous to me now to keep replacing the same bad disks with the same vendor+model.

foobarian• 8 years ago

I've been doing a poor man's version of this for home videos using 3 hard drives and rsync. It's easy to replace a drive and they are not likely to go out at the same time. But one thing that bugs me is that unless the drive fails hard (e.g. noticed by SMART or unable to read at all) how do I know the data on the drive is not corrupted without reading it? Are there best practices to continuously compare the replicas in the background? Does that impact durability of the drives?

hinkley3• 8 years ago

You could also have a bad batch of drives, causing a bunch of failures to happen over the period of a couple of months. Sources of failure don't take a number. Like in your example, they can and will overlap, which is why when we try to design bullet proof systems we aren't satisfied until every bad even requires a number of separate things to go wrong all at once.

But if you have faulty power or bad hardware chipping away at your equipment, your depth of resiliency is degraded until the issue is corrected.

mrb4• 8 years ago

«The chances that you only have 16 good drives for a file is about (0.97^16 ∗ 0.03^4)∗20∗19∗18 = 0.3% Yet your customer has 100k files! The chance they can recover all their data is only (1-0.003)^100000... Which means every customer suffers data loss :-(»

Your math is completely wrong. In reality 92% of customers suffer no data loss.

The chance of a file having 4 failed drives (16 good drives) is: .03^4 = 0.00008100%

The chance of a file being irrecoverable is the chance of having 4 or more failed drives: .03^4 + .03^5 + .03^6 + ... = 0.00008351%

The chance of a file being recoverable is: 1 - .03^4 - .03^5 - .03^6 - ... = 99.99992151%

The chance of a customer's 100k files all being recoverable is: (1 - .03^4 - .03^5 - .03^6 - ...)^100000 = 92.0%

Therefore only 8% of customers encounter one or more 8MB file that is irrecoverable.

barbegal• 8 years ago

The original maths is correct. You have neglected the fact that the 4 failed drives can be any of the 20. Your calculations are for 4 specific drives failing.

jdcarter• 8 years ago

I recently looked into the research on this topic, and it absolutely agrees with you--failures are not independent. Papers of note:

https://www.usenix.org/legacy/event/fast08/tech/full_papers/...

http://www.cs.toronto.edu/~bianca/papers/fast07.pdf

Take note of section 5 in both papers for their statistical models and how they match with real-world data.

hinkley3• 8 years ago

There was a statistics class in college that kicked my ass. I never quite understood how to determine if two variables were independent or dependent. You get a vastly different answer if you get it wrong.

I run into people all the time that seem to have the same problem, to the point that it makes me wary of any software developer putting forth numbers that seem fantastical.

My gut reaction is that if Backblaze wants to keep reporting their disaster preparedness numbers that they need the assistance of an actuary to calculate them.

KindOne• 8 years ago

Year 2038 seems a bit simple to test. Can you just set the BIOS date to January 19 2038 03:00:00 UTC and wait 15 minutes to see what happens?

riku_iki3• 8 years ago

if you store 1 million objects in B2 for 10 million years, you would expect to lose 1 file.

Can this be reformulated: you store 10 trln objects (e.g. 100TB of 10 byte records), you lose 1 record each year.

Also curious what are the stats from other providers.

jsjohnst5• 8 years ago

Can this be reformulated

Roughly, yes.

Also curious what are the stats from other providers.

As to other providers, most are 6+ 9s that I’ve looked at, with many in the 8-9 range. Anything over 8 is (as they admitted) essentially marketing porn and not a useful metric (for reasons they mentioned as well as ones said by other comments here).

riku_iki3• 8 years ago

And yet, no much complains about lost GMail letters in internet because of disc failures.. And GMail is likely stores much more than 10 trln objects and 100TB of data.

berns• 8 years ago

Azure:

locally redundant storage: 99.999999999 % (11 9's)

zone redundant storage: 99.9999999999 % (12 9's)

geographically redundant storage: 99.99999999999999 % (16 9's)

https://azure.microsoft.com/en-us/pricing/details/storage/

Dylan168075• 8 years ago

The raw number would imply that, but I'm pretty sure the math breaks down when you're storing 10 byte records.

Chunks of your data are going to be stored together, so it's a very small chance of losing a big block of 10 byte files. There's no failure mode that loses just one, and does so often.

riku_iki3• 8 years ago

Chunks of your data are going to be stored together

It probably depends on their infrastructure, e.g. if storage is something like cassandra, records would be evenly distributed by key hash.

I agree that numbers likely are not like that though, just wanted to demonstrate that such calculation approach can bring unexpected conclusions.

londons_explore4• 8 years ago

I've very disappointed their recovery time is 6 days!

Recovery workload should be spread across the whole cluster, so that the recovered data gets distributed evenly. In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done in under a second. Maybe 10 seconds with some sluggish tail machines.

Why do you need it done in under a second? While the data is down one replica, it is at dramatically higher risk. Also, drive failures can be dramatically accelerated, for example in the case of a bad software release erasing data - you need to be able to move data faster than bad software gets released. And releasing software at a rate of one machine per second still means a release takes 3 hours!

ChuckMcM4• 8 years ago

Be careful here, it isn't 6 days until data is recovered it is 6 days until it is fully protected again, there is a big difference. During the 6 days the data would be available it just might have to be reconstructed on the fly by the error correcting rather than read directly.

In most systems we assume that "primary traffic" (read/write stuff) is prioritized over "rebuild traffic" which is recovering lost shards. So when you specify these things it is best to specify "how long to rebuild a shard while the array is providing storage services at its maximum specified rate." This assures the customer that if they have a 24/7/365 non-stop traffic pattern their data will still stay protected in the face of drive failures.

brianwski8• 8 years ago

Disclaimer: I'm the author of the blog post. :-)

During the 6 days the data would be available it just might have to be reconstructed on the fly by the error correcting rather than read directly.

Correct. More specifically, the FIRST time the data is accessed in any 24 hour period it must ALWAYS be reconstructed from the Reed-Solomon encoded parts on 17 other drives on 17 other machines. Any 17 is fine, so it's totally fine if 1 or 2 drives are not available. Once reconstructed it is stored in a set of front end cache computers that have fast SSDs for this purpose.

The second time the same file is accessed in a 24 hour period, it will be fetched out of the SSD cache layer so it won't even hit the spinning drives and won't care if all 20 drives are offline.

"primary traffic" (read/write stuff) is prioritized over "rebuild traffic"

Yes. Backblaze balances between the two if only one drive has failed, but as a tome (20 drive group spread across 20 computers) becomes more badly degraded Backblaze begins favoring the rebuild. When two drives have failed out of 20, Backblaze stops allowing any writes to that tome because more writes will tend to fail yet another drive. Fewer writes offloads the tome. But we still allow reads. At Backblaze, we have never been 3 drives degraded out of 20 (knock on wood), but if this ever occurs the 20 drive tome is now running without parity -> so in that case we even stop allowing reads AT ALL until we are returned to at least 1 drive of fully redundant parity.

jsjohnst5• 8 years ago

In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done in under a second.

I want to know where you can find a drive that can write 12TB/sec of data!

(In other words, you clearly missed half the problem. To add a new replacement drive, you have to be able to write to it the data from an original drive. Also RS code calculation is fast these days, but it ain’t that fast)

riobard• 8 years ago

Additionally, it implies that data needs to be spread across 10,000 drives, which is unrealistic anyway.

londons_explore4• 8 years ago

It's assuming that the newly recovered data is spread across free space of the other 10,000 drives, a few megabytes on each.

It will also be recovered by reading the recovery data, which should also be approximately evenly spread across the 10,000 drives.

TheDong• 8 years ago

you need to be able to move data faster than bad software gets released. And releasing software at a rate of one machine per second still means a release takes 3 hours!

Since others have already demonstrated why the remainder of your comment is overly simplistic, I'll tackle this bit.

Generally, software releases are not rolled out at a constant rate to all machines. A typical thing to do is to release it to staging, then to a "canary" subset of machines (e.g. to 1% or 5% of the machines).. Once all seems well there (e.g. metrics are clean and the canaries have handled X writes, reads, and simulated drive failures), it can be rolled out to a larger subset, and eventually to all machines.

In that way, the release can take whatever total amount of time is desired while still catching any such bugs fairly reliably.

Ideally, at backlblaze they could ensure that their canary instances are "data-redundancy aware" such that even if the 5% they roll to for the canary test all explode, data is still safe.

Regardless, any talk of "recovering data faster than software releases" is completely silly and totally misses the reality of how releases are done, how recovery is done, and what sort of bugs might happen. The math based on faulty assumptions about rate is also pointless.

Animats• 8 years ago

Financial failure or service shutdown by the provider is the highest risk for long term storage. The backup services CrashPlan, Dell DataSafe, Symantec, Ubuntu One, and Nirvanix all shut down. Nirvanix only gave two weeks notice for users to save their data.[1]

[1] https://www.computerweekly.com/opinion/Nirvanix-failure-a-bl...

chx• 8 years ago

Just for fun, here's my report of Nirvanix in 2008. I feel no problems sharing it since both NowPublic and Nirvanix is long gone. Let's just say there were reasons they went under:

1) Sideloading. I was unable to benchmark or even to get this to work. Images requested to be loaded from media-src.nowpublic.com to node4 @ Nirvanix never showed up. I showed my code to [nirvanixcontact] who said that the code looked OK but someone did a load testing on node4 without informing Nirvanix and that steps are made so that such a situation won't occur again. He told me that the requests are not lsot but they still have not landed. Again, my code can be at fault and I would be happy to run a sideload example code.

2) Upload speeds. I have uploaded from d2 to Nirvanix 100 images each between 180-8584 kbytes totalling almost exactly 100 MB (101 844 931 bytes). The upload was a single HTTP request. The uploads took 18-19 minutes (I repeated the experiment). To give us a comparison I changed the URL in the script to a one line PHP script on another server (at hostignition.com) which just echo'd the number of uploaded files. This took 16.25 seconds and echo'd 100 so seemingly the files landed.

2a) I tried to get another node via the LoginProxy method which we would need for uploads anyways. While LoginProxy itself did work, GetStorageNodeExtended https://services.nirvanix.com/ws/IMFS/GetStorageNodeExtended... always fails with ResponseCode 80006, ErrorMessage: Session not found for ip = 67.15.102.70, token = e7b00d25-fc35-431c-9437-9a4302767f46. Seemingly, does not pick up the consumerIP.

3) The image conversion itself is blazing fast though. These images took a total of 101.58 seconds to convert and this includes 300 HTTP requests (200 sent to d2 from Nirvanix, 100 to Nirvanix).

brador• 8 years ago

I would add a third risk of an account ban by some type of future automated copyright content ID. Especially if it is silent/without warning.

stefan_2• 8 years ago

Forget account ban, there is a nonzero risk a false positive of the automated kiddie porn search all the cloud storage providers do has your home searched and puts you in handcuffs.

the84722• 8 years ago

That's one of the reasons to encrypt locally and only store encrypted data on backup services.

pbhjpbhj2• 8 years ago

How about USA government instruct the owners to cease access or take ownership like with Kim Dotcom's New Zealand (?) based storage company.

I don't recall hearing about anyone recovering access to their data in that case?

This makes every additional user an increase in risk, as even without a warrant it seems if USA TLAs consider someone a valid target then those servers are going down (or getting taken over by people with unknown service standards in order to run a sting, or ...).

tl;dr you have to worry about accusations (of law breaking or copyright infringement) against others too as some jurisdictions have a strong overreach in such cases.

toolslive4• 8 years ago

add Bitcasa to the list: https://en.wikipedia.org/wiki/Bitcasa

rafaelgarrido• 8 years ago

is the website down?

hartator2• 8 years ago

It was the 0.00000001%.

cascom• 8 years ago

One simple question - what is the math if one data center goes down?

https://www.nytimes.com/interactive/2018/05/24/us/disasters-...

shub• 8 years ago

Just one missing piece: the actual loss rate.

mikeryan• 8 years ago

There’s another piece too. Practically they’re a backup service so knowing how often someone needs to recover their data and not had an opportunity to reset their backup based on their current state. I’ve used Backblaze for years and only needed it once. (I also back up to a time capsule as well since data recovery is more Practical/Easier from that)

anderspitman2• 8 years ago

Off topic but I just want to say how happy I've been as a Backblaze customer. B2 is a fantastic product for my backup needs, and their hard drive stats have always been handy when selecting drives.

atYevP5• 8 years ago

Yev from Backblaze -> Thanks! Glad you're enjoying it :D

toolslive4• 8 years ago

The best durability is probably achieved by Amplidata, but it does not matter.

You need to do the same calculation for your meta data, which is probably not erasure coded. If you lose this, you don't lose your data, but you no longer know where you put it.

So you probably add your meta data to your data as well in some kind of recoverable format. That's fine, it means that you can harvest the meta data again.

But how long does this take ?

Pissompons• 8 years ago

It's a really nice blog post but coming from Backblaze, it would have been nice if they wrote it _after_ bringing the Phoenix DC fully online. When Amazon or Google say 11 9s, I can believe it but Backblaze still only has a single datacenter for most data. All it takes is an earthquake.

sneak2• 8 years ago

Or an overzealous prosecutor.

fencepost3• 8 years ago

One question since I know some of the Backblaze folks respond to these threads:

In addition to calling (and possibly getting blocked/ignored), does your customer service staff send text messages? I suspect that a big percentage of the phone numbers you have are for cell phones these days, and I see a lot less SMS spam than I do telemarketing. SMS would also allow you to get a bit of info visible to recipients (e.g. "Backblaze CC Expired") with more detail once a message is opened.

atYevP5• 8 years ago

Yev from Backblaze here -> I believe we do send SMSs in the case of a Cap or Alert getting reached, so yes that could be possible - though I'm not sure if an SMS is part of our billing failure process - that's an interesting question!

u02sgb• 8 years ago

Backblaze B2 customer here. My credit card stopped accepting your billing and no SMS for me. Took me a month or so to notice the emails and update my details. I've got SMS alerts active for Caps. Would be worth adding that as was a bit scary when I noticed the mail (think it was the third one you'd sent!).

hartator2• 8 years ago

The math on calculating all this is extremely complex.

Hum, I would be more reassured by past statistics than a probability evaluation. Did they happen to have loss data since their creation?

duxup• 8 years ago

I just wish Backblaze would not go "oh man a lot of your data changed, you should probabbly check for integrity and start your backup with us again later...."

Oh gosh thanks Backblaze, I'll just dig through several TB of stuff....

egonschiele• 8 years ago

why calculate poisson and binomial when both are ultimately gaussian?

RA_Fisher• 8 years ago

They're not ultimately gaussian. :) The normal distribution is continuous and has unbounded support whilst neither the binomial or Poisson ate cts and unbounded.

rossdavidh• 8 years ago

Fun read, but the whole time I was reading it I could hear Nassim Nicholas Taleb making exasperated noises of outrage.

skybrian• 8 years ago

The billing problem is mentioned, but just left dangling. I wonder if anyone has done any interesting work at fixing this?

Jabbles• 8 years ago

to lose a file, we have to have four drives fail before we had a chance to rebuild the first one.

I wonder how many times they've had 2 or 3 drives fail before they've rebuilt and if that matches their predictions.

aikinai2• 8 years ago

I really want to like Backblaze and they seem to do a lot of good work, but whenever this comes up, I also feel responsible to let people know the dark side so they're informed at least.

I've written in more detail before[0], but just to share the gotchas in case anyone here is thinking of switching to Backblaze:

1. They backup almost no file metadata.

2. The client is very slow (days or more) to add new files and there's no transparency (it claims everything is backed up when it's not).

3. There are still bugs in the client that can put your backup into an invalid state where it gets deleted.

4. Support is terrible, and won't be any help when you run into these bugs.

[0] https://news.ycombinator.com/item?id=16301626

Dylan168075• 8 years ago

I didn't like their main client very much (though it was a while ago) but I'm still planning to use B2 for some things. So it depends on what you're buying from them.

rendaw• 8 years ago

As much as this is off topic, I'd like to continue this conversation. Do you have any details/reference for point 1?

I've been using rclone (got the recommendation here) which has been reliable.

Also, does anyone know if Backblaze has any plans to offer u2f? I've switched dns and email providers to get u2f.

aikinai2• 8 years ago

Yeah, it's a bit old, but here's an article about Backblaze not supporting metadata. [0] "It fails all but one of the Backup Bouncer tests, discarding file permissions, symlinks, Finder flags and locks, creation dates (despite claims), modification date (timezone-shifted), extended attributes (which include Finder tags and the “where from” URL), and Finder comments."

And I don't know if it supports U2F, but it does support TOTP.

[0] https://mjtsai.com/blog/2014/05/22/what-backblaze-doesnt-bac...

zimbatm• 8 years ago

answer: because all the files are in the same, single data center which has a lot less nines

contravariant• 8 years ago

The sub result for 4 simultaneous drive failures in 156 hours = 1.89187284e-13. That means the probability of it NOT happening in 156 hours is (1 – 1.89187284e-13) which equals 0.999999999999810812715 (12 nines).

Minor nitpick, this ignores the possibility of more than 4 failures, although this error only affects the fourth digit after the nines. Much more egregious is the following:

there are 56 “156 hour intervals” in a given year

This is too simplistic, there are in fact infinitely many 156-hour intervals in a year, some of them just happen to overlap. This overlap can't simply be ignored because even if none of their 56 disjoint intervals contain 4 events this does not rule out the possibility of there being 4 events in some 156 hour interval they didn't take into account. In fact failing to take into account even one of the infinitely many intervals creates a blind spot (consider what happens if the drives happen to fail precisely at the start and end of a particular interval). You can still get a lower bound by e.g. ensuring none of the 56 intervals contain more than 1 failure, or by adding more intervals and ensuring none of them have more than 2 failures etc.

Their binomial calculation contains the same mistake.

A quick improved lower bound can be obtained by calculating the probability that any failure is followed by (at least) 3 other failures within 156 hours. For one failure this probability is given by the Poisson distribution and is

    Pc = 1 -\sum_{k<3} e^-λ λ^k / k! = 5.18413e-10.

Now we get into some trouble because the failures and the probability of a 'catastrophic' failure are dependent, however the probability that any particular failure turns catastrophic is constant, so the expected number of catastrophic failures can't be greater than the expected number of failures times that constant, this gives a lower bound of

    Pc (365·24·λ) = 6.63154e-9

this is a lower bound, but that's still three fewer nines left than their claim.

Anyway let's just hope their data centres are more reliable than their statistics.

Edit: This last calculation can be justified by noting that the probability that 1 critical failure starts in a particular time interval is Pc times the probability of 1 failure in that interval plus some constant times the probability of more than one interval. Similarly the probability of more than one critical failure is at most the probability of more than one failure.

Now the probability of more than one failure in a time interval is dominated by the length of the interval, therefore if you calculate the density those parts fall away and you're left with a density of Pc λ critical failures per hour.

This seems to be an exact expression for the expected number of critical failures, and not just a lower bound. Although it is still a lower bound for the probability of a critical failure, albeit a fairly tight one.

sneak2• 8 years ago

This article supposes that a meteor impact is more likely than disaster that renders northern California (their only data centers are in Oakland and Sacramento AFAIK) without power or civil order within ten million years.

As someone else pointed out, it’s overly simplistic. They’re a great low-cost alternative to S3, sure. But keep a backup on another continent if you need your data 100 years from now.

simonebrunozzi• 8 years ago

The Backblaze blog post points to Amazon's CTO (Werner Vogel)'s blog post, in which he states that "...These techniques allow us to design our service for 99.999999999% durability."

(side note: Werner is a great person)

Unfortunately, there is a difference, a huge difference, between a system "designed" for 11 9s of durability, and a system "offering" 11 9s of durability.

I wish Backblaze, or Amazon, or anybody else, would clarify durability using very honest terms.

An example?

"This system offers X 9s of durability over a period of one year, on average. This is a technical paper that describes how we tested that durability", followed by measurements and test specifics.

Any other claim has much less value to me.

HelloNurse• 8 years ago

"Our nines go to eleven"

Well written, but there are other significant risks like losing access credentials (e.g. a password stored only on one device that is destroyed in the same accident in which its only user, who remembered the password by heart, dies) or being hacked by someone who gains access to cloud storage and intentionally erases or corrupts data.

Specialization is good, but if Backblaze is strictly in the business of storing data on hard disks, who's going to help with designing and maintaining the reliable complete system on top of their service that users actually need?

juancn• 8 years ago

My experience is that it's bullshit. I had a backup (damn I still have one) with Backblaze, when attempting to restore it, maybe 30% of the files survived restore. The rest are lost in smoke.

They don't have any way to detect corruption in the data or if they have, the backup clients are oblivious to it.

I lost about a 150GB of family photos and videos.

Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

Related Stories