Backblaze Hard Drive Stats Q2 2020

https://www.backblaze.com/blog/backblaze-hard-drive-stats-q2-2020/

by caution • 6 years ago

294 119 6 years ago

I win. The 8TB drives I used to populate my NAS scored 0.10% better than the 0.81% average. That means my NAS is better than everyone else's.

This big takeaway from these numbers is how dramatically low they are in comparison to drives 5/10/25 years ago. If you treat them reasonably, modern drives are rock stable.

RealStickman_• 6 years ago

You could have done 0.7% better :)

ghaff2• 6 years ago

In the last 5-10 years, I've definitely retired more drives because they became too small for what I was using them for than because they failed.

jhardy54• 6 years ago

Have you considered using them for RAID?

https://en.wikipedia.org/wiki/RAID

programmertote4• 6 years ago

Tangential question: What NAS device and set up do you use? I'm thinking of buying one and would like to learn from a fellow (technically better acquainted) HN user. Thanks in advance!

sandworm1012• 6 years ago

Synology DS1019+

It might not be the cheapest, but 4+1 drives is more efficient than 3+1. Currently I have three 8TB drives: 1xWD Red and 2xIronwolfs (wolves?) with one running as parity (SHR). No complaints. It does everything I need it to do. The interface is simple and everything "just works" as anticipated. If/when I run out of space I plan on adding another 2x8TB drives, probably more ironwolves.

I wouldn't go bigger than 8TB as they already take DAYS to add to the array.

xur17• 6 years ago

Not the OP, but I'll answer. I have the Synology DS-918 for the past 2 years, and it's been great. They have fairly decent included management software + an application library. I have it setup to automatically backup to Backblaze every night, which makes me fairly confident anything I store on it will be recoverable.

yumraj2• 6 years ago

First thing you need to decide is buy off the shelf NAS or make your own based on some hardware you may have lying around.

I have an old TS140 with an i3 that can take 4 HD, has ECC RAM with 4 slots. I'm currently exploring setting it up with Freenas (or Truenas after the merge) based on ZFS. So far it looks pretty good and performant.

For off the shelf NAS, Synology options look good..

execution2• 6 years ago

I'd recommended building a custom one, if you're technically inclined. But off the shelf ones are fine, just not as cost-effective, but they are smaller though!

I previously had a HP N54L with FreeNAS, modded with extra two extra drives to get 6x4TB and that setup was very simple to do.

Now I moved over to a custom one which is a Fractal Node 804 case, i3 8100T, C246M-WU4 mobo and 32GB ECC DDR4 RAM. It has a SF450W Platinum PSU, and it pulls less than 40W in use - could get it lower with some underclocking. This runs Debian Buster with ZFS on Linux, and it's been rock stable for over half a year now - moving the drives over from my N54L was surprisingly simple. Plex transcoding works fine as well. Definitely recommend it.

JamesSwift• 6 years ago

I'm really happy with the synology DS418 I have. I recently attached a UPS as well, which I wish I would have done before (had a scare with losing my disk array). The system is incredibly easy to setup (once you decide on your volume strategy) and administer, and the one time I needed support it was extremely well handled.

NikolaNovak2• 6 years ago

I've tried virtually everything over the years - unRAID, SansDigital boxes, Linux RAID, WIndows RAID, Hardware PCI card RAID...

This year I broke down and bought a Synology (918+). It's not perfect but I'm still a HAPPY happy camper --> I'm at a point in my life where RAID is a means/tool, not a goal / fun project to pour lifetime of hours upon ¯\_(ツ)_/¯

bouvin• 6 years ago

Synology DS1815+ with 16 GB RAM, and a DX513 extension cabinet. While early versions were marred by the Intel Atom bug (I had mine replaced under warranty), it has been a solid performer. I backup to an DS1511+ (also with an extension cabinet) hosted at a friend, and host his extra box in return. The management software is very nice, and as it can run Docker, it is straightforward to augment with additional functionality.

ksec• 6 years ago

If you dont want the hassle, there are basically only two choices on the Consumer NAS market.

Synology, and you should also look for and ONLY buy models which support Btrfs. Silent file corruption is very real.

Personally I am waiting for Qnap's Hero Edition OS, because I just love ZFS.

Other than these two, I am not aware of any other consumer NAS which support either of these file system.

Or may be someday Blackblaze decide to enter NAS market :D

ramraj07• 6 years ago

One of my main questions is how different these stats are for hard drives that are not running all the time. Personal experience over a decade in an underfunded lab was that if you took a drive offline and left it for couple years, you had 10-40% chance of it failing.

leetcrew• 6 years ago

I had a couple drives that no longer worked after leaving the computer dormant for a few months. most of my drive failures have occurred immediately after hardware upgrades though. seems like I lose at least one drive every time I replace a mobo. I'm always surprised that breaks them, but they always survived trips to/from college in the trunk of my car.

londons_explore3• 6 years ago

I'd give those drives a reformat... Some motherboards have firmware that does odd things to drives, like hiding the first or last sectors to store its own data or for various OEM recovery techniques.

That can make your data look corrupt. Yet reformatting (or just putting the drive in with the old motherboard) will magically fix them.

java-man2• 6 years ago

Interesting how HGST has consistently low failure rates [0]. What makes these drives so reliable? Japanese fixation on quality? Specifics of Hitachi design?

[0] https://www.backblaze.com/blog/wp-content/uploads/2020/08/Ch...

Hamuko• 6 years ago

Isn't HGST just Western Digital now?

misanthropian00• 6 years ago

Only if Western Digital was dumb enough to kill the golden egg laying goose. If I were they I would try to keep the Hitachi manufacturing just as it was when they bought it. They do seem to still brand them differently than their own drives and charge a hefty premium and market them to data centers rather than individuals. For me they are way too expensive now. So for me the only Japanese option is Toshiba. I have had much better luck with Toshiba than with Seagate and would easily pay 25% or more of a premium to avoid Seagate. It is so sad because I remember when Seagate was a great and very reliable brand so many years ago.

cehrlich• 6 years ago

Interesting to note that this wasn't always the case - The Hitachi Deskstar line used to have the nickname "Deathstar" due to their low reliability (this was maybe 15-20 years ago)

zokula• 6 years ago

The DeathStar name was never used when Hitachi made the drives. Deathstar was used omwhen IBM used to make the drives

java-man2• 6 years ago

Back when the drives were manufactured by IBM?

jandrese4• 6 years ago

It's also worth noting that the "Deathstar" fiasco was such a disaster because previously people who cared about reliability had been selecting the IBM drives due to their excellent track record. It took only one generation (or two) of defective drives to completely ruin IBMs reputation.

Years later people noticed that the drives were highly reliable again and the Deathstar problem was a fluke. But now it's too late. All of the HDD manufacturers have been gobbled up by just two companies and all of the interesting competition is on the SSD side.

system22• 6 years ago

While the WD and Seagate models range from $99 to $109 for 4tb, HGST is $149 on amazon. That might be the reason?

jaclaz• 6 years ago

This deserves to be cited:

HGST drives win the prize for predictability. Boring, yes, but a good kind of boring.

codezero3• 6 years ago

Also worth calling out that the worst performer, Western Digital, acquired HGST, now let's see if WD improves or HGST gets worse :)

Edit: See comment below, looks like this happened a while ago, so not likely a real factor :)

dv_dt2• 6 years ago

WD acquired HGST in 2012, but was strictly independent until 2015.

https://en.wikipedia.org/wiki/HGST

cactus2093• 6 years ago

The problem is, as far as I can tell you can't really buy them anymore. At least from consumer sites, maybe they still sell large orders to enterprise customers. But it seems like WD is finally discontinuing the HGST name.

ebg13• 6 years ago

It's interesting that, despite Seagate being relatively bad compared to HGST, Backblaze keeps installing more and more Seagate drives and not that many HGST drives. I guess the analysis that's missing here is the cost per drive hour?

atYevP5• 6 years ago

Yev from Backblaze here -> Yes, we tend to weigh heavily in favor of price and Seagate thus far as been affordable with performance that meets our requirements, so that's why you'll see those drives in higher numbers. We're also trying to diversify our fleet a bit, so it's not purely one vendor, but we do try to keep an eye on cost since we run a pretty lean ship!

AnIdiotOnTheNet• 6 years ago

Yeah, pretty much the same reason people keep buying Seagate drives generally: They cost less and they aren't so much worse that it is worth paying a higher price for a drive that's less likely to fail.

kube-system• 6 years ago

It sounds like cost is the primary factor, but diversity can also help mitigate the risk of unknown bugs.

Lets say you found that HPE SSDs were super reliable over your test period so you decided to put everything on a bunch of new drives.

Then this hits, and 100% of your drives fail at the same time: https://www.pcmag.com/news/time-to-patch-hpe-ssds-will-fail-...

While these stats are a great analysis of random failures -- that's only one type of risk.

hinkley4• 6 years ago

I have seen homogeneity 'kill' twice in my career. The first time was unprovisioned hardware, then second time we lost half of the hard drives assigned for dev machines in the space of about 9 weeks. I had so many processes in place by that point that it was a huge inconvenience (mostly due to whole disk encryption) but zero data loss.

The worst thing you can do is to put all of your eggs into one basket. And bulk ordering might get you a set of drives all from the same batch. Bad batches will tend to all fail for a similar reason. Drives fail on a bell curve, right? So the first drive may fail way before any others, but in a RAID array, rebuilds are stressful. Eventually you will hit a statistical cluster. Multiple drives failing close together. If that happens during a rebuild, you will lose a RAID 5 array. If you are very lucky, your RAID 10 array loses two drives in the same mirror. If you have two failures during a long rebuild, even RAID 6 won't save you.

I just bought a Synology box for home. This is my third and probably final RAID enclosure for personal use. I was having trouble finding Backblaze-tested drive models to populate it, so I filled it with drives from a Drobo and kept looking. Initially I had populated the Drobo with 4 drives I bought at once. When one failed, I bought 2 HGST drives and replaced a pair. When the new drives arrived, I started trying to cycle them through, and one of the drobo drives failed. I'll give you two guesses which one.

There is, as far as I can tell, no prosumer multi-disk filesystem that uses consistent hashing to stripe+mirror files across an arbitrary number of disks, instead of the heavy linear algebra RAID5 relies on. It requires touching the whole disk on every rebuild, and I believe that's why Object Storage is slowly taking over from the top end. It's a simpler form of redundancy.

I hope that it's worked its way down to my price range by the time the motherboard on the Synology burns out.

hinkley4• 6 years ago

As I recall, Backblaze does what some other companies have also figured out: amortized maintenance costs across racks.

Long ago we hit a point where power and cooling, not floor space, were the biggest bottlenecks for many data centers. Those numbers shift around, but during that era a bunch of people noticed that it's less labor intensive to fill an entire rack at a time. Put a few extra racks in as spares, and as hardware fails you power them off and power up new machines in the spare rack. When X servers in a rack are dead, you migrate everything else out of the rack, then strip it to the rails and put brand new hardware in it. You increase your server density twice in that case.

Event-driven maintenance is disruptive to other work. It tends to have a ticking clock involved, so you likely assign it to more experienced people, which is 2 opportunity costs in one. Rack-at-a-time is batch processing. You can schedule it, you can assign multiple people to it, and speed is less of an issue. I might have the new guys stripping a decommed rack, use building a new one as training for four people and one teacher. You put a third of it together, I'll come tell you why the wiring is wrong, lather, rinse, repeat.

pwinnski2• 6 years ago

I never get tired of these reports, and if I ever again buy a drive with a high failure rate for Backblaze, somebody slap me!

Bedon292• 6 years ago

Same. And whenever I need a new drive, which isn't that often, I consult the latest report to make a purchase decision.

atYevP5• 6 years ago

Yev here -> Thanks! We love producing them - they are a lot of fun!

system22• 6 years ago

Why wouldn't backblaze use only HGST then? What's the purpose of buying seagates which was the most unreliable hard drives from 15-20 years ago and their stats show higher failure rates. (Still I don't buy them because of the bad taste left.)

pwinnski2• 6 years ago

Presumably cost and availability. I remember reading at one point they were driving from store to store buying every drive they could find.

hinkley4• 6 years ago

That was during the flooding in Thailand that took out a huge chunk of hard drive manufacturing capacity. Electronics stores started putting quotas on purchases, so BB offered a bounty to friends and family who would bring in hard drives.

They had a disassembly line gutting external hard drive enclosures for the hard drives, too. One of the drives in my array was acquired via a similar process.

You should read the whole story. I should probably read it again myself.

https://www.backblaze.com/blog/backblaze_drive_farming/

xvolter• 6 years ago

I think this comes down to price. They are constantly buying drives and to avoid supply issues, it is acceptable to have a hard drive that fails to a certain degree, if the price is balanced.

It would be interesting if these reports included prices, but that might be a problem for Backblaze to reveal so much about their business operational costs.

uj8efdkjfdshf• 6 years ago

Cost and availability, especially at the higher capacities. Note that HGST was bought out by WD in 2012, and WD started sunsetting the HGST brand in 2018.

atYevP5• 6 years ago

Yev from Backblaze here -> Some of the responses down below are correct - it really does come down to price. We run a fairly lean ship and so the price per gigabyte is weighed pretty heavily when we make our purchasing decisions. We have a post where we go into the cost of hard drives over time, it's about 3 years old now, but still a good read -> https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/.

toomuchtodo• 6 years ago

Do you have an algorithm of sorts that takes into account price and historical failure rate by drive model and gives you an easy "yea or nay" for purchase decisions? Seems like you might be able to automate drive purchases with such a system (similar to limit orders on financial securities).

Twirrim• 6 years ago

I would think you'd also want to be hedging your bets a bit against some kind of catastrophic bug. Flexibility in your supply lines has also got to be a big plus (and help smooth sailing with various disasters)

jandrese4• 6 years ago

If you have a system that can efficiently deal with a drive dying without losing data then it makes sense to weight availability and cost higher over reliability.

For home users a dead or dying drive is such a headache that it's usually worth spending the extra bucks on something that is less likely to fail on you.

ping_pong• 6 years ago

Why did they stop buying WDC drives? Is there a known issue with them?

Also, why can't I find HGST drives for decent prices? On Amazon they are either refurbished drives or crazy expensive prices for new ones?

Unklejoe• 6 years ago

they are either refurbished drives

Just a heads up: there's no such thing as a refurbished hard drive. It's not economical. Instead, those drives are actually just used drives with the SMART counters cleared.

I've been personally burned by this.

neilv• 6 years ago

So they're just rolling back the odometer, and destroying records of accidents (errors)?

donatj• 6 years ago

Why can't I find HGST drives for decent prices

Because they're discontinued, Western Digital rolled them into their server line, and people like me are snapping the remaining stock incase drives in our RAID die.

boring_twenties• 6 years ago

Western Digital bought HGST and those drives have some different branding now, I forget exact which.

iooi• 6 years ago

Ultrastar

andy4blaze4• 6 years ago

Andy at Backblaze here. To put a pin in it, the three main factors for which drives we use are cost, availability and reliability. We have control over reliability as our systems are designed to deal with drive failure. That leaves the market to decide on cost and availability. Assuming a competitive market we can buy the drives that optimize those factors.

gentleman113• 6 years ago

My main concern with third party backups is the privacy aspect. I never want to worry that some silent TOS change means that ad companies are scanning all my documents. Does backboaze have any options for e2e encryption or some sort of iron clad privacy policy without a “we can change it at any time” clause?

bronco210162• 6 years ago

Several backup tools have encryption in place so that your data can be encrypted before it leaves your device. Rclone for example has encryption and Backblaze B2 capability.

ewams• 6 years ago

They have an option in software to enable encryption. You provide a key and supposedly the encryption happens at the client in your end. Obviously you have to trust their software and terms like you mentioned. Though backblaze has a great track record and is very open. If you don't want to trust them there are other softwares you could use to encrypt your data before giving it to backblaze.

But one of the benefits of backblaze is the simplicity. Simplicity of setup, backups, and restores. If you muddle with that by encrypting before giving to backblaze you lose out on part of the value.

It also usually seems when people roll their own it opens up risk of forgetting something. BB is easy, set up their encryption and you should be fine.

Just some thoughts.

akersten• 6 years ago

I use HashBackup[0] (I think the author is on HN) and it has a B2 option. It encrypts your data by default, and you can set up an intermediate backup (like an external HDD) to sit between your live system and B2 so you have multiple layers of backups.

If you're comfortable setting up a cron job, it's a great fit. I use it to back up a 1.5TB Samba directory and wind up paying about $5/m for B2.

[0] http://www.hashbackup.com/

treymo54• 6 years ago

I've been using Backblaze B2 + restic for automated backups of my working documents and photos. restic backups are encrypted by default and Backblaze B2 is one of the supported backends.

luma• 6 years ago

If your 3rd party storage provider plays any role in your encryption strategy then I would suggest that you’re probably doing it wrong. Data should be encrypted before it leaves your world and not decrypted until it comes home. That way, I don’t care a whit what they do to my data so long as it’s there when I ask for it back.

tuatoru• 6 years ago

My main concern with third party backups is the privacy aspect.

You want tarsnap.[1]

Edit to add: Colin Percival is the author of scrypt[2] and has worked extensively with FreeBSD's portsnap, so he knows what he's doing.

[1] http://www.tarsnap.com/

[2] https://en.wikipedia.org/wiki/Scrypt

guenthert2• 6 years ago

We have control over reliability as our systems are designed to deal with drive failure.

That surely assumes an upper limit of the likelihood of a drive failure. There was a perception that the quality of 3.5" floppy disks declined drastically in the early 21st century. Must we not fear something similar for spinning-rust hard drives once most everyone uses SSDs?

skj• 6 years ago

Briefly, any drive (floppy disk or tape drive) has some likelihood of failure. You can minimize loss of data (the reliability being discussed) by replicating data in more than one storage item. It just becomes a matter of how many you buy (and how good you are at keeping them all properly organized).

TrainedMonkey• 6 years ago

Hey Andy, many thanks to Backblaze for putting this together. Your stats are the first thing I looked at when looking for new NAS drives.

Since you mentioned cost, do you have data on average price per GB, both acquisition and electricity cost / year? With how many drives you guys bought there should be enough datapoints for a pretty neat graph.

andy4blaze4• 6 years ago

The average cost per GB is about $0.02/GB or $20/TB for the drive. Electricity varies based on the data center and the negotiated or local rates, so harder to calc. I like the idea of the chart, but may be tough to get the right data.

astrophysician2• 6 years ago

Hey Andy, I've got a question: I went to analyze your raw data dumps and saw a couple of things that I wanted to understand:

* A hard drive may go missing in the data from one day to the next without appearing as "failed" -- does this mean you've taken these hard drives offline for some reason?

* A hard drive may appear as failed one day, and then appear the next day with a failed=0 flag. Does this mean that these drives have been serviced and returned to the front lines?

mobilio• 6 years ago

I would like to thank you for putting numbers here too!

gruez• 6 years ago

Are there trends for AFR by drive age? ie. for a specific drive or manufacturer, what is the failure rate for drives that have been in use n years? It'd be interesting to see how the failure rate go up/down as they get older.

chiph• 6 years ago

Electronics typically follow a "bathtub" reliability curve. You'll have a large number of early failures (unflatteringly called "infant mortality") then the curve levels off for a long period of time, and then starts rising again as the items start wearing out.

https://en.wikipedia.org/wiki/Bathtub_curve

It'd be interesting to see if BackBlaze sees this in their drive populations.

astrophysician2• 6 years ago

I really enjoy reading the Backblaze analysis on this every year, it's such a valuable and interesting data set. I do have one suggestion: it would be great to go one step further and add confidence intervals for the AFR estimates. E.g. if you see 0 drive failures, you don't really expect an AFR of 0 (that is not the maximum likelihood estimate), and the range of AFR's you expect for each drive decreases as a function of the number of drive days (e.g. if one drive has 1 day of use, we know basically nothing about AFR so confidence interval would be ~0-100% (not really, but still quite large), or it would be smaller if you wanted to add a prior on AFR).

It would also be interesting to see the time-dependence (i.e. does AFR really look U-shaped over the lifetime of a drive?). That would require a dataset with every drive used, along with (1) number of active drive days, and (2) a flag to indicate if the drive has failed and of course (3) which kind of drive it is. Does Backblaze offer this level of granularity?

EDIT: They offer the raw data dumps!

https://www.backblaze.com/b2/hard-drive-test-data.html#downl...

Backblaze, god bless you.

bronco210162• 6 years ago

I've recently joined the WD shucking crowd so, unfortunately, there are no stats for the drives I'm running. The cost/GB has just gotten too ridiculously low on shucked drives and my array large enough that a failure or two isn't the end of the world, oh and 3-2-1 backup rule.

I've always really enjoyed reading Backblaze's reports though and have made past buying decisions based on their information.

I wonder if it might be possible for a community sourced version of these reports? A small app that checks SMART data and sends it to a central repository for displaying stats? Many from the self-hosted/homelab crowd are running these shucked drives so there has to be a large pool of stats out there if it can be gathered?

andy4blaze4• 6 years ago

Andy at Backblaze here. We've looked at this and the one "issue" is the consistency of the environments from the community data. In our DCs, the drives are kept a decent temperature, hardly ever moved, and our DC tech sing to them every night (OK, just kidding about that last one). Community drives will comes from all types of environments, from pristine to dust bunny hell. Still, it might be interesting to compare the two data sets if a community cohort could be collected.

touisteur• 6 years ago

Be careful not to sing too loud at them https://m.youtube.com/watch?v=tDacjrSCeq4

Incidentally I've tracked down a HDD (spinning rust) performance (and later breakage) problem to a very loud (and I mean awfully painful, intense vibration on your eardrums) alarm test every wednesday...

Do you monitor the noise in your DCs? Maybe for high peaks?

chromedev2• 6 years ago

What is Wasabi doing different where they're able to outprice B2, and get better throughput?

tln• 6 years ago

I'm guessing you work at Wasabi and already know :)

Just looking at the pricing pages, pure storage is cheaper at B2, but the API calls and egress are not free.

chromedev2• 6 years ago

No I don't. I'm genuinely curious how they might be saving money.

ggm• 6 years ago

The volumes of units are high enough I think this is north of 'too small sample size for statistical validity' So the question in my mind is, how significant is the variance in the failure rates?

Is this underlying manufacturing error tolerances, or is this shipping/deployment effects, or is this .. Aliens?

Backblaze Hard Drive Stats Q2 2020

Related Stories