Our 6 TB Hard Drive Face-Off

https://www.backblaze.com/blog/6-tb-hard-drive-face-off/

by ehPReth • 12 years ago

204 85 12 years ago

EwanG• 12 years ago

For those not willing to read a full page, in the end run it is more an energy decision than a reliability decision. And they mention at the end that they are already looking into the 8TB Helium and 8TB SMR drives.

pbreit• 12 years ago

My sense was that the 20% speed advantage was as important or more than the small energy savings. Is that common for slower drives to handle data faster?

pfg• 12 years ago

Maybe it's a side-effect of increased vibration for 7.2k drives? Hard drive performance drops in environments with high vibration. I could imagine that being a factor when you put 45 drives in a single server.

kijin• 12 years ago

Anecdotal evidence, but I often find WD drives to be faster than their Seagate cousins, even at lower rotation speeds. I wonder if this is because of better firmware optimizations or because of higher platter density. Maybe it's both. Or maybe WD lies about their rpms ;)

Hitachi/HGST, now owned by WD, also makes really fast drives. Their 7200rpm 2.5" drives are the fastest laptop drives I've ever used, except SSDs.

miahi• 12 years ago

The WD they use is a 5-platter HDD (1.2TBx5). The Seagate has 6 platters (1TBx6). The higher density means that the linear reads/writes are faster at the same spindle speed (more data passes under the heads in the same time interval). The interesting thing is that they are faster even at different speeds.

fubarred• 12 years ago

Backblaze is pretty awesome for open-sourcing, effectively, their costs in a high-competition segment. It would be difficult (read: pointless) to compete in that market without massive vertical integration, greater scale, deep pockets and compelling, differentiated value (eg something different enough besides all the other entrants).

For anything critical in business (customer list, billing, finance, etc.), I would recommend also using Tarsnap in addition to whatever other (non-AWS) vendor/s are used for "offsite"/"offcloud" backup.

java-man• 12 years ago

Firefox: Unable to Connect Securely Firefox cannot guarantee the safety of your data on www.backblaze.com because it uses SSLv3, a broken security protocol. Advanced info: ssl_error_no_cypher_overlap

can we trust them with our data?

warp• 12 years ago

Peculiar. According to https://www.ssllabs.com/ssltest/analyze.html?d=www.backblaze... they only support TLS 1.0, not SSLv3. (Though I'm still surprised by how many issues SSLLabs identifies on that site... )

Someone1234• 12 years ago

They get a "B," Microsoft gets a "B," Google gets a "B," Bank Of America get an "F," and so on.

Nothing unusual about their results.

theandrewbailey• 12 years ago

I thought it was just me (I've disabled some older cipher suites in about:config). Turns out this site only supports broken ciphers (RC4 and 3DES). Ewwww.

https://www.ssllabs.com/ssltest/analyze.html?d=backblaze.com

yuhong• 12 years ago

3DES is not broken but it is legacy.

Bluestrike2• 12 years ago

The article mentions having to buy 10k+ new drives in the next few months? Is that number just your average increase for new drives and/or replacements for failed drives? Or does it include planned replacement of older pods with smaller drives? I'm curious if there's a point where you've found it less expensive (or less problematic with drive failure rates per pod) to replace and upgrade older pods compared to just letting them continue to run.

bluedino2• 12 years ago

They used to say they don't spend any money on drive replacements (except for labor, obviously) because of the drives having 3-year warranties. Those were using 1.5TB drives. However, the original 67TB pods came out in 2009 and wouldn't be warrantied anymore.

According to their blog they took until 2011 to hit 10PB, which would be roughly 150 of the original 67TB pods. The 6TB drives are 4 times better, so they'd be able to replace all of those with just 40 of the new pods, I'm sure the savings in electricity and rack space would be too huge to pass up.

Then again, they have their own datacenter space now, so space probably isn't an issue but I'm sure electricity is. They said something around $2,100 a rack (which holds 10 pods) for electricity, space, and data. They were incredibly stoked to cut that in half by moving to 3TB drives with the 2.0 version of the pod so I'm sure cutting that in half excited them as well :)

brianwski11• 12 years ago

Backblaze employee here. Some of the drives are "voided warranty" now, like the ones we "shuck" out of USB enclosures. We just include that into our cost analysis - for example if 5 percent of the drives fail, the warranty saves you 5 percent of cost - so if purchasing the warranty costs 3 percent we do it, but if purchasing the warranty costs 7 percent we do without.

brianwski11• 12 years ago

Also, we still pay "per rack" but prices are now a fraction of the $2,100 (I'm not sure if I'm allowed to say exact numbers).

brianwski11• 12 years ago

Backblaze employee here. We have about 900+ pods, each with 45 drives, so that makes a little less than 45,000 drives. We publish our annual failure rates, but let's say blended between drive types it's like 2-3 percent. Think we replace 4-6 drives every day?

But the main reason we buy drives is we deploy 30 new pods a month right now, and it's increasing. So we deploy 1,400 drives a month or so in new pods. Sometimes we buy in 3 month batches (to get a break on price) so then you start talking about a 5,000 hard drive purchase in one shot.

At $260/drive, you can see how we care deeply about EXACT failure rates. Why isn't Google and Amazon publishing failure rates also? Why only Backblaze?!

justcommenting3• 12 years ago

the critical assumption in this post seems to be that drive speed was the only rate-limiting factor when they were collecting data... but as i understood the post, it seemed like perimeter load-balancing, sata controller/nic load, and/or customer usage patterns could be responsible for some of the differences they observed.

lsc3• 12 years ago

You have to screw up a storage system pretty good to have something other than the hard drives be the limiting factor.

asdfaoeu• 12 years ago

True but that doesn't imply you run at capacity all the time.

justcommenting3• 12 years ago

this is generally true, but even small variations in other factors could propagate through a complex architecture to account for some of the differences presented here, especially over time. these data are still useful, but some of the caveats could be made more explicit.

similarly, most drive benchmarks are 'artificial' in any number of ways and come with another set of caveats. since drive testing is already part of backblaze's process for putting drives into production, it could be especially interesting to compare and contrast performance data from testing with data from production use.

i pick these nits because real-world storage performance info seems like a valuable public good (particularly for enterprise and enterprise-like usage of consumer drives), and i'm grateful to backblaze for offering it up.

SamReidHughes• 12 years ago

It's pretty easy to have software to be the limiting factor, such that you aren't taking full advantage of your throughput because the data's not there yet.

Tepix2• 12 years ago

Exactly. I found this particular blog post weaker than their previous, highly interesting ones.

lsc3• 12 years ago

hm. I wonder if their data access patterns are largely sequential; I imagine the 7200rpm drives would beat the lower-RPM drives in an environment where random access matters.

wmf• 12 years ago

It's a backup company. Write one, read never.

justcommenting3• 12 years ago

i have similarly wondered how important drive cache size might be for these types of use cases. multivariate regressions including brand, cache size, rpm etc. as predictors of various performance metrics would be super interesting in a place like blackblaze.

lsc3• 12 years ago

Cache gets you very little if you can feed in sequential data at a constant rate and at the blocksize of the device.

Cache becomes more important as the input data becomes more messy; cache is one tool for turning bursty and random data into constant, sequential data.

benguild• 12 years ago

It’s really great that BackBlaze does these real world, mass comparisons. Otherwise it’s just limited to what a single test says, often at a single time, with a single batch of drives from various manufacturers.

These guys are probably one of the only qualified places to do this at scale and report real world results. Glad they do!

emsy• 12 years ago

I often think of it as open sourcing a company. They give up a competitional advantage but probably gain as much or more in return (reactions on their article as "pull requests" for instance). I don't want to sound hyperbolic but I think it also advances humanity (The information and the spirit behind it).

Other companies do talks and papers too, but the Backblaze info seems really unfiltered and close to the real world.

Ironchefpython• 12 years ago

the Backblaze info seems really unfiltered and close to the real world.

Cloudflare seems to be a similar company. I recommend them both to anyone somewhat technical who needs a CDN.

My only complaint about Backblaze is no Unix client. I thought about setting up a Windows VM to manage backups, but I was terrified that Gninjas would sneak into my house at night and shave my beard.

tzz• 12 years ago

Backblaze should expand their service to a cheaper cloud storage service similar to Amazon S3. They already have the infrastructure and the know-how.

dsl2• 12 years ago

The business model is based on only rarely having to retrieve old data. I suspect they spin down the drives (or possibly hibernate the whole box) when not in active "load" mode.

You would need to compare the pricing here to Amazon Glacier, if anything.

brianwski11• 12 years ago

Backblaze employee here. Our drives never spin down, although we have considered this because electricity is a large monthly bill to us.

Because it is "backup" we are pretty relaxed about shutting down our pods to do maintenance, like replacing a drive. But that only takes 10 minutes, we try not to have machines offline more than that if possible, or it causes support issues from people trying to restore files.

venomsnake• 12 years ago

Backblaze tech is not that impressive to give them competitive edge over the economy of scale of the big boys have. Amazon & co are not selling at cost, but at whatever the market with bear.

patrickg_zill• 12 years ago

AS cool as it sounds, they probably feel they should continue executing on this very focused strategy first. They can branch out later.

ck2• 12 years ago

Helium filled drives are literally a dead-end as their lifespan can be shorter than SSD

ghshephard5• 12 years ago

Re: "Helium filled drives are a dead end" - Citation?

Also, SSDs come with 10 year warranties [1], so, which exceeds anything you can get with a normal spinning HD. If anything, I'd have more confidence with my SSD than I would with a spinning HD - particularly as you don't have to worry about the drives failing to spin up over time.

[1] http://www.amazon.com/SanDisk-Extreme-2-5-Inch-Warranty--SDS...

dsl2• 12 years ago

One legitimate argument against helium drives is they contribute to the depletion of the relatively scarce helium supply.

http://www.boston.com/news/science/articles/2010/10/17/scien...

jedberg• 12 years ago

In a rack of multiple drives, their short lifespan isn't a problem if the cost model works out. Also, citation on that lifespan claim? They haven't been around long enough to know.

unicornporn2• 12 years ago

I'm so looking forward to the Seagate 8TB Archive HDD. Finally a drive that will hold all my photos and quite a bit more. It will make my backup procedure a whole lot easier.

medecau• 12 years ago

It's also a single basket for all your eggs.

ghshephard5• 12 years ago

As the original poster noted, sometimes it's far better to have a single basket for your eggs, that you keep well backed up, than it is to have multiple baskets, none of which are backed up.

pjc50• 12 years ago

Related ask HN to you and others: what software are you using to backup/sync to that drive? (What would you use on Windows?)

flyinghamster• 12 years ago

I can't speak for Windows, but the setup where I work involves four drives (currently 2 TB each), with three running as a triple-mirror ZFS pool. For backup, I split a drive off the pool, pull it, slide in the previous backup, and resilver the newly-inserted drive. Each time, I use a different drive bay.

The ZFS pool itself provides backup for several other systems, and snapshots are made periodically by a cron job to provide a further line of defense.

arm• 12 years ago

I use OS X on my main computer, so I haven’t used this personally, but Bvckup 2 was mentioned on HN in the past:

https://news.ycombinator.com/item?id=8027405

unicornporn2• 12 years ago

Rsync. I don't think I'd recommend that on Windows though.

IgorPartola• 12 years ago

Here's what I use, though with different drives: Ubuntu + ZFS + rsnapshot. Works well, mostly declaratively configured, and ZFS means I don't have to worry about bit rot. It has its gotchas, for sure, but if you want your data to stay intact, this is the thing to use.

SideburnsOfDoom• 12 years ago

No failures recorded at all. This is pretty impressive for the drive manufacturers:

"Let’s review the Seagate and Western Digital drives so far:

Initial reliability (how many drives failed) – No failures.

Running reliability (3 months) – No failures

SMART Stats (3 months) – No error conditions recorded for the 5 stats that we utilize."

Cthulhu_2• 12 years ago

Of note is that since Backblaze is a backup company, their hard drives will probably not see a lot of churn / activity - write once, wait until something happens. Maybe refresh the data every once in a while.

I'd like to see similar tests from companies like Amazon and Dropbox (although the latter probably uses the former), in various use cases.

chrisan• 12 years ago

Wouldnt they see the same amount of churn a typical user would see?

I do not use them but they say "That’s why Backblaze backs up your data automatically and constantly looks for new and changed files to backup. Install Backblaze and never worry about losing a file again."

Unless you are running a database it seems like the most common use case for files is write once, update every once in a while. Document creation (code for us or office docs non-coders) would be a bunch of writes up front then follow the same pattern of update every once in a while

I suppose their reads are much less than a typical user, but I didnt think reads contributed much to eventual drive failure

rwmj• 12 years ago

I wonder if anyone has real world experience of Shingled Magnetic Recording (SMR) drives? I saw a talk by Red Hat's Ric Wheeler about a year ago, and they seem both interesting and pretty strange - as in, we may need new filesystems to cope with them.

brianwski11• 12 years ago

Backblaze employee here: we will run our first experiment with SMR soon! We're SUPER curious about whether they will work in our application.

yc10102• 12 years ago

Does anyone know if there is a set of command line tools (linux) or an api for uploading, downloading and listing files on your account or do they specifically forbid this type of use and obfuscate?

e12e2• 12 years ago

I've been hoping for Backblaze to launch a "pro"/"enterprise" tier that allowed for this kind of use. Unfortunately (for me) it would appear they prefer to stay within their chosen niche: home user backups.

The service I'm aware of that is closest to what you describe is spideroak[1]. Last I used them (years ago) they were a little too slow, and had some other minor kinks -- but might be worth checking out again now.

It looks like they've also finally made good on delivering Nimbus.io -- both as source and as a service (as I said, it's been years since I looked at Spideroak...):

https://nimbus.io/

https://github.com/SpiderOak/nimbus.io

[1] https://spideroak.com/

yc10102• 12 years ago

So its not really "unlimited" storage they offer, there is a ceiling that matches the size of a users primary hard drive (max 6-8 TB at present?) with most users probably using laptops with small drives. For example I would love to backup 3-4TB on my NAS, and be able to occasionally access+download and check integrity

Tepix2• 12 years ago

It's unfortunate that Backblaze feels the need to transfer the private key to their servers.

Their knowledge base entry at https://help.backblaze.com/entries/20203731-Can-you-tell-me-... doesn't clarify wether or not the secret private password is transmitted to their servers to decrypt the private keys there which would make the scheme rather weak.

Ideally, the private keys would remain on the user's PCs.

foldor• 12 years ago

While I do agree that private keys and transparency is important, I feel they almost need to store the private keys as well. But they should be password protected by something only the user knows, and the password shouldn't need to be transmitted over to their servers.

The reason they would need the private keys should be obvious. They are a data disaster recovery service. In the event of a disaster where all of your drives fail, or are inaccessible you would still need access to your data. So if the only copy of the private key isn't on their servers you would be shit out of luck, not exactly a good situation for a company like theirs.

But yeah, store the private key, but make it encrypted by a secure password only decrypted locally, and update documentation to make this clear.

twoodfin• 12 years ago

But yeah, store the private key, but make it encrypted by a secure password only decrypted locally, and update documentation to make this clear.

I think even this is "too secure". I forget my 'secure passwords' regularly, and I'm sure I wouldn't be the only one. It's not a huge improvement over telling a user to print out a private key and keep it somewhere safe.

Given the likely threats to Backblaze, I'd encrypt user backup private keys with a threshold or other multi-key scheme. No individual Backblaze employee's key would be sufficient to decrypt a backup; you'd need the active participation (and private knowledge) of at least a few. That prevents individuals from snooping, and requires compromising several employees to decrypt backup data.

brianwski11• 12 years ago

Backblaze engineer here. The private encryption key is a completely standard PEM file from OpenSSL. We have two modes: by default the PEM file is secured by a "pass phrase" known by Backblaze. In this lower security mode, the user data is essentially only secured by their email address and their chosen password on the Backblaze website, and the user can recover their password via email. That is a great mode if you are just backing up stuff you don't terribly mind somebody getting a copy - like photos of your cat.

If you are concerned about security, the second mode is you set your own "pass phrase" on the PEM file. But oh my lord, make sure you remember that pass phrase because Backblaze never writes it to disk and if you forget it, NOBODY ON EARTH is getting your files. Not you, not the USA government, not Backblaze, that data is GONE. You cannot recover that password. This is a good security mode if you would be arrested if the NSA got a copy of your data, because we don't think it can be broken.

bumbledraven• 12 years ago

Even in "secure mode", your passphrase must be sent to backblaze's servers whenever you do a restore. They then have posession of decrypted version of your files for a while. This is unacceptable.

As brianwski wrote in https://news.ycombinator.com/item?id=8169040:

...[I]f you lose a file, you have to sign into the Backblaze website and provide your passphrase which is ONLY STORED IN RAM for a few seconds and your file is decrypted. Yes, you are now in a "vulnerable state" until you download then "delete" the restore at which point you are back to a secure state.

plus• 12 years ago

Unfortunately not the most direct apples-to-apples comparison. The WD Red drives are designed for use in NAS systems, whereas the Seagates used are desktop drives. Seagate does have a brand of NAS drives (called simply enough Seagate NAS), though not apparently at 6TB.

sp332• 12 years ago

It's not saying that the WD drives are better in all cases. Backblaze has a specific use-case and the WD drives are the best fit.

Our 6 TB Hard Drive Face-Off

Related Stories