HGST gets closer to shipping 10TB HDD

http://www.zdnet.com/article/hgst-gets-closer-to-shipping-10tb-hdd/

by tanglesome • 11 years ago

89 64 11 years ago

ars4• 11 years ago

Wow, hard drives are getting complicated!

You have to write 256MB to change a single bit?

And you have various zones, and you have to keep track of where data is written, because it's not written in order, and can be on any zone?

You would need some sort of battery backed scratch/cache area to pull this off, so that you don't have to write very often, otherwise I can't imagine performance will be very good.

the84723• 11 years ago

You would need some sort of battery backed scratch/cache area to pull this off

Or a copy-on-write filesystem. With some tweaking they can essentially fill one zone by linearly writing new copies (and metadata) to it while garbage-collecting (copying GC, with compactation) old zones in the background

ryan-c2• 11 years ago

Having maybe 16 to 64 GB of flash storage (for writeback cache) would probably make a significant difference here. I think over the next five to ten years tiered storage will probably go mainstream.

mtanski4• 11 years ago

Tiered storage is already here. If you're using block storage in the cloud those guys are already do all sorts of optimizations (for their costs) that include tiering. And on your own desktop / own server side you can do this today on Linux with bcache or dm-cache. Works great for databases.

gamegoblin• 11 years ago

You only need to rewrite the whole zone if you want to modify something that's before the write pointer. That is, you can write data sequentially without penalty. It's modifying existing data that's the issue.

fiatmoney2• 11 years ago

The 8 TB HDDs available now take a full 15 hours or so for a linear scan at max speed. God help you if you ever need to rebuild a RAID array made of them; the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high.

We're nearing the point at which the throughput relative to the capacity and the risk of failure or corruption makes further capacity less and less useful, unless you're doing some pretty sophisticated replication, validation, & caching.

the84723• 11 years ago

Is there some fundamental obstacle that prevents raid rebuild times from being within a small, constant factor of linear scan time of a single disk?

Naively I would think that it would simply have to stream large chunks of each drive in parallel, apply the parity coding and write out the stream of calculated blocks to the new disk.

fiatmoney2• 11 years ago

Oh sure, it's a big linear streaming read/write - the issue is that if you're taking a 8, 12, 15, etc. hour window to rebuild, there's a fairly high chance of something else going wrong. Particularly because you have to worry that, eg, your flaky power source has been shortening drive lifetimes, that's why you saw the failure within the expected lifetime of the drive, and that's why the another one is going to die in hour 14 of your rebuild.

Of course, that's in the context of an actual RAID array (or ZFS pool, etc.) - if you've got a replication strategy that spreads multiple entire copies over multiple systems, you're less vulnerable and these kinds of disks are a good choice.

MPSimmons• 11 years ago

Shingled drives are probably more likely used in a RAID-10 scenario where you're not recalculating parity, but just rebuilding linearly.

These drives (like the Western Digital Purple) are used predominantly for archival data. You should never have to randomly write to, for instance, surveillance video, or similar streamed media.

This isn't for your SQL database server.

mtanski4• 11 years ago

This is pedantic, but unless the filesystem is build specifically for WORM devices you always end up with some non-sequential data. In the use cases you're describing they'll be some filesystem metadata (freemaps, directory entries, inode tables) that will need to be rewritten.

hyperion2010• 11 years ago

Can't this problem be partially solved by having more read/write heads? I don't think it will fit in the same form factor, but if the limit is how fast you can spin the disk then having multiple independently controlled read/write heads per platter seems like one way out of them problem.

morcheeba• 11 years ago

seagate dual head design: http://www.tomshardware.com/news/seagate-hdd-harddrive,8279....

moe2• 11 years ago

the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high.

Two disk failures in the same array within 15 hours is very rare.

ghshephard5• 11 years ago

Unless the root cause is something to do with the array - flakey controller, or power supply.

vidarh• 11 years ago

Had it happen 3 times within a one month window (as in 3x double drive failures in the same array too close together for a rebuild to complete). It may be rare, but manage enough servers and sooner or later these things happens, especially if the cause of the failure is a power supply or the controller.

(We didn't lose any data - everything is replicated elsewhere in addition to the raid)

letstryagain3• 11 years ago

At scale everything happens, even very rare events.

DiabloD32• 11 years ago

I have to say this is false. I work in that industry, and even when you have, say, a 4 drive RAID, and all 4 drives are the same model but intentionally different batches, you will still, more often than not, have 2 drive failures within 24 hours.

Now, what does this mean in, exactly? In that four drive scenario, RAID10 has a 1/3rd chance of failure (both failed drives being both the A or B parts of the inner RAID 0s is two out of six possible scenarios), RAID5 will fail no matter which two drives, RAID6 will survive no matter which two drives, ZFS's RAID-Z3 (three parity RAID5/6) will survive no matter which three drives, and RAID1 will survive no matter which three drives.

RAID10 is the most performant of the possible outcomes, and the usual makeup of a 4 drive array unless you absolutely need the storage, then its usually RAID5; unless you seriously do not care about sanity at all, then its RAID0 or just independent disks.

Now, my suggestion for 15 hour rebuild times? Whatever you do, have a hotspare so it can begin immediately rebuilding and do not use any RAID variant that can't handle more than 2 failures in 15 hours. This means no RAID10 or RAID5, only RAID6 or RAID-Z3.

Ceph mitigates this 15 hour time because it can rebuild a lost drive by just allocating new blocks on every other drive simultaneously and maxing out your network and/or storage IO (whichever is slowest) so the window of potential doom is much smaller (depending on Ceph cluster size, obviously, bigger the better in this case).

jacquesm2• 11 years ago

Frequent enough that I've seen it twice on different arrays. The reason for this is that a drive failure can be caused by an external factor (usually multiple drives fed by the same broken power supply, but not yet so broken that it goes off-line).

TheCondor• 11 years ago

This is true of natural age type failures. When in a RAID every drive is effectively experiencing the same experiences, down to the writes and reads, if the failure is due to the drive's interaction with its environment then one failure almost always predicts more failures.

Natural age failures are pretty rare.

ghshephard5• 11 years ago

This is probably why people like BackBlaze use RAIN (Redundant Array of Inexpensive Nodes) instead of RAID.

Also, their drive failure is 10 disks/day out of a population of 44,100 drives, about 0.02%.

So, between the lack of correlation of failure (their Nodes, or, "Pods" are in different racks), their ability to lose an entire POD regardless (they have 3 parity pods), and the relatively low disk failure rate - the large drives aren't a problem. A rebuild in 2-3 days is more than fine.

sitkack• 11 years ago

There is a very interesting engineering optimization strategy at play here, would love to do/see a blog post on it.

Simple structures, with a large safety margin and simple control flow vs complex structures, very light weight but at the limits of capacity.

Like an ethernet switch doesn't need complex flow routing algorithms when the raw backplane bandwidth is some multiplier higher than will ever experience contention.

It always seems to me, that things built near their design limits will eventually expose a catastrophic flaw.

Human engineered items that have stood the test of time, Roman Aqueducts, Brooklyn Bridge, DC3, Dodge Dart, Toyota Corolla, AK47, PDP8 all have commonalities in their design philosophies.

eck• 11 years ago

the risk of an additional failure in say a RAID5 or RAID6 configuration is just too high.

Well there's one issue that another whole drive will fail and you're screwed.

The other issue is that modern disks have an unrecoverable read error rate compared to their size such that a total cover-to-cover read -- necessary on every remaining disk to rebuild a RAID5 -- is kinda unreliable, even with a supposedly healthy disk.

ghshephard5• 11 years ago

I'm interested - do you have a citation for that? I'm wondering if manufacturers of large drives accommodate for the statistically increased chance of a cover-cover failure (based on having so much data) by increasing their redundancy data/checksums to keep it constant.

static_noise• 11 years ago

Well there's one issue that another whole drive will fail and you're screwed.

You do know that in a RAID6 two drives may fail without causing data loss? In any way you need one or more hotspares ready plugged in in order to keep the time window short.

RAID6 is not perfect but the probability of 3 drives failing during the rebuilt window is much lower than the probability of 2 drives failing. (One also has to consider errors such as memory corruption, chip failures or catastrophic failure to the power supply where no RAID level will protect you from.)

A RAID1 built of two RAID6s may be necessary to avoid performance drops during rebuild. In the case of multiple failures a RAID6+1 setup will protect you from at least 4 hard drives failing in the rebuild time window.

The other issue is that modern disks have an unrecoverable read error rate compared to their size such that a total cover-to-cover read -- necessary on every remaining disk to rebuild a RAID5 -- is kinda unreliable, even with a supposedly healthy disk.

This is another reason for a RAID6. Not only does it recover when one or two disks fail. It also recognizes and recovers broken blocks when one disk returns the wrong data. You scrub the disk weekly and remap broken sectors or swap out (soon to break) harddrives.

epistasis3• 11 years ago

Systems like Ceph are great replacements for RAID. Instead of matching a handful of drives together in a set, each drive is a member of thousands of Placement Groups. When a drive fails, all the placement groups that a drive belongs to must be rebuilt, but this is spread over all the drives in your system, and not just a small subset. So if you have a 100 drive cluster, each drive does on average 1/100 of the work to rebuild. (Though using erasure coding amplifies the amount of work to be done.)

ghshephard5• 11 years ago

A Drive still needs 15 hours to rebuild though, so still the same amount of work. You just have more redundancy with Ceph, and don't have the ticking time bomb of losing an entire array if you don't complete the rebuild before another drive in the array goes.

Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives going down in a large array can wipe out all the copies of the object?

insaneirish• 11 years ago

So, yea, Ceph: https://twitter.com/tbmcmullen/status/566856092185923584

agumonkey• 11 years ago

I felt weird reading this. HDD are now for backup/capacity mostly, becoming CD-RW/Tapes of the day. Compared to recently shown Apple/Samsung SSDs going above 1GBps [1], feels even more out of place.

[1] http://www.macrumors.com/2015/03/11/13-inch-macbook-air-ssd-...

cdr• 11 years ago

Apple must not be using the top-tier Samsung M.2 SSD; they can write at 1500MB/s+ and read at 2100MB/s. 1024MB/s is nothing. Now that SSDs are moving to using PCI-E lanes instead of SATA, we'll probably see quite a speed race. Really looking forward to the Intel 750, which will straight up be a PCI-E card for the desktop version.

http://www.tomshardware.com/reviews/samsung-sm951-m.2-pcie-s...

pkaye2• 11 years ago

I see a new feature for SATA called Rebuild Assist becoming available to help reduce the RAID recovery times.

ars4• 11 years ago

http://www.hgst.com/tech/techlib.nsf/techdocs/CCA92CECECFCC7...

The way it works is by realizing there are only a few bad sectors on the failed drive, so rebuild that area first, the used the "bad" drive to help rebuild the rest of the array at full speed.

letstryagain3• 11 years ago

We stopped using RAID5/6 several years ago because of rebuild time and associated performance drop. Now with these massive disks we don't even use RAID10 any more! Every disk is just paired with one mirror and that's it. (RAID1)

The plan is to move to JBOD soon and just use software to store each file on 3 random disks across 3 separate servers.

jacquesm2• 11 years ago

That creates a substantial window during which you could lose the second drive.

Viper007Bond• 11 years ago

That seems slow. I have a 6 x 6TB RAID 5 array (27TB) for non-critical data storage and it takes less something like 22 hours to do its weekly parity check. I'm running firmware RAID via an onboard Intel controller on an ASUS board.

kondro• 11 years ago

Before people make too many comments about the complexity and speed of these drives, it is important to realise that these drives are designed for append-mostly archival of data.

toolslive• 11 years ago

One of the problem with these HDDs is that you still have the same number of spindles. These spindles now need to cover a lot more data, and they were already `challenged`

qt_scientist• 11 years ago

Yup, RAID is being replaced by object storage for large scale systems. Last week, HGST acquired object storage startup Amplidata...

mtanski4• 11 years ago

We spent a bit of time talking about about these "shingled drives" at the LSF/MM summit earlier in the week in Boston. Storage is becoming a more fascinating topic with so many technologies (SSDs, shingled disk, non-volatile memory) and so new open source solutions (Ceph, Luster).

atYevP• 11 years ago

Yev from Backblaze here -> gimme gimme gimme gimme!

HGST gets closer to shipping 10TB HDD

Related Stories