andy4blaze

Andy Klein from Backblaze here. We have done some survival analysis (kaplan-meier curves). In our case, we need to have a reasonable number of failures over the observation period to get decent results. You can take a look at some of our work here: https://www.backblaze.com/blog/hard-drive-life-expectancy/ and see if that is what you were expecting.

Andy Klein from Backblaze here. Your point is a good one in that temperature fluctuation can be an important factor. We actually sample smart stats, which contain the temperature attribute, multiple times a day looking for such changes. The Drives Stats data is captured once a day, so it looks static, but behind the scenes the monitoring is more dynamic.

Andy from Backblaze here: The WDC model (WUH721816ALE6L0) seems to only be available in Europe, not the US. It was 549,00 EUR last time I looked back in November. I believe the last digit is about data erasure with 0 being "Instant Secure Erase", 1 being Bulk Data Encryption (SATA), 4 being Secure Erase (overwrite only). We only have reliability data for the model listed.

Andy for Backblaze here: I looked at that 3-4 years ago. It looks like power cycling increased failure rates, but we don't power cycle our systems very often, maybe 1-2 a year, so not the best use case. This is on my list for a relook one of these days, if we find anything interesting we'll let folks know.

Andy for Backblaze here: A while back we did an analysis of drive failure over time, i.e. the bathtub curve. It is probably a good idea to update that, as I believe we are seeing lower failure rates upfront these days.

Andy at Backblaze here. We do look at drive model failure over time. We did a post on this topic several years ago, 2015? At the time, most drive followed the bathtub curve of failure, but I'm not sure that is still the case. I think its time to update that report.

Andy from Backblaze here. Larger drives do take longer to rebuild, but to date we haven't changed the encoding algorithms we built. There are other strategies like cloning which can reduce rebuild time. We can also prioritize rebuilds or drop a drive into read-only mode as needed. The system was built expecting drive failures.

Andy at Backblaze here. All the drives are in data centers with temps around the 75-78 degree mark. Vibrations are kept to a minimum via the chassis design. We publish the data, including the SMART stats for all of the drives and there are attributes for temperature (SMART 194) and vibration (multiple) see https://en.wikipedia.org/wiki/S.M.A.R.T. for more info in SMART attributes.

Andy from Backblaze here. Actually you can download a spreadsheet with all the data from the tables. There's a link at the end of the post. Better than parsing HTML for the data.

Andy from Backblaze here. Nice thinking about the bulk ordering and considerations for RAID. All things we have considered. We use our own Reed-Solomon encoding with a 17/3 set-up across 20 drives across 20 different systems, we call that a Tome. Then we have a specific protocol we follow as drives fail in a Tome to protect the data at all costs. We have the luxury for example to stop writing to a given Tome as we have plenty of others available. This takes a lot of the stress off of the system. Your thoughts on bulk buys and bad drive batches/models is solid. We test drives in small batches first, and we follow drive failures so we don't get to the point of hitting the wall. It would be great to mix and match drives, but you end up with a system that maxes out at the least performant drive. So not optimal.

Andy at Backblaze here. We've looked at this and the one "issue" is the consistency of the environments from the community data. In our DCs, the drives are kept a decent temperature, hardly ever moved, and our DC tech sing to them every night (OK, just kidding about that last one). Community drives will comes from all types of environments, from pristine to dust bunny hell. Still, it might be interesting to compare the two data sets if a community cohort could be collected.

The average cost per GB is about $0.02/GB or $20/TB for the drive. Electricity varies based on the data center and the negotiated or local rates, so harder to calc. I like the idea of the chart, but may be tough to get the right data.

Andy at Backblaze here. To put a pin in it, the three main factors for which drives we use are cost, availability and reliability. We have control over reliability as our systems are designed to deal with drive failure. That leaves the market to decide on cost and availability. Assuming a competitive market we can buy the drives that optimize those factors.

Andy at Backblaze here: Good to know. We used to do this with the lifetime stats, but that got lost somewhere along the way. I'll look at getting them back. Thanks.

Andy at Backblaze here. We use SSDs in our core servers and more recently in boot drives as they both need to speed. To store data in our case we don't need the speed so its not worth it yet. Given the amount of data growth, most predictions have HDDs still with about 50% of the storage market in 2025.

Andy from Backblaze here: The capacity of the boot drives ranges from 80 to 500gb typically. They are mostly hard drives with some SSDs added recently. We are switching over to SSDs for boot drives. The workload is reasonable, but on the higher side as they not only boot the systems, they also are used to store log files temporarily - so lots of reads, writes, and deletes. Since we only have a little over 2,000, any data we published not be very accurate. If you are really interested, the boot drive data is in the data files we publish each quarter.

Andy from Backblaze here. Thanks for pointing it out. The correct range is low: 0.5% to high: 1.1%. We have updated the chart.

In the Backblaze case, they are securely wiped clean and then recycled.

I suppose if you had a single 32TB drive that went offline for say a week, and then once it came back online you'd have some type of pent up demand and a "slow" transfer speed. Storage systems in general spread the load across multiple devices so the effect of slow transfer speed is near zero in most backup and archiving applications and maybe more problematic in transactional applications.

Website is fine, Backup service is fine. Blog is overloaded and we're working on it.

The Backblaze Vault design mitigates that as the "raid array" is scattered across 20 different Storage Pods in twenty different racks. You'd need more than three racks to go down before you would be offline. Backup systems in place make that highly unlikely. Andy at Backblaze.

The Botezatu et al paper is quite good. The recommendations they make are well worth further consideration in our environment. -- Andy at Backblaze.

Besides stock management, they help use determine the overall health of a Storage Pod or Vault. They also help find trouble with other components. For example, if a backplane or cable were failing, the drives via their SMART stats may notice first. So they SMART stats are part of what we use to evaluate the whole system health.

Nice analysis.

Andy from Backblaze here: Here's the post we did about that if you're interested: https://www.backblaze.com/blog/crowdsourcing-hard-drives/