Farming hard drives: 2 years and $1M later

http://blog.backblaze.com/2013/11/26/farming-hard-drives-2-years-and-1m-later/

by uptown • 13 years ago

166 107 13 years ago

blog.backblaze.com

pcunite• 13 years ago

I personally (for home use) held off buying drives for two years, found out I did not really need so much space and discovered SSDs would do the trick.

dublinben4• 13 years ago

Same here. I bought my last 2 TB for $80 shortly before the crisis, and was able to scrape by at 90% capacity until 4 TB drives became cheap enough.

carlmcqueen• 13 years ago

I've been trying to reduce my data hoarding. I had a larger back-up hard drive go down and take a lot of data with it. I felt sick knowing I potentially lost years of funny images, videos, music, writings, etc.

What I found is I had back ups of back ups of the things I actually cared about and the things I lost I hadn't truly looked at in a long time. A lot of the music I hadn't listened to in years.

I have found that I can fit on small SSD drives and as a result of the things I do wish I hadn't lost, I'm just keeping up with the practice of keeping multiple back ups.

TL:DR: I have made an effort to keep a smaller data footprint instead of hoarding.

alexkus2• 13 years ago

If the only instance of a thing is on your "back-up" drive then it isn't a back-up.

TheLoneWolfling• 13 years ago

Agreed.

I currently have two external drives: an archive drive and a backup drive: both 2TB. (Most things on the backup drive are compressed and deduped, which is why I can fit everything.)

mikeash2• 13 years ago

Data that isn't backed up (as in a second copy) doesn't really exist.

Likewise: data that isn't backed up off site isn't really backed up.

post_break• 13 years ago

Why didn't they touch on why the Hitachi drives weren't the same quality as the other brands with this statement:

"Switching hard drive manufacturers from Hitachi to Seagate and Western Digital. The assumption here is that all drives are the same and as we’ve learned that’s not really the case."

Where is the footnote. Why aren't they the same?

tachion• 13 years ago

It doesnt implicate that Hitachi were not the same quality as others. I undestand that the main reason to switch was the availability to price ratio, nothing else. What I think they meant when talking about drives to be different is the variety of sizes you get when purchasing 'a 3TB' drive.

ewams• 13 years ago

In the parent article they said they switched because of cost: "When Hitachi drives got too expensive we switched to Seagate and Western Digital. When internal drives got too expensive we turned to external drives. When hard drive availability became an issue we turned to Drive Farming."

From their article on v2 they said: "The Western Digital and Seagate equivalents we tested saw much higher rates of popping out of RAID arrays and drive failure. Even the Western Digital Enterprise Hard Drives had the same high failure rates. The Hitachi drives, on the other hand, perform wonderfully." http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v...

There are other articles where they mention that they do testing and some brands perform better than others.

sard420• 13 years ago

We used to make PODs like this at work for netbackup storage.. started with WD drives ~128 WD drives, at about two years we were at ~8 WD drives still working. At least once a week we had people in there swapping drives. I have yet to see a WD not fail.

recuter• 13 years ago

Aren't they big enough at this point to cut deals directly with the vendors? 10,000 drives at a discount, that sort of thing.

SloopJon2• 13 years ago

In the previous story, user brianwski of Backblaze said, "We have been told that until we buy blocks of 10,000 drives, we cannot deal directly with the drive manufacturers. We're not there yet."

https://news.ycombinator.com/item?id=4633213

grecy• 13 years ago

Cool.

I know a few Backblaze people are reading/posting in these comments, so I'll ask here:

Do you guys know the approximate price of when you can purchase them direct in 10,000 blocks?

brianwski5• 13 years ago

brianwski from Backblaze here again - this reminds me to check once again on this info. We heard a rumor that the number of drives might be lower because we only buy the largest and most expensive drive. The 10,000 number might be a rule of thumb for a retailer like Radio Shack that stocks a blend of different drive sizes.

With all that said, EVERY MONTH we bid 20 different suppliers against each other and choose the lowest price. For goodness sake, anybody on earth who wants free money just has to undercut the other suppliers by 1 penny and you'll win our business. Always open to new suppliers, please, take our money!!

eterm• 13 years ago

But they claim to have added 50 Petabytes of storage in 2 years, which surely runs to over 10K drives per year.

It's a cool story but something just doesn't add up here.

ISL• 13 years ago

The original "Drive Farming" link is at least as interesting as the new article:

http://blog.backblaze.com/2012/10/09/backblaze_drive_farming...

SloopJon2• 13 years ago

And here's the discussion of that story:

https://news.ycombinator.com/item?id=4631027

I browsed the comments with two things in mind: was anyone uncomfortable with shucking and farming on this scale, and were the drives of equivalent quality?

Regarding the latter, user rsync of rsync.net claimed that "the drives inside these enclosures are the worst spec'd, highest failure rate drives." User budmang of Backblaze, however, said, "The drives were literally the same inside the external enclosures." User atYevP of Backblaze posted a similar comment.

jotm2• 13 years ago

High failure rates of external drives can also be attributed to the USB controllers - they're better than their IDE predecessors (which failed a lot, sometimes taking the drives with them - only low level utilities could revive them). Otherwise, I don't see how or why the hard drive manufacturers would bother to send the worst batches (which aren't that much worse) to external hard drive assemblers.

atYevP13• 13 years ago

Yev here ;-) -> http://blog.backblaze.com/2013/11/12/how-long-do-disk-drives... We posted this earlier in the month. The drives inside the external drives that were shucked performed pretty well. We were guessing that they would fail more frequently but that just didn't turn out to be the case. We're currently collecting even more data and hope to release it soonishly :)

sliverstorm5• 13 years ago

Interesting. I've never once found a standard drive model inside an external.

I mean, of course they are standard form factors, standard interfaces, etc- they are standard 3.5" hard drives- but they are always some odd model number I've never heard of with a 32kB cache, 1 month warranty, and 3600RPM. (slight exaggeration).

wayne_h2• 13 years ago

Data recovery guy here...

Failure of external drives has mostly to do with their handling.

They get bumped around while runnning.

People trip over the cords and knock them over.

They stack them like pancakes and they overheat. They block the fans and cause overheating.

Many enclosures are designed to look stylish - but don't cool well.

Heads fly at nanometers over the surface - they dont tolerate bumps well.

The drives we recover from external enclosures appear to be the same as bare drives that we recover.

michaelt• 13 years ago

The extrapolations in the article were pretty optimistic to begin with.

I mean, when you look at the dotted trendline in this graph: http://blog.backblaze.com/wp-content/uploads/2013/11/blog-co... you'd think by 2016 drives would cost $0 per gigabyte and by 2020, drive companies would be paying you to take drives off them.

minikites• 13 years ago

Just like the old razor blades graph: http://imgur.com/IBjL9tT

sitharus• 13 years ago

It scares me that we now have 6 blade razors: http://www.dorcousa.com/sxa1000-6-blade-system-for-men/

I'll stick with a single blade.

pedantry• 13 years ago

Well, at that point, if the current graph charts cost per gigabyte and approaches zero, then you would change the scale of the graph to show costs in terms of terabytes and eventually petabytes.

Ultimately, after traversing enough benchmarks, the scale would go logarithmic, and measurements would have to take orders of magnitude into account.

redacted• 13 years ago

Indeed, if you click through to the article the historical data is drawn from, you will see it plotted on a semilog plot. The trend is well described by an exponential function.

lsc• 13 years ago

you are going to get a more intuitively realistic trend-line mapping bytes per dollar rather than dollars per byte.

alayne• 13 years ago

This seems like something that should be part of their business instead of waiting for catastrophes.

dublinben4• 13 years ago

Since the price of storage is consistently declining, buying any more than necessary is wasteful over the long term.

clarky073• 13 years ago

yeah, but it seems like you'd give yourself more than a few weeks of runway. Don't have to buy 5 years worth, but perhaps 3-6 months would be reasonable. The peace of mind has got to be worth the ever so slight cost increase.

EGreg3• 13 years ago

Sadly I read this as a precursor to shortages in water, fish and fruits/vegetables that are coming. The populations of monarch butterflies, bees and other insects are dropping like a stone year over year. The fish near asia's costs are disappearing. The culprits include pesticides, overfarming, overfishing, and pollution. I am not even as worried about the garbage anymore -- scandinavian countries have figured out ways to burn the garbage and "contain" the waste gases. However, the main problem of capitalism is the numerous externalities that are exploited until it's too late to fix. It's not just global warming. It's raping the ecosystems for profit. No futures are going to save us from that. We need sustainable practices but I really don't see a way that'll happen at this point.

TrainedMonkey2• 13 years ago

Too fatalistic. In 19th century they predicted that Earth can't sustain population over 1 billion. In comes oil and agricultural revolution. I agree what we are doing with ecosystem in general is a crime, but it won't kill us. At least not anytime soon.

EGreg3• 13 years ago

Just because advances were able to stave off collapse and allow billions of people to consume resources at an ever greater rate does not mean that the same can be done with the effects of overfarming, overfishing, and pollution. The fallacy is thinking "since we did it once we can always continue doing it." The probability exponentially rises that you can't, since this is basically the same as the probability of always succeeding in time to avert a disaster.

Actually garbage / messing up ecosystems is a different type of problem than obtaining resources. It is somewhat related to the second law of thermodynamics. Once you introduce so much entropy into a system the genie is out of the bottle.of course this isn't exactly thermodynamics but I can say it this way -- putting humpty dumpty together again is different than figuring out how to make humpty give you more food for decades.

xenonite• 13 years ago

Yes, it is true that the modern agriculture was enabled through abundance of oil.

But that leads to a big problem as soon as the oil reserves diminish.

How should one not be fatalistic then?

tmp123415• 13 years ago

Try to imagine that this is the stock market: http://upload.wikimedia.org/wikipedia/commons/b/b7/Populatio...

shmulkey18• 13 years ago

I wonder why external drives are cheaper than internal drives. One would think that they would be more expensive, since the cost to the manufacturer of an external drive includes not only the drive itself but the adapter contained in the enclosure, the enclosure itself, cables, possibly a power supply, more elaborate packaging and the additional labor required to assemble the external drive. I assume that this consumer cost differential arises from differences in the markets for internal and external drives, but naively it seems like the internal drive market would be more price-sensitive and thus lower margin.

Does anyone know why external drives are typically cheaper than internal drives?

mangotree• 13 years ago

Maybe, with the advent of laptops becoming more popular than desktops, there is just a bigger market for them. Cost is usually due to demand constraints.

sliverstorm5• 13 years ago

I doubt it. Server farms consume an unreal quantity of drives. Desktops consume 1 drive per person.

bluedino• 13 years ago

They only have to QC them with the controller they ship with, instead of a hundred different PC's or motherboards. Not sure how much of a difference in cost that is.

sliverstorm5• 13 years ago

I will throw in that, from my experience, external drives usually contain lower quality disks. This is fitting; external drives are primarily used for sequential access on a sporadic basis. E.g., backups, movies, archives

wayne_h2• 13 years ago

I bet it has to do with the target market.

4tb external drives are something an end-user consumer would shop for based on price.

While a 4tb bare internal is shopped for by techies building systems.

atYevP13• 13 years ago

Yev from Backblaze here -> Personally, I think that the biggest part of the is price sensitivity. External drives are geared towards consumers, who simply will not buy the drives if they go up too much in cost. The internal drives we were buying are targeted at business' who have a choice to make, either buy at the higher price, go out of business, or get creative. We try to go the go creative route because we really don't want to go out of business, and paying the higher prices for drives, especially in this case, would have gotten us there.

Backblaze tries to avoid the "enterprise tax" as often as we can, and internal vs. external drives are just another example of how "enterprise-grade" items get marked up because the target market will pay for it.

tedivm6• 13 years ago

Why do they need hard drives when they don't even back up data?

Seriously, you can find reports of this all over the internet. It's on reddit, twitter and elsewhere- these guys do not back files up properly, and their systems are so shoddy that they'll report they have files even when they don't.

My girlfriend brought her computer in for repair and they changed the hard drive. She specifically checked before she left that the backups were set, and the dashboard said they were. When we went to restore, however, none of the files could be.

We contacted support and they kept insisting we did something wrong. Finally they admitted that they had erased the files (yes, actually erased them). Due to a "bug" their dashboard reported we still had the files, even though they were gone.

They did everything they could to hide their mistake. They lied to people on twitter about what I was saying (since the only way to get real support was to hound their twitter account). They lied about all sorts of things. I had to take screen shots of everything and post it online before they'd even admit to screwing up. Even at that point their dashboard was still saying it had all of these files, when it in fact had nothing.

Backblaze is a completely untrustworthy company. If they spent half as much effort on backing up files as they do on marketing that may be a different case. Anyone who uses them for backups might as well not have backups at all.

kalleboo2• 13 years ago

The second rule of backups is, always test your backups. Never trust a cloud service, backup software or tape drive status. Test a restore periodically, or you can't be sure you're backed up at all.

bruceboughton• 13 years ago

Very true, but victim blaming gets us nowhere.

atYevP13• 13 years ago

Hey Tedivm! Yev from Backblaze here -> It's true that the data was removed from your account. After your trial lapsed and was not paid for, we removed the data after 7 days as is our standard practice. Unfortunately in your case the overview page did not update to show that you had no data associated with your computer, so when you purchased the license, there was no data on the machine that it attached too, so when you erased data from the machine while performing maintenance, you assumed the data was on Backblaze, when it was in fact purged from our system a few weeks prior.

We've since fixed that visual glitch (if you have a trial that expires and is not paid for, once the data has been deleted, that will be reflected on the overview page) AND we're currently working on some other notification systems to help people avoid losing data. As always, I apologize that you had this problem, but I'm happy to inform you that we've since fixed that visual bug on our end.

tedivm6• 13 years ago

So everyone else is clear, we did pay for the account after the trial ended. So we were paying customers, this wasn't an incident where a trial expired and we tried to get our data back.

Also, to be clear, the customer service experience was horrible. The support team repeatedly ignored what we were saying and send us form responses back which were just not relevant. The twitter team claimed we misread the retention policy. The whole thing was kind of ridiculous.

Even more to the point, and the main reason why I tell this story, is that people should know how this company operates. People should know that they don't have backups of backups, that there is no margin for error should your account lapse, and that their systems in general are not robustly programmed in the sense that there are ways that your data could be lost and you wouldn't know it.

From my perspective Backblaze only has one thing going for it over competing companies, and that's it's marketing department.

icelancer• 13 years ago

How did you compensate tedivm for his issues, and how do you plan on fixing the support-related issues tedivm brought up, like your CSRs lying about the problems?

sliverstorm5• 13 years ago

I don't know a whole lot about backblaze; as they are a backup service, is there a good reason why they couldn't ramp up usage of tape drives? Perhaps keeping recently uploaded backups on-disk, and shuttling older ones to tape?

petemc_• 13 years ago

Yes. Tapes and tape drives are expensive and awkward to use.

hrrsn3• 13 years ago

You also have to store them, test them and then find them when a customer wants their data. With drives in a pod they're always online and in a RAID so when a drive dies they can be alerted quickly and customers can access data essentially 24/7.

smackfu• 13 years ago

Their backups are progressive, so a single disk image might be spread across 100 tapes. A restore would be a nightmare. Tapes really only work if you are doing a full image each time.

JoeAltmaier• 13 years ago

or a program of partial backups interleaved with full backups, say 3-to-1. You balance a reasonable restore time with a reasonable storage requirement.

ghshephard2• 13 years ago

I have a colleague who works in the industry and purchases a few petabytes of spinning disk a quarter. I love reading the backblaze blogs, and constantly forward them to him. He thinks the backblaze guys are pretty crazy with their "shucking strategy", finds it hard to believe seagate/WD guys will honor warranties on those "shucked" drives, but if they will, all the more power to backblaze.

He does note that his costs per drive didn't go up dramatically after the Thailand floods, and they were pretty close to what backblaze was playing for their "shucked drives" - though, they didn't go down signficantly either.

I'm guessing Backblaze is still at the volume levels where they can't negotiate directly with Drive Vendors - I'm not sure where that level is - 10 Petabbytes/Quarter? 100 Petabytes/quarter?

Regardless, I continue to be very impressed by how backblaze manages to circumvent what others might consider insurmountable hurdles. Truly hacking the system in a good way.

atYevP13• 13 years ago

Yev w/ Backblaze here -> Yea unfortunately when we shuck the drives the warranties are voided, but we add that factor in to the costs that we are willing to pay for the hard drives. Truth was, it was cheaper for us to buy the externals and shuck them, even with the voided warranty, than pay the larger fees for internal drives.

Part of the is price sensitivity. External drives are geared towards consumers, who simply will not buy the drives if they go up too much in cost. The internal drives we were buying are targeted at business' who have a choice to make, either buy at the higher price, go out of business, or get creative. We try to go the go creative route because we really don't want to go out of business, and paying the higher prices for drives, especially in this case, would have gotten us there.

As for the negotiation, BrianWski mentioned it in another reply on here, but we keep getting weird answers from vendors and manufacturers. Typically they want a minimum order of about 10,000 hard drives in order to work directly with them and we're simply not willing to buy that many at a time, especially since normally the price goes down monthly, meaning carrying inventory is leaving money on the table.

Love that you enjoy your blog posts, stay tuned ;-)

digitalzombie• 13 years ago

Still... no linux.

pwg• 13 years ago

When the Drive Crisis started, industry pundits estimated that the hard drive market would take anywhere from 3 months to 1 year to recover. No one guessed two years. Was the delay simply an issue in rebuilding and/or relocating the manufacturing and assembly facilities?

This is a typical response in all markets that are commodities (or approaching commodity status). Once makers have had an excuse to move prices up, they want to hold onto the extra profit for as long as they can, so prices are fast to rise, but slow to fall.

The most direct effect can be witnessed in gasoline prices. Notice how a news report that a camel farted in the desert of Saudi Arabia results in the price of gasoline at your local station rising by 25 cents a gallon (or more) nearly instantly. Even though the gas in the underground tanks was the same gas (at the same cost to the station) that it was 15 minutes ago.

But then, it takes weeks before that 25 cent increase goes away when no more camel farts occur over those same weeks. Same effect here, just with hard drives rather than gasoline.

brianwski5• 13 years ago

Six months after the Thailand drive crisis, drive prices "peaked" at about double their earlier prices and began falling. Just about then, an article came out featuring a bunch of panic inducing quotes from a Seagate VP loudly claiming it might be a year before they could get enough drives made, shortages would be terrible. It felt so disgusting to me, none of it was true, the VP was just trying to encourage pre-buying and drive hoarding to artificially pump sales.

Farming hard drives: 2 years and $1M later

Related Stories