Farming hard drives: how Backblaze weathered the Thailand drive crisis

http://blog.backblaze.com/2012/10/09/backblaze_drive_farming/

by zorlem • 14 years ago

187 100 14 years ago

blog.backblaze.com

If BackBlaze are adding 50TB of data a day, are they really buying drives from NewEgg?

zorlem4• 14 years ago

That's one of the things that caught my attention as well. It seems strange, I would've imagined that at 50TB/day (~500 x 3TB HDDs/month) they could get a nice deal with a direct importer. I guess there is a small PR twist to the whole story ;-) with newegg being a well-known name.

I don't know if the 50TB/day figure includes the replacement drives or only expanding the capacity as I guess their failure rate is quite high.

edit: fixed a typo.

bruceboughton7• 14 years ago

My guess would be that the 50TB/day figure is before de-duplication and not at all relevant to the actual storage, but even at 10% of that I would expect them to be sourcing them wholesale, not retail.

In fact, this whole article is making me doubt my faith in Backblaze as an off-site backup solution.

smackfu• 14 years ago

Why not let NewEgg hold the inventory if their prices are good?

mbell• 14 years ago

NewEgg isn't just retail anymore. They have a 'NewEgg Business' program with pretty solid volume discounts.

michaelt2• 14 years ago

A few months ago I wanted to buy 1000 Microsoft Lifecam Cinema HD webcams. I could order them individually from Amazon for £27 each with free delivery.

I phoned up all the official Microsoft distributors in my country, who phoned up Microsoft and whatnot. Not one of them could offer me a price below about £35. And getting in contact with Microsoft hardware directly? Ha, I should be so lucky.

andrewf• 14 years ago

A few months ago I wanted to buy 1000 Microsoft Lifecam Cinema HD webcams.

I'll bite. Why? :)

corin_2• 14 years ago

More often than not it's a case of who you know when it comes to stuff like this.

brianwski4• 14 years ago

(Backblaze engineer here.) We have been told that until we buy blocks of 10,000 drives, we cannot deal directly with the drive manufacturers. We're not there yet, thus we NORMALLY bid out each drive purchase (about one large order per month) to four or five regular "resellers" and "distributors" like CDW, Ingram, B&H, etc and take the best price that month. We were down at $90 or so for individual Hitachi 3 TBytes at the low point before the crisis. We're currently paying around $160 from CDW and friends and we can afford that without raising our prices. During the Thailand flood crisis, CDW offered "not enough drives" (half our order) for $350 per drive. We said "no", because we couldn't afford "yes". :-)

Anybody can win our business by under bidding the other resellers, and we're ALWAYS open to suggestions. If you know any way to achieve lower drive prices, let us know! PLEASE!!

polyfractal• 14 years ago

I don't understand this comment at all. If anything, it has redoubled my interest in remaining a (paying) customer of BackBlaze. Look at the lengths they will go to securing drives so that I can continue to have stupid-cheap backup!

I can't ask for more.

And realistically, if push comes to shove, I'm sure BackBlaze would send out a heartfelt email to their loyal customers explaining the situation and kindly requesting a price increase. Especially after reading a post like the one they just submitted...I'm sure no one would have a problem ponying up a few more dollars.

pella• 14 years ago

same news ; and comments:

http://news.ycombinator.com/item?id=4630910

damncabbage• 14 years ago

This one is the original article, and is better-written too.

(Gigaom blogspammed it.)

yock2• 14 years ago

If I were their customer, I would be looking for alternatives right about now...

zorlem4• 14 years ago

Why would you do that? Due to reliability concerns, since they're using 3TB SATA drives, sourced by Cotsco and Best Buy?

bitJericho• 14 years ago

I'd be shocked if they were not the same drives you buy from a supplier.

aes2565• 14 years ago

The post struck me as slightly unprofessional, but from a technical standpoint I don't see anything that would change my opinion of the service they provide.

They achieve the best bang for buck by using ordinary consumer hard drives in RAID arrays. During the hard drive crisis, the cheapest way of procuring these was by purchasing external drives from ordinary stores.

There's nothing wrong with that. If anything, it shows agility and ingenuity.

yock2• 14 years ago

And, I would add, intense vulnerability to future market fluctuations. If this was their contingency plan, they've exhausted it. What happens next time?

nl• 14 years ago

Why?

They have built a backup architecture designed to use low reliability commodity hard drives.

This post shows the lengths they go to just to keep growing. Imagine what they'll do to keep the system running!

pgrote• 14 years ago

I am a customer of theirs both of their backup service and their plans for machines. The backup service has never let me down and at work we've built two of their machines using their plans saving over 10K.

The machine plans are on their site and they explain why they chose off the shelf components. My only regret is that no one on our team has welding skills, so we couldn't make awesome face plates. lol

Backblaze is one of the more transparent companies operating in the space, which increased the trust level.

budmang3• 14 years ago

Gleb from Backblaze here. Glad you're enjoying the service and the spec design. As for the faceplates, our welding skills aren't great either. However, the company we had cut our steel case for us (www.protocase.com) will cut a custom logo for you into your faceplate as well.

tallanvor• 14 years ago

Although I am in the process of moving away from Backblaze, it's not because of this. Rather it's because they don't provide the flexibility I've come to need for prioritizing content to backup, and when I recently ran into a problem after swapping out hard drives, the error message Backblaze provided was the most unhelpful piece of drivel I've ever seen. --I was told to contact support with the error code "horse" rather than just being told that it was a timeout issue!

atYevP4• 14 years ago

*Yev from Backblaze here -> Yea, that code was very early and was intended for support to gather data from the field as we were collecting data on what was causing the timeout, the message was to prompt folks to write in so they could collect some additional information. We'll probably be changing it in the near future!

mikeash5• 14 years ago

I am a Backblaze customer and this post made me admire their commitment to provide their service even in times of crisis. I won't be switching away, and I don't see why I would.

bruceboughton7• 14 years ago

The fact that external drives that have to be "shucked" of their casing and circuit boards (at what environmental expense?) were cheaper than the same internal drives makes you wonder whether the internal drive prices were realistic.

It can't be cheaper to produce the external drives, so the internal drive prices must have been raised above the actual cost to the manufacturers, presumably because the internal drive demand was guaranteed. Consumers might hold off purchasing an external disk if the prices doubled, but Apple aren't going to stop selling Macs because of it.

Sometimes capitalism isn't perfect.

DanBC2• 14 years ago

Were they the same as the internal drives? Or just the same capacity?

I get the impression they just kludged as many 3 Tb drives in as they could, and are ignoring any other specs. Maybe I'm missing something.

bruceboughton7• 14 years ago

The article implies that all internal drives rocketed in price. The drives used in the external drives will most likely be a subset of the internal drives available.

aes2565• 14 years ago

They may have different branding or model numbers, but in terms of technical specifications they are no different to internal drives.

dsr_• 14 years ago

The prices on external drives lagged the prices on internal. The externals are generally sold with reduced warranty -- 1 year, or sometimes as low as 90 days -- and my suspicion has always been that many of them failed an internal longevity test and were binned.

brigade• 14 years ago

MyBook Essentials (pictured) use the same Caviar Green internal hard drive, with the same warranty, which is still priced $10 more on amazon.

budmang3• 14 years ago

Gleb from Backblaze here. The drives were literally the same inside the external enclosures. We bought a few to start with to check.

We use consumer-grade drives across our entire 40+ petabyte storage farm because we do not need high-performance drives. Instead, we need low-cost, low-power, high-density drives to put into our custom-designed Storage Pods: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v...

We tested "enterprise drives" and found that from a pure reliability perspective, they actually performed worse in our testing than the consumer drives. With about 5 years and 40+ petabytes of data, we're fairly confident this holds true.

Finally, we assume that all drives will fail and replace about 10 drives every week. The software is written such that the system takes drive failures into account and data is stored redundantly and continuously cross-checked.

3pt14159• 14 years ago

Sure it is. You are watching capitalism in action. The company that could effectively utilize shucked hard drives did so. Larger companies that don't have the management oversight to follow this type of detail just let them be fore sale at what they already were.

sebtoast2• 14 years ago

Most of the drives in external devices are "green" drives which has higher seek time and/or lower RPM. The reason why these drives' price have not increased as much/as fast is because the demand is lower than the "regular" internal drives.

I did the same thing when the "crisis" happened, I bought a load of external drive and removed the hard disk from them.

bruceboughton7• 14 years ago

What is a "green" drive?

patio11• 14 years ago

Capitalism doesn't guarantee that things are priced in order of production costs. In particular, after these goods were produced, the ongoing inventory situation for hard drives had a wrench thrown into the works. One would hope the market then allocated a once-commodity component efficiently, such that users with the highest need for hard drives received them and users with less need made the decision to purchase hard drives later. For social (non-market) reasons, some people who sold hard drives decided to essentially sell hard drives below their market value, with rationing, to a) maintain availability of supply for a market segment with a relatively low need for hard drives while b) avoiding selling those hard drives into a market segment with a relatively high need for hard drives.

Economics (not quite capitalism) says that if you attempt to replace a market with a rationing system, you will get a market anyhow, because markets are emergent features of reality and as soon as you have multiple agents with differing values for the same goods trade happens. (Capitalism would add "... and that's a good thing!") Rationing will generally result in there being an artificial consumer surplus created in the system versus a functioning market, and that consumer surplus will generally be captured by a market-maker rather than either any of the intended beneficiaries of the rationing scheme or the entity who has the product requiring rationing in the first place.

In the absence of a rationing scheme, a) Backblaze would have had as many drives as it wished to purchase at the market price (slightly lower than the price they think is unconscionable in their normal channels) and b) most home users of hard drives would have been unable to source new hard drives at rates they were willing to pay, and would have turned to substitutes like "use refurbished hard drives", "decrease consumption of hard drive space", and "defer hard drive purchases."

stephengillie5• 14 years ago

Those are loss leaders. Much like the 29c/lb turkey at Thanksgiving, or the $29 laptops the day after, these deals are designed to get consumers into the store or website, so each will buy 1 or 2 and several other items and accessories. High-margin accessories like USB cables and wireless mice, or potatoes and stuffing, easily offset those loss leaders.

The entire point of the 2 item limit on deals like those is to prevent companies from doing exactly what Backblaze has done. By snapping up all of these deals, they're reducing the number available for consumer purchase, which impacts the revenues of these big-box retailers. Basically, Backblaze absorbed the (economic) consumer and retailer surpluses for HDDs in the Bay area and passed it on to their consumers.

rsync3• 14 years ago

I agree with the gist of this, but it's worth noting that the drives that you typically buy for arrays like this are enterprise SATA drives (or at least 24/7 duty cycle desktop drives) and the drives inside the enclosures are the bottom of the barrel ...

brianwski4• 14 years ago

(Backblaze engineer here.) Online backup is a unique application for hard drives. This is not a high transaction environment like a database getting pummeled with SQL queries. In online backup, we write the encrypted data only once, remember a SHA-1 to verify the data later, then we read it once a week to make sure the data still has perfect integrity (the SHA-1 must match). We now have over 10,000 hard drives spinning in our datacenter, some for over five years, and we've collected some pretty detailed statistics on drive failure rates vs many parameters. For our application (online backup) drives from external enclosures and consumer drives last every bit as long in our environment as "enterprise" drives.

shard• 14 years ago

Partly because Costco charges only a 15% markup to the cost they acquired the goods at (http://en.wikipedia.org/wiki/Costco#Sales_model), which means that the external drives, purchased before the flood, were able to be sold at prices below the newly raised internal drive prices.

JakeFratelli• 14 years ago

Their execution was poor. Start by depleting SF and then look elsewhere? Short-sighted at best.

They should have moved quickly before Best Buy/Costco implemented constraints that should have been anticipated. During one weekend, via train or one-way flights, employees could have been mobilized to cities across the US. Utilizing USPS flat rate boxes or low-cost equivalent, hard drives could have been shipped back to HQ en masse. Three days, disaster is over.

I give them a C- on execution.

rtkwe• 14 years ago

That's predicated on them being able to spare many employees just for drive sourcing, which as a relatively small company they probably couldn't. They went with the lower cost stop gap at first, if the shortage hadn't lasted as long they would have been better off with how they did it.

unreal37• 14 years ago

Yeah sure. In a weekend they could have purchased 1500 hard drives when there is a shortage and stores are limiting them to 2? I would challenge you to go to retail stores in a single day under those conditions and buy 25.

andrewcooke2• 14 years ago

what happened to the post that raised ethical questions about this? i think it had a valid point.

the hard drive shortage hit us all. this one company coordinated a mass buying operation from commercial stores when the rest of use were looking for personal use.

while i don't think this is illegal, it doesn't make me feel warm and fuzzy inside either.

when there's a shortage, the "noise" in traditional distribution means that small buyers (like me, and many others) can still find what we need if we put in some extra effort. it functions like a "soft buffer", making the transition to a complete lack of resources more gradual. action like this removes that buffer.

this feels like one group of people - driven by their desire to do business - beat out a load of other people who didn't have the same logistical resources. there's something uncool about it. i am not sure i have explained very well, but it's to do with crossing the line from business to personal.

[edit: ah, thanks - separate thread: http://news.ycombinator.com/item?id=4630910]

[edit2: wtf? i thought someone replied here? was about to upvote you. thanks anyway.]

corin_2• 14 years ago

> this one company coordinated a mass buying operation from commercial stores when the rest of use were looking for personal use.

Important to remember that really it wasn't this one company, they were just in the rare situation of needing drives enough to try and buy lots, and not needing enough that they could go further up the food chain in the search of them.

Think of companies such as PC manufacturers, you'd be amazed what was going on behind the scenes.

mikeash5• 14 years ago

What would you have them do, just sit there and eventually be unable to provide their service to their customers?

andrewcooke2• 14 years ago

i don't know. surprisingly enough, when it comes to difficult moral questions, i don't have all the answers.

amorphid• 14 years ago

I don't see an ethical issue here. BackBlaze came up with a creative solution to a supply chain problem. I see it like there not being enough food to go around; it doesn't make sense for everyone to starve to death. If BackBlaze had started the drive shortage, that'd be a different issue, but they found a way to survive in a tough spot.

gadders• 14 years ago

I believe this is called "hustle".

finnw• 14 years ago

It does not fit any definition of "hustle" that I am aware of, including these: http://en.wiktionary.org/wiki/hustle

lusr• 14 years ago

While I really enjoyed the story, I felt much the same way as you about the ethics. They justify their decision to run business as usual, supporting the same rate of natural growth, etc. by saying they didn't want to raise prices for their customers, but I don't think that's really a fair reason (eminent business failure would be a fair reason). There were other options:

1. freeze registration of new accounts and purchase only the drives necessary to keep existing contractual obligations

2. charge new accounts at a different rate, promising to drop the rate once drive prices returned to normal

3. change plan storage volume limits

The only reason they didn't pick any of these 3 options, as far as I can tell, is because all of them were bad for their profit margins and it would be nice if they were upfront about that as the real driver for their actions. It's not clear that their business would have failed had they chosen one of these options.

So they chose a path that would screw over the little guy needing a new hard drive in San Francisco. Perhaps I'm wrong, though, and the timing made it very difficult to act on these other options, or their cashflow couldn't handle it. That too would be nice to know, if it's the real driver of their actions.

Entirely rational behaviour (in the same position I'd do the same), but while an entertaining story, it's a hard to story to feel enthusiastic about even though in every other way it shows great leadership.

rsync3• 14 years ago

We had this problem at rsync.net ... the price of our main consumable tripling overnight, etc.

We really never considered "shucking" external drives, since the drives inside these enclosures are the worst spec'd, highest failure rate drives ... never mind the (usually) nonsensical enterprise vs. non-enterprise, they're just crummy drives.

Good for BB for making it work ...

BTW, hard drive price weirdness continues to this day - 3TB Hitachi enterprise drives two months ago were $198 (or so) and last week they were $248 (or so).

mrb• 14 years ago

But Blackblaze's infrastructure is properly designed to tolerate hw failures. They use RAID6 and replicate your data to 2 or 3 different storage pods. Smarter than beint forced to buy "better" expensive drives IMHO when your primary business expense is raw storage.

luser001• 14 years ago

If a disk fails, it still needs to be replaced. RAID etc lets you avoid data loss until you replace the drive.

w1ntermute• 14 years ago

I wonder if the hard drive manufacturers will reconsider locating their entire supply chain in a third world country with frequent natural disasters after this incident. With rising wages in the third world and stagnating wages in America, perhaps the financial benefits of outsourcing aren't so apparent anymore.

rogerbinns• 14 years ago

They have more data about all the pieces and processes leading to their ultimate market share and profitability so I have no doubt they try to maximize those.

I suspect the biggest issues are that all their suppliers and customers are in that region. Think fabs to make the chips, the various motors and related electronics, coating processes, and then sending them off to the places that make HP/Dell/etc systems. If just the drive factory were moved then you'd still have the drive parts crossing the ocean to get to the US and the completed drives crossing once again to go back into the systems.

Usually the relocations happen when a disruptive solution comes along. You get the advantage of a somewhat clean sheet. For example SSDs don't need all the mechanical components and fine tuning, so they can be made with a completely different supply chain.

It won't happen in hard drives but for other products customization is what can spur relocation. For example Lenovo is opening a manufacturing plant in the US to deal with customization which allows quicker response times. (Customized orders often can charge a little more which offsets increased costs.)

granite• 14 years ago

I was impressed with the effort to get those drives. If the massive companies who produce hard drives were unprepared for such a natural disaster I can't say I expect more from a small company.

abalone• 14 years ago

Now that's a startup story. Backblaze has some of the best down home old fashioned spit n glue startup stories on hacker news. Like the one where they invented their own storage pod out of commodity components to beat amazon's pricing. And now this stay-alive-against-all-odds story. Fantastic.

Farming hard drives: how Backblaze weathered the Thailand drive crisis

Related Stories