135 Terabytes for $12,000

http://bioteam.net/2011/08/why-you-should-never-build-a-backblaze-pod/

by gourneau • 15 years ago

158 79 15 years ago

rbanffy• 15 years ago

Well... You can make all 46 drives into a zpool with raid-z and boot FreeBSD or *Solaris from it.

BTW, I can't see the vibration sleeves around the disks on the pictures. Vibration must be a problem when you pack so much rotating media in such a limited space.

cbs• 15 years ago

Backblaze's blog talks about the vibration problem and their solution.

moe3• 15 years ago

Just a datapoint, yes, vibration is a real issue: http://www.youtube.com/watch?v=tDacjrSCeq4

dmk23• 15 years ago

The real lesson here is that if your service relies on "Big Data" processing it pays off to build your own storage.

If you were to buy 135Tb on Amazon EC2/EBS the same capacity would cost you ~$19,000 per month, not counting charges for I/O and bandwidth.

beagle3• 15 years ago

Well, space, air conditioning electricity and redundancy comparable to what Amazon offers would add a few thousands per month (I guess $2000-$5000, depending on where you are).

Still, way cheaper than Amazon, but there are nontrivial running costs beyond the purchase and build price.

moe3• 15 years ago

You should be able to rent half a rack with sufficient power to house this for around $1000/mo in just about any civilized datacenter.

DanBC2• 15 years ago

No. If you rely on a lot of storage it pays to have really smart people building the hardware and software, and maintaining it, with service agreements in place.

If 100 TB was as easy as buying a bunch of discs and slapping NASlite or FreeNAS on them then everyone would be doing it.

DanBC2• 15 years ago

Not sure why that's controversial?

tomkinstinch• 15 years ago

I am mildly surprised they can dissipate enough heat from the pods--those drives are packed so close together.

I can't tell from the photos: Is there any airflow between them?

(For that matter, do higher temperatures decrease MTTF for hard drives?)

mturmon• 15 years ago

Temp does not matter as much as has been thought. See:

http://labs.google.com/papers/disk_failures.html

IIRC, up to about 40 C, failure rates are pretty flat as a function of temperature. The paper has some nice graphs.

There is one other large-scale study in circulation that showed the same thing, but I don't recall the source.

drv• 15 years ago

A Google study about hard drive failure indicated that they couldn't find any conclusive relationship between temperature and failure rate. However, they did mention it had some effect at the extreme high end of the temperature spectrum (based on the graph, above 45 °C or so).

http://static.googleusercontent.com/external_content/untrust...

barrkel• 15 years ago

It doesn't take a huge amount of work to drive a disk over 45 degrees. Using 5K drives helps a lot, since many 7K drives will idle at 35 degrees in a room temperature environment and passive airflow.

I've seen actively cooled 7K drives in my ZFS NAS start giving checksum errors while silvering (i.e. recalculating parity etc.) after replacing a bad disk; smartmon reported temperatures of about 60, IIRC. Using an LSI 8-port SATA controller, I was silvering a 4-disk raidz at a rate of perhaps 300MB/sec, and it was making the drives too hot. I had to fall back to the motherboard SATA connectors (and ~150MB/sec array throughput) to keep things cool enough to complete.

mrb3• 15 years ago

It's only 400W in 4U (see my other comment in the thread). 100W/U is tiny, easy to cool. That's about two 120V-20A circuits per cabinet, which is very common.

brianwski• 15 years ago

Backblaze employee here. Yes, the 6 fans move a lot of air through the drives. We monitor every drive in every pod in our datacenter with the smartmontools (/usr/sbin/smartctl) which includes the temperature of the drives. The drives stay well within their recommended operating temperatures as long as the fans are running. In the past, we have detected that several fans failed in one single pod when alarms went off saying some drives were running hot, so the fans are working (normally) and the air is flowing. Side note: personally I think it's really awesome that consumer grade drives come with such detailed internal monitoring for free. I also think it is disappointing that desktop consumer OS's don't monitor these things BY DEFAULT and popup dialogs explaining to you when your drive is exhibiting high temperatures or high data loss!

rorrr• 15 years ago

Seems expensive.

2TB drives are now around $70 (Maybe even cheaper if you buy in bulk).

135TB = 68 drives.

68 * $70 = $4,760

Is that rack so expensive?

m0shen• 15 years ago

They're using 3TB drives

ceejayoz• 15 years ago

The prices are entirely laid out in part 3, titled "Real World Backblaze Costs". http://bioteam.net/2011/08/real-world-backblaze-costs/

wazoox3• 15 years ago

The box has 48 slots, so they must use 3 TB drives. They may be not completely crazy and use professional, not desktop drives. A 3TB pro drive costs about $200.

duskwuff• 15 years ago

If you're sufficiently crazy to trust your data to a Backblaze pod, you're probably going to go full retard and use desktop disks anyway. If you cared about your data you probably wouldn't be building one of these things in the first place. :)

Also: depends what you're considering "pro drives", but enterprise SATA 3TB disks are in the $250 - $300 range -- about 2x the price of a similar consumer-grade disk.

reemrevnivek• 15 years ago

You need to go back to the top of the page (not the "related posts" at the bottom ... >_>) and find part 3:

http://bioteam.net/2011/08/real-world-backblaze-costs/

Projected cost from Backblaze was $7384, for drives at $120 each and other parts sourced in quantity. A lot of the expense for these guys came from the $5400 everything-but-the-drives kit from Protocase.

The drives were more expensive because a certain model of 3TB Hitachi drives are the parts recommended by Backblaze; they didn't get whatever happened to be on sale at Newegg that day. The drives were supposed to be $120 each in Backblaze's quantities. They ended up finding a sales rep at CDW.com who sold them for $129 in quantity, but they were more expensive elsewhere.

ww520• 15 years ago

What would the electric bill and cooling cost be? Power bill is quite expensive in data center.

mrb3• 15 years ago

Very low. I estimate the power consumption at the wall to be ~400W. That's $29/month at $0.10/kWh. Host this in a datacenter with a non-remarkable PUE of 1.5, and that's $43.5/month including cooling.

* 45 x Hitachi HDS5C3030ALA630 drives, 4.8W each (average when idle), a tiny bit more when seeking so let's round to 5W. Source: http://www.hitachigst.com/tech/techlib.nsf/techdocs/02D91977... and my measurements that HDDs seeking only consume a tiny bit more current than when idle http://blog.zorinaq.com/?e=50

* 100 Watt for the mobo, CPU (73W TDP), SATA controllers, and all the fans. Source: my own clamp-meter measurements on dozens of PCs http://blog.zorinaq.com/?e=42

* 85% average efficiency of the Zippy PSM-5760 power supply. Source: its 80PLUS certification report http://www.plugloadsolutions.com/psu_reports/SP105_EMACS_HU2...

* Hence: (5.0*45+100)/.85 = 382 Watt

mrb3• 15 years ago

"The backblaze 2.0 pod has exceeded expectations when it comes to data movement and throughput. We get near wire-speed performance across a single Gigabit Ethernet link." - http://bioteam.net/2011/08/backblaze-performance/

Oh my. These guys' expectations were very, very wrong. Many single-drive configurations can sustain 120MB/s+ at the beginning of the platter. A single drive! It should be no surprise that a 45-drive beast can saturate a GbE link!

icefox• 15 years ago

It is too bad they didn't bond the two ethernet links. Reading through I didn't see a mention of what the other ethernet link was used for. That should have given them a jump in speed.

mmetzger• 15 years ago

We're considering building one of these but I used to work for a clustered storage vendor. The import thing to notice here is this is single client performance - aggregate performance is often considerably better. ie, you might be able to push 1GB/s with 10 clients, but only 100MB/s with one...

blhack2• 15 years ago

Okay, so maybe I'm missing the point here, but, uh...

No shit?

Is there anybody who seriously is tossing 135Tb at a single JBOD?

I'm not a storage guy, not even close, and even I know that that is a frighteningly ridiculous, get-you-fired idea.

These are meant as individual parts in a larger, highly redundant setup. That's the entire point.

Is the next post going to be: "Why you won't be able to download music at 1Gbps even with a GIGABIT SWITCH! Cisco lies!"?

tommi• 15 years ago

Read the post. In particular "Why the folks at Backblaze don’t care about the “risks”" section.

blhack2• 15 years ago

I know, my point is that anybody with enough knowledge about this topic to be interested in this article would already know.

smackfu• 15 years ago

It seems like an interesting one-off kind of scenario. They have a lot of downloaded data, none of which is irreplaceable, but which would take a significant amount of time to redownload. This pod is a backup but a not very rigorous one, since the goal is to never have to download all the data, not to never have to download any of the data.

serverascode4• 15 years ago

I'm looking at enclosures that can hold 60 disks. So 3TB SAS disks would be 180TB. Most enclosures of that kind have dual everything. Some SAS drives even have dual connections I believe (should look that up again).

At some point it's really a function of the size of the drives that makes the volume large. I think 60 drives is going to pretty much be the physical limit in 4 rack units.

You can buy the NL-108 from Isilon which has 108TB (36 3TB disks) if you want "enterprise." ;) Eventually they'll use 4TB drives in those.

soult• 15 years ago

I take it you only read the first part of the article.

They used RAID6 over 15 drives each, combining those 15 drives via LVM, allowing the loss of 2 drives from the same group without data loss. This is the same configuration Backblaze uses as well, because even though their fancy custom software mirrors all data to different storage pods, it is easier to just replace a drive and rebuild the raid than rebuilding the whole JBOD from the data on other pods.

nobody3141592• 15 years ago

The problem is that a large number of the failures that would take out 2 of 15 drives in a raid array will also take out all 15!

Think bad power supply, mains spike, cooling failure, fire, flood .....

If you need raid, you also need a separate copy in a separate location.

johngalt2• 15 years ago

There may be a reason to build 20 backblaze pods, but I doubt there is a reason to build 1. The whole point of a backblaze build is to move the redundancy out of the individual box.

asto• 15 years ago

If you're handling large amounts of data on your sever(s) and want a single backup just in case? I think that's what they wanted.

johngalt2• 15 years ago

By itself this device is poorly suited to backup. Specially for their use case. A huge pile of desktop Hitachi disks isn't something you can stick in a rack and forget about, and people like to forget about backups.

Best case: you have someone who's constantly checking/maintaining this box and replacing the drives as they fail.

Worst case: an expectation of a backup, but every time you refer to it, it doesn't work.

Most likely case: A mix of both of the above. Where the box takes a lot of attention, and occasionally works as intended.

The only redeeming factor is that they proclaim all the reasons why this is a poor choice. So it seems someone is at least thinking about it.

serverascode4• 15 years ago

I like the backblaze pod just in terms of using it as a comparison, where it is the absolute cheapest possible storage really. (Sure you could prob go cheaper.)

I think they've done something that makes sense for them. It doesn't make sense for me to use it for storage where I work, but it works for them in this instance. And it seems to work for Backblaze.

driverdan3• 15 years ago

I've been following the online storage space for years. Every time something like this comes out I redo my back-of-the-napkin math on potential profitability, get excited, then find excuses to not move into the space.

Prices have gotten so low that there's no excuse for lack of quality online storage providers. There are many, but there is only a handful I would consider good and their pricing isn't following the market. I see this leaving a huge opening for competition.

If anyone else is seriously interested in online storage contact me (see profile).

lsc11• 15 years ago

eh, I'm working on it right now; I'll be bringing the prgmr.com values, which is to say, the focus is on cheap, standard, and transparent, uptime goals will be... realistic. I'm doing it with my co-author, and I have at least one investor who is interested, though I'm not sure we'll need him... even at our target of a penny per month per replication, it's going to be ridiculously profitable, probably enough so that we likely won't need money. What are you bringing to the table?

gourneau3• 15 years ago

I think there would be a big market to provide $100,000ish per Petabyte collocation, when the hardware gets cheap enough for that.

lsc11• 15 years ago

Am I missing a zero? that's about $0.10 per gigabyte, no? which is dang close to what amazon charges. Unless you mean one hundred grand up front rather than one hundred grand a month.

dpe82• 15 years ago

I've been thinking for a while about how you could make an S3-like service by loading up 36 drive boxes and shipping them off to lots of different cheap colo providers throughout the country, duplicating storage across them and managing the whole bit on a reliable provider like AWS.

wmf4• 15 years ago

Given that an empty pod costs $5,395, Supermicro might actually be cheaper (although a little less dense) and it's a real product instead of a prototype.

lsc11• 15 years ago

I'm looking at

http://www.supermicro.com/products/chassis/4U/847/SC847E26-R... (36 drives, and room for a motherboard)

http://www.supermicro.com/products/chassis/4U/847/SC847E16-R... (45 drives, das. I have had serious outages caused by DAS back in the SCSI-3 VHDCI days, so I'm likely to start with the 36 bay system.)

for my mass storage project. they are both well under $1500 with power supplies, backplanes and expanders taken care of.

For me, the big thing is that with my storage model, I'm going to be replacing disks as they fail and rebuilding the RAID, so having easily accessible and easily swapped disks is worth paying a premium. (I am planning to have some cross-chassis redundancy by using zfs snapshots, but I'd rather just keep the nodes going as is.)

Also, rack density? doesn't save you that much money. Most of what you are paying for in a data center is power. At the cheapest co-lo I'm in, here is the cost breakdown:

1 full cabinet (44u) with two twenty-amp 120v circuits: $875 1 full cabinet (44u) with one twenty-amp 120v circuit: $530

So, if I can double my density, I save $185 a month; and even at the disk density for my compute nodes (close to 100 disks in a rack) I get one disk failure maybe every two months per rack; so if I have to slide out the whole goddam computer, causing some chance of the power getting disconnected and downtime? yeah, with my model? it's probably worth paying the premium.

I'm just saying; if you are small enough that paying five grand for a backblaze pod.

Sure, at-scale, it's best to design your systems so that you can get zero downtime even when hardware fails. But, that's really difficult to do without introducing new failure modes; even amazon has trouble with it. My strategy is to accept that hardware failures mean a truck roll and downtime for customers on the hardware in question. As long as you don't have any one server go down more than once a year, (and a particular server failing once a year is pretty pathetic) and as long as you don't have a system where any one server brings down everyone, you are going to see pretty good reliability using this strategy.

driverdan3• 15 years ago

Backblaze designed their system to be redundant across multiple systems with the ability for one to go down seamlessly.

If losing one system from your network will cause unacceptable downtime then you shouldn't consider using Backblaze's design.

nwilkens• 15 years ago

I just built two of these 36 drive systems. $9k for 108TB raw. Feel free to drop me a line, and I will share any details.

e1ven• 15 years ago

Who would you buy through? Googling for the 45 drive version gives me slightly higher pricing.

Also- The big advantage of the BackBlaze version is that people have done it before, and it mapped out. With the SM case, there's less community.. But it does look like a good solution.

It looks like the drive bays are pre-wired up, so you wouldn't need to worry about that? What else would you need? Mboard/CPU/RAM, Raid cards, HDs and sleeves?

serverascode4• 15 years ago

This is a great thread here.

I think there are a lot of people, even institutions like where I work, that are looking hard at storage that isn't block based storage from some huge vendor or storage that ends up being $5000+/TB (looking at you Isilon). Ok, maybe it's just me all alone here at my work place. :)

Even running RAID6 or RAID10 over 3TB SAS drives on some largish enclosures (ie. 36,45,60 drives per 4U) will be very cost effective even just using md, lvm, and xfs.

Or even better, use ZFS if possible. Not a huge fan of Solaris, but there is OpenIndiana, or FreeBSD has ZFS.

And object storage such as openstack swift is really gaining momentum and will likely replace most storage systems for large amounts of data over the next couple/5 years. There are single orgs that have put out 5.5 petabytes of openstack swift storage! Right now!

charlesap• 15 years ago

Grown defects. Graaah.

gourneau3• 15 years ago

What do you guys think about using this with Swift from OpenStack? Could I buy 100 of these pods, drop OpenStack on them and get close to 135,000 Terabytes with some level of redundancy?

notmyname• 15 years ago

Some of the hardware choices that Backblaze has made in their pod design are interesting. Most of the concerns I have (like the difficulty of getting to drives to replace them) can be addressed by operational practices (update the swift ring rather than immediately replace the drive). Other concerns (like 2gbe for 135TB) are more nuanced. Lack of redundancy in the Backblaze pods is addressed by swift itself--swift will ensure that no two copies are on the same pod.

I would love to see someone run swift on some Backblaze pods. If you'd like to talk further, my contact info is in my profile (or drop by #openstack on freenode).

iamelgringo• 15 years ago

Wow. The cost of 1 Petabyte of spinny disk storage is $100k.

A friend of mine who raised money in 1999 said that to get their MVP off the ground, it took 20 engineers, and $7M in venture funding. I doubt $100k would have paid for the Oracle licenses back then.

mckoss• 15 years ago

The application here is write once, read rarely. Would it be possible to unpower drives once they are full (until needed)?

135 Terabytes for $12,000

Related Stories