Petabytes on a budget: How to build cheap cloud storage

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

by bensummers • 17 years ago

401 105 17 years ago

blog.backblaze.com

dmillar• 17 years ago

This is unbelievably awesome. Can you imagine what we could achieve if every company had this level of transparency?

PStamatiou2• 17 years ago

Thinking the exact same thing! While technology isn't more than any typical hacker can slap together in their apartment (minus maybe the case fab), I think the application is ingenious and the full blog post about it, like dmillar, pointed out is rather awe-inspiring. And I was all happy because my new Core i7 box has 2TB of space and 6GB of ram... sigh

That being said - did seagate ever fix their firmware issue on the 1.5TB drives that would cause random corruption? (heard about it maybe 6 months ago)

barrkel• 17 years ago

I should think so - I have 4 of them in my ZFS pool, and scrubs are turning up no errors.

sophacles3• 17 years ago

I have the 500GB in the same series (with the corrupted firmware) and the fix did very nice things for it. Also, I have never had problems with the 7200.12 (the newer version of the same drive) in a 1TB model.

jbellis2• 17 years ago

> While technology isn't more than any typical hacker can slap together in their apartment

I think you missed the part about testing a dozen SATA cards, etc.

The attention to detail here is a lot more than something you'd slap together in your apartment.

bestes2• 17 years ago

Yes, they did a firmware update. Here is one article that talks about it: http://www.tomshardware.com/news/seagate-barracuda-1.5TB-fre...

lsc13• 17 years ago

as for the firmware 'bug' - yeah, they fixed that one, However, consumer sata disks are still not something you want in most raid systems. See, a consumer disk, when it encounters an unrecoverable error, retries over and over, hanging your raid until you fail the bad drive by hand. (tested on a 3ware hardware raid and with MD. I am given to understand that zfs handles this in a sane manner.) Western digital allows you to change the behavior, search for WDTLER, but in general.

I'm sure the backblaze folks have built something that handles that sort of thing into the system, but just saying; it's something you have to think about before you slap consumer drives into a raid that needs to keep working when one drive fails.

edw519• 17 years ago

I always thought it was best to focus on what I did best (application software) and leave the infostructure to others. Until I saw this:

  Raw Drives      $81,000
  Backblaze      $117,000
  Dell           $826,000
  Sun          $1,000,000
  NetApp       $1,714,000
  Amazon       $2,806,000
  EMC          $2,860,000

I had no idea. Kinda makes one rethink what business they want to be in.

spydez• 17 years ago

Well, to be fair, Backblaze is using consumer-grade hardware here. No redundant PSUs. No mirrored OS drives. Cheaper components. Etc.

They're going Google's route, which is fine, it's just... a whole different direction than a Sun Thumper. I'd be interested to see how their costs per petabyte stack against Google's.

blasdel• 17 years ago

No, they aren't going Google's route!

Google's operation has at most half the disks-per-rack density of Backblaze, but with 4 or 8 times the server density. Google does almost all their computation on the same nodes where the data is stored. Google is also storing each tablet a minimum of 3 times within each cluster, with most systems having multiple replica clusters.

Backblaze's system is going to have several orders of magnitude less bandwidth:

  way too many drives in one server (some of them on the PCI bus!)
  use of port multipliers (causing 5 drives to share one SATA cable's 300MB/s)
  RAID6 with too many drives per array (15-way XOR is no fun)
  why use JFS?
  only access is via HTTPS, not clear if SSL is done on the pod

cperciva• 17 years ago

  Backblaze      $117,000
  [...]
  Amazon       $2,806,000

I cry foul. Backblaze's "67 TB" pods actually only hold 58.5 TB, so their hardware cost per PB of storage is $134k, not $117k; and that's without any high-level redundancy. Servers fail -- both catastrophically, and by silently corrupting bits -- and Backblaze's $134k / PB doesn't have any protection against that. Datacenters also fail -- power outages, cut fibre, FBI raids, etc. -- and any system which stores all of its data in a single datacenter lacks any protection against that. Store each file on two different servers in each of two different datecenters, and suddenly Backblaze's $134k turns into $536k. The price for Amazon, in contrast, is based on the assumption that their prices remain fixed for the next 3 years -- which seems a rather radical assumption.

Is backblaze's solution cheaper than S3? Absolutely. But they're also twisting the numbers a bit.

skolor3• 17 years ago

Interesting, but Amazon doesn't guarantee most of those things. They guarantee a 99.99% uptime, but that isn't counting a complete datacenter failure. In fact, it sounds to me as if they have a similar set up to the backblaze people.

99.99% uptime means roughly 1 hour per year of downtime. I don't know what the specific failure rates on the components are, but it seems reasonable that A) the data drives are hot-swappable, and will not cause downtime when they are replaced and B) the rest of the components fail once a year (or less) and take ~10 minutes to replace and reboot the system. With 4 main places of failure (PSU, Boot Drive, Motherboard/Ram/CPU, Drive controllers), as long as you have staff constantly on call and they can respond within 5 minutes of a failure, 99.99% uptime seems reasonable.

I don't know where you got the idea that your data was on 4 different servers when using S3. I can't find even the slightest amount of information on that. Yes, that would be nice, and it is cool to think that, but its rather doubtful that they're actually doing that (or they could probably add another 9 to their uptime).

modoc2• 17 years ago

It really depends on what they ascertained their needs as. Perhaps they don't feel they need the additional protection, or they have another backup scheme in mind. If their solution works for them (which is seems to) then indeed, for their needs at least, the cost comparison is pretty accurate.

It might not add up the same for your needs (assuming you want multi-data center redundancy of all data, etc...).

Periodic3• 17 years ago

That's just the cost of hardware. How much more do you think it would cost if they included all their research and development, installation and assembly hours, etc?

That is just the cost to buy all the components they have. If you budgeted that in you'd just have a bunch of boxes at your office that weren't even wired up. On the other hand, from Dell or Sun you at least have all the hardware in the chassis, if not some basic configuration and OS install done for you. Go with Amazon and you've already got it racked up in a data center with dedicated techs and replacement parts on hand.

It isn't a fair comparison.

cakeface• 17 years ago

Another thing that they aren't taking into account over S3 is power costs, physical datacenter rent costs, and bandwith hookup. Still its probably a lot cheaper but important to note that they are comparing apples and apple cider.

johnrob• 17 years ago

This is not apples vs apples. Some of those items are living cows, and some are cooked steaks served with a side of mashed potatoes. I'm sure the cow looks cheap (per pound) until you are tasked with turning it into a steak.

mmt4• 17 years ago

As a sysadmin for organizations of different sizes, I've been acutely aware of this for quite some time.

The more painful corollary is that, even if one measures not by total storage but by performance (a more critical aspect of, e.g., databases), the prices are still an order of magnitude or two apart.

I would argue that the reason "No One Sells Cheap Storage" in the backup/archive sense is that there isn't enough demand. Obviously, this is starting to change.

I am, however, startled that a driveless pod is so expensive ($2467 or $54.82 per drive), considering that external performance is, effectively, limited by gigabit Ethernet.

Two SuperMicro 846E1s would be $2k for 48 drive bays, and one could easily connect 10 of them to a single $300 SAS RAID card. Add another $1k for mobo and cables, and that's $11300 for 240 drives or $47.08 per drive, without having to do a custom case.

Granted, it takes up almost twice the rack space, but it's still a decent power density at 50W/U AC (assuming 6.5W per drive DC).

It takes 52 groups of 13 1.5TB disks (Hmm.. 13 pops up an awful lot in their setup) to make a PB using their pods. 51 groups means 17 pods is .9945TB for $134k ($134.5/PB).

Using my pods, it would be .936TB for $120.3k ($128.5/PB), but what's $6k/PB between friends?

lsc13• 17 years ago

you might also want to check out coraid. it's at a similar price point. http://www.coraid.com/PRODUCTS/SR2421 - now, I've heard mixed experiences with it, and have no personal experience, but if you are talking about wiring up something yourself, it's definitely something to consider.

Personally, I'm working on a system whereby I just fill up otherwise unused slots on my chassis and use that for backup. The system isn't done yet, but I've got 6 disks out there waiting for me to figure it out.

wmf2• 17 years ago

Two SuperMicro 846E1s would be $2k for 48 drive bays, and one could easily connect 10 of them to a single $300 SAS RAID card.

I don't think a single SAS card can support more than 128 (or maybe 256 if you're lucky) disks. This doesn't affect your math much, but I wouldn't want someone to try this at home and be disappointed. If you're not willing to build a custom case and don't feel like suffering the vagaries of SATA expanders, LSI SAS cards plus Supermicro JBODs are the way to go.

modoc2• 17 years ago

I love how open they are about their solution, and I also want one for Christmas:)

psranga• 17 years ago

Great stuff. Would anybody care to clarify why access is through HTTPS and not HTTP?

I presume all accesses to these pods are from within their data center? Or do they directly expose these boxes to clients (whoa!)?

phsr• 17 years ago

BackBlaze runs online, off-site backups (like Mozy and Carbonite) at $5/month for unlimited storage. HTTPS is used to keep client data protected, I assume

tsuraan• 17 years ago

67TB of storage, with 4GB of cache. I'd really love to see some performance numbers versus the way-too-expensive competition. If the systems are being used as tape drive replacements, I could see this working well, but as an actual NAS-like device, I can't imaging how it could perform acceptably. Of course, if those Intel motherboards have the dual 1Gb/s NICs that Intel boards generally do, it will probably take a while to fill the drives anyhow.

notaddicted2• 17 years ago

http://www.wolframalpha.com/input/?i=(67+TB)+%2F+(2*(1+Gb%2F...)

It'll take 3 days at the theoretical max of the networking equipment to read/write the 67TB. The overhead of HTTPS constrains the networking so this is too low.

I'd expect that their internet connection (i.e. in/out of the data center) is the real bottleneck.

I believe that the system is being used as a tape drive replacement.

lsc13• 17 years ago

If the load is relatively sequential, then yeah. But if the load is mostly random, then there is an argument for having more cache.

lsc13• 17 years ago

Actually, that's the problem with most NetAPP/EMC foo, too. they are way light on the cache.

These guys only use 4GiB ram because they are saving money on the motherboard, I bet. Personally, if I were building it, I'd increase the cost by another grand or so and use dual low power opterons with 32GiB ram. (of course, that would also increase the space taken by the motherboard, so that would require some case redesigns. Still, opterons and registered ecc ddr2 are both incredibly cheap right now.)

pmorici3• 17 years ago

What if they used AndrewFS with SSD drives for cache on front end nodes.

gstar3• 17 years ago

They might have missed a trick here.

To address vibration, acoustics and gyroscopic effect, what I've seen done in highly dense enclosures is to rotate every second drive around 180 degrees in a bit of a shotgun approach to balancing stuff.

Still, awesome.

sh1mmer3• 17 years ago

This is the trick, though. They want to improve their system. By sharing with you they may have solicited some really helpful advice.

ciupicri• 17 years ago

This has sounded ok for me too, but when I've thought a bit about it I've realized that the drives must be synchronized so that each one would "cancel" each others vibrations. The vibrations generated by the drives might have indeed the same amplitude and (spatial and temporal) frequency, but I doubt that they'll have the same phase offset. For this to happen they would need to be started at _exactly_ the same time which I don't think that happens in practice.

Do you have some pictures of those enclosures?

gstar3• 17 years ago

I don't have pictures. I can describe, though.

Two disks were mounted in a frame linearly, both screwed to the frame, with the power/sata connectors toward the middle of the frame, and one drive upside-down.

These cassettes were removable as a unit for hot-swap, and were inserted linearly into a half-deep 19" rack enclosure.

That the two drives were physically connected to the same frame, and removed and replaced as a unit would make it seem as if they would be started in phase. Now I'm not physicist, but I'm not 100% sure that's so important - if you have two gyroscopes contra-rotating and firmly connected, running at the same speed, surely they resist movement by sheer gyroscopic effect?

Andys3• 17 years ago

Unfortunately the SATA backplanes are only aligned one way, so they could only rotate a set of 5 drives at once.

swombat• 17 years ago

Wow.

Do they offer S3-like storage? They should. If they can offer something like S3, but at one third of a penny per gigabyte per month (heck, let's splash out - a whole penny per gigabyte per month) I know quite a few people who'll be interested in talking to them... (including myself)

jacquesm• 17 years ago

Apparently they use HTTPS for input/output using tomcat and some custom application.

That's a strange choice, HTTPS would incur quite a bit of overhead for something that is essentially a (large) drive at the end of a network cable used internally only. Why the encryption ?

quote from the article:

"A Backblaze Storage Pod isn’t a complete building block until it boots and is on the network. The pods boot 64-bit Debian 4 Linux and the JFS file system, and they are self-contained appliances, where all access to and from the pods is through HTTPS. Below is a layer cake diagram.

Starting at the bottom, there are 45 hard drives exposed through the SATA controllers. We then use the fdisk tool on Linux to create one partition per drive. On top of that, we cluster 15 hard drives into a single RAID6 volume with two parity drives (out of the 15). The RAID6 is created with the mdadm utility. On top of that is the JFS file system, and the only access we then allow to this totally self-contained storage building block is through HTTPS running custom Backblaze application layer logic in Apache Tomcat 5.5."

That's an odd choice for a storage server protocol stack.

sophacles3• 17 years ago

One thought I had regarding that -- their client talks to a "main server" management system, which talks to each pod on using the tomcat layer. The "main server" then replies to the client with an IP or similar to send data to directly. Since this goes over the web, https is used for security. Since https is already being used, it is not so difficult to use it all the way for authentication etc.

antonovka• 17 years ago

That's a strange choice, HTTPS would incur quite a bit of overhead for something that is essentially a (large) drive at the end of a network cable used internally only. Why the encryption ?

To help prevent a network compromise from resulting in storage management compromise? Just because something is internal doesn't mean it's safe. Once a host/network segment is compromised, you don't want it to be easy to jump to the next.

Otherwise, you've built M&M security. Hard candy shell on the outside, soft gooey chocolate insides. Mmm.

sophacles3• 17 years ago

It seems to me that their business model, and their requirements, and their implementation solve one problem, and they do it well. The problem backblaze is solving is backups. Their front-end software sends a pile of data to the datacenter, where it is written en-masse. Should restore ever be needed, the data is then transfered back en-masse. Both of these operations have needs that are different from an S3 type service. Backup/restore can have latencies built in, can have slow seek times, etc. This allows them to take advantage of the lower end hardware.

For example each of these "pods" only has 4GB of RAM. If I was doing lots of random I/O on 67TB of drives, I would want a heck of a lot more ram for caching efficiencies.

jbellis2• 17 years ago

It's actually a fair bit harder to offer generic key/value large object storage than to offer a backup system that talks to a client you wrote yourself.

Of course these days there are well-understood solutions to those problems, compared to when S3 first started, but it's still not stuff you can pull off the shelf. (Although with the Cassandra distributed database, the metadata problem is close to that point now. </plug>)

Periodic3• 17 years ago

The cost they are quoting is the hardware cost. It isn't including developer time, housing costs, maintenance, etc.

Hardware is cheap, power and developers aren't.

tricky• 17 years ago

effing love this.

I wonder what the reliability stats on this setup is, though. Is it really cheaper to jam all those drives in one unit without redundant PSU's, MB's, or a boot drive?

I'd guess you'd have to build at least 2 of these units and mirror them to get any sort of reliability. And, at that point, how long does it take to copy 58 TB across https?

data is hard.

skolor3• 17 years ago

I would assume they don't care about any of the hardware, just the data. If you look at the setup, and how the drives are raided, there 45 drives are sub-divided into 15 raid arrays, of which it would take 3 to die before they lost data. Essentially, they would need 20% of their drives to die simultaneously for them to lose data.

Now, for the rest of the hardware, its not that important to if it fails. If one of the other components die, you're only looking at some down time (and a possible dead hard drive or two from a PSU dying, which I assume they monitor regularly). As long as the data is secure and in one piece, it doesn't really matter whether the pod is up or down, until someone needs the data. Just send out your repair guy to replace it and reboot it, and its fine.

Andys3• 17 years ago

In sysadmin terms, they have aimed for data integrity over availability.

If a system goes down and they have to replace a disk or a power supply or motherboard, the data is still safe, and if by chance there's any $5/month users who need to restore data held on that one unit, well they can wait a few hours :-)

byoung2• 17 years ago

Very interesting insight! I may try building a scaled-down version just for fun.

The cost comparison between raw drives, their custom solution, and Amazon S3, etc was a little skewed. S3 is designed for pay as you go storage so you're not paying for capacity you don't need. If you just need a few dozen gigabytes, it's a much better deal. If you need terabytes or a petabyte, a dedicated storage solution is more economical.

It's the same argument as vacation house vs timeshare. If you lived in a timeshare all year, it would cost more than buying the house.

dabeeeenster2• 17 years ago

What happens when one of the PSU's fails catastrophically, pushing a big electrical surge through the hard disks, frying half of them?

I didn't see anything in here that discusses that eventuality? And when you have that many servers, it IS going to happen at some point...

notaddicted2• 17 years ago

It is called a fuse.

papercrane3• 17 years ago

It's addressed in the section "A Backblaze Storage Pod is a Building Block"

From the article:

When you run a datacenter with thousands of hard drives, CPUs, motherboards, and power supplies, you are going to have hardware failures it’s irrefutable. Backblaze Storage Pods are building blocks upon which a larger system can be organized that doesn’t allow for a single point of failure.

dabeeeenster2• 17 years ago

Sounds to me like they haven't implemented this yet. I wouldn't want my data on that sort of solution. Backed up data would surely need a level of geographic redundancy.

ars• 17 years ago

To get all the various logos and certifications on them, they are required not to do that. Every line is voltage limited.

sschueller• 17 years ago

How do you replace a drive? With so many drives it seems like a lot of effort to pull the entire server out to just access one drive.

papercrane3• 17 years ago

They have three raid-6 volumes per machine, so they probably leave it in the rack until two of the volumes are unusable and then refurbish the machine.

rarrrrrr• 17 years ago

I doubt it very much. When drives start failing out of a RAID6 array, you lose it's ability to automatically detect and correct single bit errors on disk.

With that much data, single bit errors will happen predictably.

If their usage patterns are anything like ours, disk failures are well under normal rates. Most backup data is stored and ignored. The drives just don't get much stress. Downing a machine for maintenance is probably acceptable to their usage pattern.

ShabbyDoo2• 17 years ago

What terrified me was the number of low-level issues they had to address. SATA protocol problems, custom-designed SATA cards? Wow. I'd have no idea how to begin with that stuff. As other posters have noted, this company's business is storage. How many PB must one host for such specialization to make sense?

Bjoern• 17 years ago

What a pity, I'd like to have a way to store just a encrypted backup directly (via ssh) instead of installing a app which crawls my system. Oh did I mention I use Linux? sign

Does somebody know any other company which provides this solution? I don't trust other people to crawl my system and to "encrypt" it for me.

EDIT: Added Question

californiaguy2• 17 years ago

Yeah, rsync.net

bestes2• 17 years ago

Did you look at tarsnap (http://www.tarsnap.com/)? I don't use it, but read about it here quite a bit and it sounds really great.

blhack• 17 years ago

I'm not entirely sure if this is a troll or not, but...

rsync does exactly what you're looking for. You can also pipe it over SSH encyption on the wire. type "man rsync" at a shell for a description of how.

soult• 17 years ago

How about http://allmydata.com/? They offer a JSON-based, restful API, an FTP interface or you can use their open source software which includes encryption.

thaumaturgy• 17 years ago

I have been searching high and low for something like this for months -- I have a number of clients that have been begging me for this, as well as something for my own needs.

I'm excitedly posting a link to this on my personal site, and today I have a lot of phone calls to make to clients.

Why is it that the best products and services are also the hardest to find when you're looking for them?

durana• 17 years ago

The price comparison with other solutions isn't really fair. The cost per PB of these storage building blocks is directly compared to complete storage solutions from companies like NetApp and EMC. In the end the cost of the complete solution they assemble these building blocks into may well be cheaper than solutions from other companies and that's the number they should be using for comparison.

datums• 17 years ago

Service Idea: S3 offsite backups. Run http://www.eucalyptus.com/open/ on this kind of hardware.

sh1mmer3• 17 years ago

I'd love to hear what they are doing to monitor their system. They didn't mention that. 3 out of 15 drives failures to loose a volume seems quite likely at that scale.

I'd like to know what levels of warnings and alarms they use with which system, e.g. nagios, etc.

preview2• 17 years ago

I wonder about the single point of failure posed by the power supplies. One failed box is not a big deal (since I assume the data is replicated over several). But, what if they get a bad batch of supplies and see a relatively high failure rate? I wonder how high a power supply failure rate they can handle.

The need to stagger the power on of the two supplies poses a problem. What if power to a data center is lost? When power is restored, all box will try to start, blowing all fuses. Granted, this is a catastrophic event, so its frequency should be very low. But, this also seems like an area that could be automated.

lsc13• 17 years ago

You should only use 75% of the rated capacity of a circuit, which means you have enough power to turn them all on at once.

some of the more expensive managed power supplies also support a staggered power on after power fail. But I don't worry about it; only using 75% of the power circuit solves that problem for me.

preview2• 17 years ago

Unfortunately, using 75% of the rated capacity may not be enough to handle the inrush. The article discusses this point, "...if you power up both PSUs at the same time, it can draw a large (14 amp) spike of 120V power from the socket." That would mean one pod per 20A circuit. Ugh. In normal operating conditions, a 5.6A max load would allow three pods per circuit.

Addressing this would require a little bit of design, but the problem is relatively simple. If they wanted to get fancy, they could add a chaining feature--pods on the same circuit would be connected together so that they'd power on serially. This would get away from their goal of using off-the-shelf parts. It is, as with many things, an engineering trade-off.

sh1mmer3• 17 years ago

One of the smart guys in our (Y!) cloud team just point out something to me that hadn't occurred to me.

This system is definitely optimized for backup. That totally make sense for Backblaze. However it's important to not compare this like for like with something like S3 which is optimized for much better read/write performance.

At the basic level the cooling on this system seems minimal. Those tightly packed drives would sure get hot if they were all spinning a lot. More than that since they are using commodity consumer hardware, and they already used up their PCIe slots for the SATA controllers there isn't any place to add anything more than the gitabit (I assume) ethernet jack on the mobo. That means there throughput is limited.

Again, this is a great system for backup. Most of the data will just sit happily sipping little power. However, if you are thinking of this as equivalent to a filer, that's an unfair comparison.

hyuen• 17 years ago

I was thinking about the IOPS that you can sustain with this type of device, and as mentioned here, this is intended for backup, which may or may not be very good acting as a filer.

Probably you can get some high IOPS with all these drives in parallel, but probably not as much as the other more expensive ones (EMC, Netapp).

But I have to agree this is an awesome setup optimized for a specific application. Way to go

Andys3• 17 years ago

Fast 120mm fans can move a pretty decent amount of air - up to 120 ft^3/min each, and they used three in parallel.

It would take about a dozen 80mm normal-speed computer fans to reach this.

pmorici3• 17 years ago

Seems along the same lines as Capricorn Tech. http://www.capricorn-tech.com/products.php

ggruschow• 17 years ago

Not really. BackBlaze's post is about how to get on-line storage up for the lowest cost per bit. They appear to have done a great job at quadrupling the density of standard 1U setups for a similar price.

Capricorn appears to be in the business of selling standard 4 drive per 1U setups and support. The fact you have to contact their sales department to even get close to a price seems to indicate they're not competing on price.

ALee• 17 years ago

These guys are very very good at what they do. Btw folks, no funding and they're already profitable. Gleb and crew rock!

jwilliams2• 17 years ago

Awesome. Brings up a few questions for me (sure there are answers, just curious really).

Why a Core 2? An Atom mobo would be lower power and cheaper. Why 4Gb? Seems like an overkill. They are using a HD to boot. Couldn't they boot of a USB key?

yellowbkpk• 17 years ago

I don't know for sure, but there might not be an Atom-based board that has as many PCI/PCIe slots.

jwilliams2• 17 years ago

Good point. The one I've got only has one PCI in fact.

silverscreen• 17 years ago

Great read! Why a seperated boot disk? Why can't that be on the storage array?

eleitl• 17 years ago

This made the Beowulf list yesterday, and below is what I wrote in response:

"Seagate ST31500341AS 1.5TB Barracuda 7200.11 SATA 3Gb/s 3.5″ Aargh! Should be definitely substituted by 2 TByte WD RE4 drive.

Today I've built a 32 TByte raw storage Supermicro box with X8DDAi (dual-socket Nehalem, 24 GByte RAM, IPMI), two LSI SAS3081E-R and OpenSolaris sees all (WD2002FYPS) drives so far (the board refuses to boot from DVD when more than 12 drives are in though probably to some BIOS brain damage, so you have to manually build a raidz-2 with all 16 drives in it once Solaris has booted up). The drives are about 3170 EUR sans VAT total for all 16, the box itself around 3000 EUR sans VAT. I presume Linux with RAID 6 would work (haven't checked yet), too, and if you need more you can use a cluster FS.

Maybe not as cheap as a Backblaze, but off-shelf (BTO) and you get what you pay for.

Petabytes on a budget: How to build cheap cloud storage

Related Stories