Facebook's new optical storage solution [video]

https://www.facebook.com/photo.php?v=10152128660097200

by riledhel • 12 years ago

111 63 12 years ago

drcross• 12 years ago

No comment on the access time and having only one drive means reading only one disc at once. This puts this storage solution in the realm of "I don't need this data within an hour" so I presume the application this system acts as redundancy for failed disks.

baddox• 12 years ago

The guy was very explicit that this is a cold storage solution. Probably "backups of backups."

NatW2• 12 years ago

The Facebook open computer definition for "cold storage":

"where data is stored on disk but almost never read again, like legal data or backups of third copies of data — keep on increasing dramatically, there are huge demands for developing some form of cold storage system with the highest capacity and the lowest cost."

source: http://www.opencompute.org/projects/storage/

nivla2• 12 years ago

Are these Blu-ray disks rewritable? If so, wouldn't this be a great tech for Amazon's Glacier service? I believe they currently depend of costly tape drives to store data.

dsl2• 12 years ago

Why bother with rewriteable disks? This is cold storage meaning that they will likely never throw this data away. I'd suspect Amazon's Glacier runs on something similar.

nivla2• 12 years ago

Why bother with rewriteable disks?

Its okay for Facebook and other single entities for backups. However for Glacier, it would mean throwing away the disk whenever someone deletes their data. It would also means constant replacement of disks.

bluthru• 12 years ago

Do rewritable blurays last as long as write-once discs?

Nagyman• 12 years ago

In the video he said those discs last 50 years, but that you can get them up to 1000 years.

akg_672• 12 years ago

Each LTO-6 tape cartridge stores 6TB of data. Each quadruple layer Blue-ray disk stores only 128GB. I am surprised that FB went with Optical Disk instead of Tape.

Optical jukeboxes/libraries for MO/DVD/CD used to be popular a decade or so ago and now considered antique. There is just not enough density with optical.

mrtksn• 12 years ago

how reliable are these tapes?

sargun• 12 years ago

My guess: 1) Lower cost for drives (commodity drives). 2) The ability to (easily) seek to an arbitrary location on disk 3) Cost of BD burners are <$100; Cost per GB of disks is $0.025/GB. LTO-6 drives are $1000+, and cost per GB is around $0.012/GB. -- I think the cost works out actually, especially as the cost of blu-ray is still falling.

mcpherrinm• 12 years ago

I believe Glacier is an all-hard-drive solution. Tapes are troublesome because they're not "online" so it's harder to verify data integrity and notice failures.

The drives are spun much slower than a regular drive (I've heard 300rpm) and so they can be crammed into much higher density racks without worrying about heat.

Maybe data is eventually moved to tapes for Glacier; I am privy to no special information.

fastball• 12 years ago

My hopes were dashed when I realized that this wasn't a new type of optical hard disc, like holographic drive technology or something.

ricardobeat2• 12 years ago

Maybe in a couple years: http://rt.com/news/5d-nanostructured-glass-optical-memory-94...

ksec• 12 years ago

With Big Data Everything, we really need an Optical Storage System like Blu Ray but with 100x to 1000x the capacity.

dsl2• 12 years ago

Facebook has built a novel implementation, but this isn't a new idea or product. Similar to how they used to buy servers and now build them to spec, they probably used to buy these and now design and build them.

http://www.disc-group.com/products/bd-series-benefits/

gatehouse• 12 years ago

After I saw this on James Hamilton's blog the other day I did a quick check of BD-R versus 4TB HDDs cost per GB. Using whatever I could find after about 5 minutes of looking around, I came up with $0.05/GB for both types of media.

If anyone can provide a link to BD media for sale at a price that make this economical I'd be interested to know.

glomph• 12 years ago

The saving is in power consumption for cold storage.

rajbala• 12 years ago

Exactly. This is low power consumption for infrequently accessed data.

mapt• 12 years ago

Are modern hard drives not rated to survive very long in spun-down, powered-down condition?

NatW2• 12 years ago

I would think the media's longevity factors into the lifetime cost. If the optical media is certified for 50+ years, that essentially means they won't have a need to recopy data if for some reason they want to preserve it.. for a long time.. I don't know how long magnetic tape lasts, but I would imagine it would be a huge pain to need to keep backing up old tapes to new tapes every X years.

crazygringo• 12 years ago

Can anyone knowledgeable explain why they use Blu-Ray instead of tape backup? When would an organization use Blu-Ray, and when wouldn't then? Is this reflective of any kind of general trend?

kitcar• 12 years ago

I'm not knowledgeable, but some total guesses:

1) As blu-ray is a consumer product its probably manufactured in greater quantities than backup grade tapes, and therefore can be had lower in price, and there are more vendors to source them from. Similarly the drives themselves are cheaper.

2) Tapes aren't random access, therefore it's slow and cumbersome to retrieve single files.

jonny21122• 12 years ago

3) IIRC most tape storage technolgies are closed source. FB has been heavily pushing their open compute project. Using BDs get them past a lot of licensing problems.

derekp72• 12 years ago

Re: 1) Looking at the prices on Newegg, 1.5TB(native) LTO5 tapes run about $30.00, so $20 per TB. Write-once BluRay looks to be about $32/TB, and rewriteable is $80/TB.

Re: 2) Although it would take an hour or so to search through an entire tape if all the files were written in a single session, normally backup software writes files in sessions containing a few GB. You can seek to a specific session in about 30 seconds average. So if you put about 4GB per session, it would take a max of approx. 30 seconds to read in that session to retrieve a file. So figure about a minute to retrieve any file. Also, LTO5 transfers about 140MB/sec, vs. 6x BDR disks (what I'm seeing for cheap right now) is only 27MB/sec.

abalone2• 12 years ago

Not knowledgeable either, but one difference is durability. LTO is rated 15-30 years while the video notes the blu-ray discs FB uses are "certified for 50 years of operation" and some go up to 1,000(!).

http://en.wikipedia.org/wiki/Linear_Tape_Open#Tape_durabilit...

akg_672• 12 years ago

Any media is as durable as the availability of media reader. I seriously doubt, in 15-20 or 50 years, you will have the media reader available to read either of the media, whether tape or BD.

How many 8" or 5.25" floppy disk drives are around today and usable with current server hardware? Anyone who is considering cold storage on media for more than 10 years lifespan is fooling themselves.

A few years ago I had a customer who had to perform data migration from DLT to LTO-3. That was over 24 month project and customer begging the vendor to provide a few old DLT drives to read. On top of that the provided DLT drives were so finicky communicating with modern server hardware that they barely kept up for a few hours before going offline.

jamesaguilar• 12 years ago

Contrary to the other two commenters, my guess was going to be that they are optimizing TB/m^3.

spligak• 12 years ago

The cost of cartridges are comparable to optical disks. However, LTO drives are non-trivial in terms of both mechanical complexity and cost. I can toss an optical tray into the recycle bin without batting an eye. Not true with LTO, and I suspect their MTBF is similar (trade the quality of the LTO drive for the mass production and cheapness of the optical tray).

Enemy #1 in a large data center is power. When he mentioned the entire rack pulls 1,000W for the density suspected in this thread (~500TB to ~1PB), that piqued my interest. Even if you can't push 2-3PB per rack, the top stat you're crunching forever and always is watt-per-gigabyte. Space is important but if it breaks into the top spot you've done something else wrong along the way.

Enemy #27 for cold(ish) storage might be, say, pressure per sq. in. on your floor. I'd be interested to see a weight comparison of a fully-loaded LTO/cartridge rack and this design.

Perhaps not what you were looking for in an answer, but I'm interested to look over the schematics.

rld• 12 years ago

At 1:55 he says "FRID'd", he meant RFID'd, right?

kraemate2• 12 years ago

Does anyone know what data is Facebook trying to archive? Old user posts? But that doesn't fit this usecase because of the high latency.

Or maybe its just a nice content delivery mechanism to the NSA. Just ship a bunch of blue-rays instead of pesky optical fiber interception.

jonny21122• 12 years ago

FB is required by law to store all its credit card transactions that every singal FB ads account has for 6 years. (So are other US based companies that process credit cards)

kraemate2• 12 years ago

Surely, credit-card info can't span 100s of terabytes, even if they record each and every transaction.

treelovinhippie• 12 years ago

Any guess on the total storage capacity in that rack and comparison to a similar sized magnetic storage?

hornetblack• 12 years ago

A comment on the video suggested 950 TB per rack.

1 Rack = 44 magazines; 1 magazine = 36 cartridges; 1 cartridge = 12 blueray discs. So, ~950TB.

nuclear_eclipse• 12 years ago

From the video: 24 magazines/rack * 36 cartridges/magazine * 12 discs/cartridge * 50 GB/disc = 518TB/rack. Based standard dual-layer discs. I cannot comment on storage of our traditional spinning-disk racks.

derekp72• 12 years ago

Backblaze Pods are about 180TB, so 1800TB per rack. And a whole lot of power consumption.

ryaneager• 12 years ago

If they use 128 GB Blu-Ray disks its about 1.26 PB per rack.

24 mag -> 36 cart -> 12 disks

10,368 Disks in total per rack

imbriaco• 12 years ago

I talked to them at the Open Compute Summit and got the chance to look at it up close. They quoted me 100GB/disc and 10,000 discs in the cabinet for 1PB. They could have been rounding those numbers for simplicity, but that gets you in the neighborhood.

nicholasjarnold• 12 years ago

Another benefit of this system is that the storage media is immune electromagnetic pulse. So, even in the case of a large-scale EMP event our cat pictures and network graphs will remain intact and recoverable. :)

MAGZine• 12 years ago

True, but it's easy to build a storage container that doubles as a faraday cage when the tapes are in long-term storage.

Actually, I'd be very surprised if this wasn't the case (pun intended).

tlrobinson• 12 years ago

I assume SSDs are screwed by EMPs, but how does magnetic media fare (the tape/disk media itself, not reader/writer electronics)?

ricardobeat2• 12 years ago

Wiped.

chm• 12 years ago

Still magnetic storage however you read/write the data...

hsshah• 12 years ago

FB: How can you get a consumer version of this so that you don't have to use Amazon Glacier for personal data archival?!

polskibus2• 12 years ago

What about bit rot though? Some of my dvdr are no longer readable after 5 yrs, stored in very good conditions. Hdd backup, esp. with Zfs can provide very good bit rot protection.

randartie• 12 years ago

These are high grade guaranteed for 30+ years. There's also 1000 year grade blu rays out there.

polskibus2• 12 years ago

I had failures of both "archival" quality DVDs and SD cards before, just after a year or two after last backup.

Do you have any links on the methodology used to measure the 30+ or 1000yr grade for a blu-ray ?

x0054• 12 years ago

Blu-rays are but a few years old. It's preposterous for any one to claim that they are "guaranteed" for 30 years, let alone 1,000 years. Not to mention that that guarantee most likely has an asterisk that limits damages to the cost of the original blu-ray disk in case it actually fails.

callesgg• 12 years ago

I was expecting some kind of optical fiber storage with reflecting crystals and shit.

This was a bit disappointing.

sneak• 12 years ago

Flash required, so it won't play here.

Really, Facebook? It's 2014.

jpalomaki• 12 years ago

I might be interested in cloud backup service which would backup my "keep forever" stuff, put them to optical disc and then mail the discs to me when they fill up.

Could be also an add-on service to existing online backup business.

cordite• 12 years ago

I did not know that they made disks that are certified for long term storage like that.

Facebook's new optical storage solution [video]

Related Stories