10 petabytes - visualized

http://blog.backblaze.com/2011/01/05/10-petabytes-visualized/

by geekfactor • 16 years ago

69 50 16 years ago

blog.backblaze.com

geekfactor• 16 years ago

I love #3, especially the Stay Puft Marshmallow Man!

rphlx2• 16 years ago

Yeah, I think Stay Puft should be mandatory in all big-stack-of-stuff infographics.

xtacy• 16 years ago

Is there any reason why 10 PB was split in that particular combination of 2, 1.5 and 1TB drives?

EDIT: And, just a comparison: Google processes over 20PB of data per day!

jerf• 16 years ago

It looks like that is their actual inventory of drives. At that scale, when 2TB drives become the most economical to buy that does not mean one immediately dives in and replaces all the 1.5TB drives. You have to do a bit more math than that.

It's also hard to tell whether they're actively decommissioning their 1TB drives or if they've just on a growth curve and they've done significantly more business while 1.5TB and 2TB drives were the more economical choice, but sooner or later the cost/benefit curves do cross and you do want to start decommissioning the smaller drives, it's only a matter of time.

quinndupont3• 16 years ago

There's a big difference between "storage" and "processing". But it's true, Google stores plenty of data too.

jodrellblank2• 16 years ago

and why it takes more 2TB drives than 1TB drives...

Retric2• 16 years ago

That's the drives they actually have.

2280 * 2 + 3166 * 1.5 + 1 * 749 = 10,058 TB.

kayoone2• 16 years ago

I am not american, nor do we use inches here in germany but when looking at #3 it should be noted that 5.75 inches is actually the drive length, not height as they state. Well depends on how you look at it but it confused me in the beginning.

coderdude• 16 years ago

They could be trying to make this seem more impressive by standing the hard drives up instead of laying them flat.

kolinko• 16 years ago

I'm curious why this didn't get more upvotes. 5.75 is a diagonal I think?

jacquesm4• 16 years ago

I'll give you a much more compact way to envision 10 Pb:

10 Petabytes is 10,000,000,000,000,000 bytes or 80,000,000,000,000,000 bits, divided by 8,000,000,000 bits per full human genome (2 bits per base-pair) that's about 10,000,000 cells (not red blood cells because they don't contain DNA), or about 5 milliliters! (10 um diameter on average so about 500 cubic um, so 2 million or so per ml), and that includes all the stuff besides the DNA in the cells.

Hard disk storage certainly is impressively compact but it still has a long way to go before we beat mother nature.

-1 already eh? Downmodders please correct my math or say what you think is wrong with the comparison, I note the article ends on: "How would you visualize ten petabytes of storage?".

melpo• 16 years ago

First off, I didn't downvote you. However, the comparison is wrong imho because disk storage is not simply about storing information. Disks also need fast read/write access, which DNA certainly doesn't have for us (for now?).

jacquesm4• 16 years ago

We are envisioning storage, a bunch of disconnected drives piled on top of each other is also not accessible for read/write access.

ramchip3• 16 years ago

I see many debatable points:

- Blood is far from being that dense, but I'll assume you mean to calculate a theoretical limit.

- If you use the whole of a cell's DNA to encode information, the cell will die. It can't be used to freely encode information like a hard disk.

- There is no way to retrieve information in this system. This is a bit like packing a large amount of extremely high-density magnetic platters in a box and calling this a storage system, which is very different from using actual hard drives with all the complex reading/writing system (heads, space between the platters, magnets...). Even if they're not connected, it still takes room.

- Reliability of hard disks is far, far better than this. Encoding information as a single copy in a single cell is wildly unsafe. Even using multiple copies, it's likely to degrade with mutations.

Edit: I was under the impression that a base pair could only encode one bit of information, since only T-C and A-G are valid combinations. Wikipedia appears to disagree with me but I see no source; does anyone know why Wiki says a base pair could encode 2 bits?

die_sekte• 16 years ago

Re: Number of bits per pair: TC, AG, CT and GA? I.e. composition (1 bit) and direction (1 bit)? Not sure, but that seems to be the most logical way.

incomethax• 16 years ago

The reason one base pair can encode 2 bits is that T-C and C-T are different valid base pairs. The same is true for A-G and G-A as well. A "single" strand (the famous double helix) of DNA actually contains 2 copies of genetic information.

stevenbedrick• 16 years ago

If I had to guess, it's because the way you read DNA is only along one of the strands (starting at the 5' end, IIRC), in which case you've still got four possible "characters" to use in your representation- A-G is different from G-A, etc. People have thought up all sorts of clever ways to think of how one might encode data using DNA base pairs, but it's been too long since I did any bioinformatics for me to be able to remember exactly how they calculated it all out.

jacquesm4• 16 years ago

> - Blood is far from being that dense, but I'll assume you mean to calculate a theoretical limit.

There are more ways of having cells arranged, I explicitly mentioned red blood cells because they are exceptional in not containing any DNA, but any chunk of tissue with that volume would do.

> If you use the whole of a cell's DNA to encode information, the cell will die. It can't be used to freely encode information like a hard disk.

Yes, that's obvious, but a cell's DNA does hold that much information, it's just not our information.

> - There is no way to retrieve information in this system.

There actually is, the information retrieval mechanism that is used to 'express' the DNA (actually, the RNA, an 'unzipped' strand of DNA, but who's counting) is a wonderful little nano machine called a ribosome, it's probably the most amazing structure that I know of outside of the DNA itself.

They're in the volume quoted, the DNA only occupies about 25% of that volume iirc.

> Reliability of hard disks is far, far better than this.

The error correction mechanism that allows your cells to be copied through very large numbers of generations is actually pretty good, most 'mutations' are lethal and only very few actually result in viable copies passing their changes on to newer generations. Mutations are also pretty rare on the whole.

You are right that only TC and AG are valid, but those combinations can be attached 'in reverse' as well (CT / GA) so that makes for four possible combinations in all.

If it weren't for that the movie 'GATTACA' would have been unpronouncable :)

epistasis2• 16 years ago

- There is no way to retrieve information in this system.

DNA is natively a content addressable storage system, due to natural base-pairing. But to first address your question in your edit: think of DNA as a pair of singly-linked lists, each with an alphabet of four characters. Each singly-linked list is the "reverse complement" of the other: the reverse sequence with an A-T swap and a C-G swap. At each position there's 2 bits of information, and the other linked list allows for some redundancy.

To probe for information in a DNA database, you construct the reverse-complement of the desired bit of information, attach a marker to your probe (such as a fluorescent dye, biotin, magnetic bead), then physically mix it in to your DNA database. A couple cycles of melting and cooling, and your probe will eventually find it's target DNA.

Of course, the thermodynamics of a physical database like this aren't particularly great. I'm not sure of the asymptotic behavior; my intuitive guess is lg(N) just like in B-trees or what have you, but I've never run the numbers or heard of anyone else running it. Also, the constant in front may be just a few orders of magnitude larger than our current systems :)

Reading DNA is getting super cheap these days, and the pace of DNA sequencing technology makes Moore's law look positively wimpy. There are about 30 serious startups working on technologies that fall into a few broad categories, and some like PacBio had an IPO this year. Writing DNA is a much more difficult challenge, I don't know of many people looking into it yet. The market for writing DNA isn't nearly as obvious as it is for sequencing. Of course if it becomes feasible to write your own pets/plants/children instead of breeding them, the market may explode.

kiujhygthyujik• 16 years ago

We can already write individual atoms on a surface with a atomic force microscope. So at one bit per atom position we could encode 10Pb in about 0.1micromoles of xenon atoms on a silicon substrate.

Well under what nature can do.

jacquesm4• 16 years ago

the 'on a silicon substrate' is a bit problematic here, a substrate typically is a lot larger than just a few atoms, the neat thing about DNA is that it is a three dimensional encoding, you are talking about a two dimensional encoding scheme, not three dimensional (an atomic force microscope typically moves in two dimensions and uses an electrical charge to deposit the atoms).

You could stack the atoms three dimensionally but then there would be no easy way to read them out again.

Confusion• 16 years ago

  -1 already eh? [..]

You really oughta know that you should wait a bit for stuff to get overaged over a few hours.

quinndupont3• 16 years ago

My 350GB of music is backed up on Backblaze (a wonderful service), but seeing those visualizations reminded me of the environmental impact of my digital packratting. That's a sizable data center keeping my data happy and backed up.

shadowpwner• 16 years ago

Off topic, but what do you listen to that takes up 350GB? Is everything in FLAC?

wipt• 16 years ago

A fair collection of (DJ) mixes (live or not) and sets can easily account for a few hundred gigabytes. Take Tiesto's "Club Life" and Armin Van Buurin's "A State of Trance" for example. With each set being around two hours or more of high quality mp3, with the former having over 300 episodes, and the latter having around a hundred. These things add up quickly.

quinndupont3• 16 years ago

I listen to a lot of music (from all genres). I only purchase/download full albums (even if most of the songs are not very good), and only listen to at least 196k MP3 (most are at least 256k or higher). Some are Apple Lossless (converted from FLAC). And throw in a few discographies in there and all of a sudden you have 350GB. I've discovered that I add about 50GB/6 months.

rphlx2• 16 years ago

I wonder how many thousands of people are paying them to back up stuff they could easily redownload. Great business model if they de-dupe internally.

kayoone2• 16 years ago

they encrypt all data and i dont think it would be wise of them analyzing their clients data for dupes...if somebody would get to know that their business would be gone.

sliverstorm• 16 years ago

If you're implying they are crazy for doing so, have you ever lost everything and tried to redownload it all? It's a huge timesink, I tell you.

As for de-duping, Dropbox does this. When you drop a Windows ISO or the Office installer into your folder, it hashes the file and realizes 'hey we already have this!' and you don't have to upload it.

skeltoac• 16 years ago

If you took all of those drives apart and put the spindles in a pile you'd have a big pile of motors. The Petabox consumes 6kW so let's be generous and call it 60kW of servo motors.

60kW is roughly 80hp. (I know this is a specious comparison in several ways but bear with me.) 80hp is enough power a small but highway-capable motorcycle. A single 60kW brushless DC motor weighs over 100kg. That's heavier than a good many of us HN readers.

Imagine a motor like that bolted to one long spindle of large platters--say the 99cm platters from a 1961 hard drive.

If these platters were remade with the data density of modern 2TB drives, how tall would the spindle be if this giant hard drive had a 10PB capacity? How over- or under-sized would our motor be?

husein10• 16 years ago

it'd be rad if you could visualize the ratios to better illustrate the scale.

10 petabytes : 10000 terabytes :: 10000 gigabytes : 10 terabytes

Detrus• 16 years ago

The visualization in storage pods is most helpful. Just put the drives, in their current size in a physical space you can compare to your office.

The skyscraper comparison is fun but misleading. The stack of drives would be very very thin. It would work better if drives were put in one box and that box was compared to a room, house, etc..

There can be other visualizations like electrical power consumption compared to a normal computer. How much time it would take to read/write that data. How many processors are needed to read/write that data in reasonable time, compare their surface area to the our unfolded brain, which is a large cloth napkin.

I think these comparisons would be more useful to people in the field than how many books would be needed to store that much data, the standard pop show example.

cperciva2• 16 years ago

Anyone want to guess how much data they actually have stored?

Backblaze's pods are a data-loss nightmare -- lots of single points of failure which will wipe out many TB of data at a time -- and backblaze has stated that they replicate data across multiple pods. Given that the 10 PB seems to be the amount of raw storage backblaze has, I'm guessing that the amount of actual data stored is much less -- depending on what sort of erasure correction scheme they're using, of course. (They're still much bigger than Tarsnap, of course!)

rarrrrrr• 16 years ago

At SpiderOak we get a 3x replication equivalent for about 35% overhead, using Reed-Solomon at the cluster level (on top of RAID6 at the machine level.) Not nearly as expensive as outright replication.

Agree those SATA port multiplies are worrisome. In the beginning, our prototype machines used them to squeeze as many drives into a single machine as possible. They have unusually low tolerance for electrical interference and make it possible for one badly malfunctioning drive to take an entire array offline until manually serviced. We've seen occasions where just touching a cable attached to a port multiplier caused the Linux kernel to emit "dazed and confused" NMI events. I am not brave enough to try them again, even in a redundant setup.

cperciva2• 16 years ago

3x replication equivalent for about 35% overhead

How did you compute this "replication equivalent"?

schumihan2• 16 years ago

Does SpiderOak only provide backup service? Erasure encoding is efficient for cold data. Do you use erasure encoding to distribute the hot data across clusters?

bane3• 16 years ago

Amazing, a discussion of data without comparing the internationally accepted unit of "library of congress". I'm more interested in how many LoCs 10 Pb is? If we built a city of LoCs, how many acres would that city be?

These are the questions that keep me up late at night.

mdemare• 16 years ago

The Library of Congress is 10TB. Zipped, it would fit on a single 3TB hard drive.

bane3• 16 years ago

Perhaps that implies we should start using kiloLoC and megaLoCs?

joanou• 16 years ago

That's a sizable operation and difficult to visualize. I'm guessing their failure points reduce potential damage to minimal levels. At AltDrive, we use ZFS RaidZ2 with a number of hot spares. Six drives would have to fail consecutively on a given machine before any data loss... we replace them as they occur. And we have inter-H/W duplication. ZFS is self healing and makes management easy. Additionally we periodically compare the integrity of the user's AES-256 CTR encrypted data against the database records and make corrections if necessary. From our customer's and white label provider's view, it just works. http://altdrive.com

drhodes• 16 years ago

Or about 9 (American) football fields covered with 4GB DVDs: https://sites.google.com/site/10petabytes/viz

alexqgb• 16 years ago

I had no idea Backblaze had gotten this popular, but I'm glad to see their business growing. If you happen to have the profile they've optimized for, it really is the best thing going.

10 petabytes - visualized

Related Stories