How Facebook stores its billions of photos

http://www.facebook.com/note.php?note_id=76191543919

by anirudh • 17 years ago

80 15 17 years ago

That was much more exhaustive than i was anticipating. I'm curious how this stacks up to sites like flickr.

viksit• 17 years ago

@thehigherlife - I was thinking about the same thing recently (an announcement about FB becoming the biggest photo app on the planet). I found two great sources of information on this..

- http://highscalability.com/flickr-architecture

Very deep analysis of Flickr's arch from a couple of years ago - including a link to a presentation to Cal Henderson's slides. I'm surprised he got canned from Y! yesterday though!

mattyb• 17 years ago

Not to be wooden, but if you reply to someone's comment, I don't think there's a need to state @[username].

furburger2• 17 years ago

i don't know anything about cal's departure, but i can say that any large co like yahoo, google, msft, apple etc is more or less impervious against the loss of one or a few technical individuals. his departure may have symbolic value but the machine that is chugging there will keep chugging, for better or for worse

gry2• 17 years ago

No doubt. I am studying this further, for sure.

The info from a public-facing, private company is striking. I'm amazed how much code and knowledge Facebook has already given back to the community given their state. Though perhaps being private and not publicly traded makes it easier. Still, it seems like they have done some things very well and wouldn't want to expose them.

SwellJoe• 17 years ago

Amazing? Really? There are an awful lot of companies dramatically smaller than Facebook that have released a lot more code. 37signals, with maybe a 10th the valuation (if we're being very generous to 37signals), has given away a lot more code than Facebook.

Facebook may be many things, but "amazing" generosity is not one of their characteristics. Call them "somewhat" generous, if you like. Heck, I'll even let "pretty" generous slide. But let's not be over generous in our assessment of just how nice Facebook is to the community. They're nice guys, and all, but they aren't saints.

ars2• 17 years ago

Why bother putting the haystack in a filesystem? Just use a raw partition.

Haystack effectively IS a filesystem - a log structured filesystem.

kmavm2• 17 years ago

I think the haystack folks made the right decision, here. A rule of thumb is that it takes 3-5 years for a novel on-disk format to gel. There are always random bugs that rely on particular sequences of allocations/deallocations, races, etc. Using an off-the-shelf filesystem that allocates the blocks and then stays out of the application's way, like XFS, probably saved these guys at least one year of screwing around.

ars2• 17 years ago

I wonder if they have some mechanism for rebuilding the raid array periodically to handle the inevitable undetected read errors. With raid6 you can actually do this, by comparing both parity blocks with the actual data.

Hard disks typically have undetected read error rates of 1 bit in 1E15, so assuming they transfer 1PB per day that's about 9 errors per day. Which isn't much I guess, but I wonder if they do anything about it.

kmavm2• 17 years ago

(Not speaking officially, obviously. I had nothing to do with Haystack anyway).

They're images. So you flip a bit every once in a great while; as long as the bit isn't part of image metadata, no human user will notice. The next time they load the image, (at least, assuming it's been long enough for it to get evicted from the CDN) they see the right data.

furburger2• 17 years ago

raw io? facebook better go ipo soon, they're running out of abstraction layers to code through

okeumeni• 17 years ago

Even if Facebook end up not making any money, it contribution to the next generation of software engineering will be important. Solving major scaling issues, make Facebook a full scale lab for tomorrow. They will set standards for how to approach some unique problems.

mildweed• 17 years ago

Here's a Flowgram presentation from last year, "Needle in a Haystack", which covers these updates. http://www.flowgram.com/f/p.html#2qi3k8eicrfgkv

How Facebook stores its billions of photos

Related Stories