Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

http://highscalability.com/blog/2012/5/21/pinterest-architecture-update-18-million-visitors-10x-growth.html

by Anon84 • 14 years ago

118 86 14 years ago

highscalability.com

pajju2• 14 years ago

They have a very Interesting Stack as seen on Quora[1]:

-Python + heavily-modified Django at the application layer

-Tornado and (very selectively) node.js as web-servers.

-Memcached and membase/redis for object and logical-caching

-RabbitMQ as a message queue.

-Nginx, HAproxy and Varnish for static-delivery and load-balancing.

-Persistent data storage using MySQL.

-MrJob on EMR for map-reduce.

[1] http://www.quora.com/Pinterest/What-technologies-were-used-t...

edwinnathaniel16• 14 years ago

Pardon my ignorance: what is interesting about Pinterest stack?

I think by now most startups, YC or not, in Silicon Valley will pretty much have a similar setup:

1) Choose the main web-stack (Rails or Django)

2) Choose the API framework (node.js, or something else)

3) Memcached for caching (or some NoSQL)

4) A message queue (ZeroMQ, RabbitMQ, or something ...)

5) nginx, HAProxy, Varnish, (or similar technology)

6) Hadoop for map-reduce (only if you need to go that path...)

encoderer• 14 years ago

Not trying to be a nit-pick, but wanted to mention that ZeroMQ is not a message queue in the sense that RabbitMQ (and Kestrel, Beanstalkd, SQS, etc) is. It's a socket library that you'd use to implement those technologies.

And while this is truly unimportant, my experience with bay area startup stacks is that Redis has very thoroughly supplanted Memcached for most use cases. Though most still have a memcached cluster or two.

That being said: 90% of software development in this world isn't done at tech companies and startups. So from that perspective, that stack is interesting.

krobertson• 14 years ago

I agree. I think the interesting aspect is that they built a service that serves an impressive number of users on fairly common technologies.

There isn't anything very revolutionary about their stack by itself. FB didn't buy them for their technology... they could have replicated that on their own.

If they want to be talking about infrastructure, the story of using common tools at high volume is more interesting that just what tools...

raverbashing4• 14 years ago

Oh, I wish it was that simple.

You can't mix and match (you can theoretically)

Let's say you began with Django, right? Or there was something already ready in Django.

And then you begin to see the warts. But ok, you keep churning along.

And the more you churn the more of a specialist in the deficiencies of each technology you become. Like the fact that Django's ORM runs like a dog.

Or see all the "we moved to MongoDB and we regret it" discussions

From that list, there are two technologies I would recommend strongly (if you know what you are doing): nginx and Redis

The rest needs to be handled with care. And Redis is very powerful if you know how to use, but it's easy to go the "lazy way" with it, and it will be fast, but not as fast as it can.

wamatt• 14 years ago

Seems like a sensible stack.

However, is it not reasonable that would someone not consider node.js, as web stack as well?

I will agree the focus is more towards backend and asynchronous API's traditionally, but the amount of front-end/UX orientated modules are growing every day.

swah2• 14 years ago

I would think that Nginx + Django + SQL, and the rest is premature optization...

rudiger• 14 years ago

It's cool that we're seeing companies reach 1 billion USD valuations with 12 employees.

blantonl• 14 years ago

It will be even cooler when we see companies reach 1 billion USD net-income with 12 employees.

edwinnathaniel16• 14 years ago

I heard Microsoft Visio team probably have less than 50 people working on it (granted the name Microsoft and the Office brand provide a huge selling point).

Visio almost hit $1B according to insider... (can't verify).

shashashasha• 14 years ago

To be fair the employee number is as of last December, and they've grown quite a bit since then.

mquinlan• 14 years ago

Based off of their team page, Pinterest is up to 31 employees.

http://pinterest.com/about/team/

bbayer• 14 years ago

Those kind of articles makes me feel so stupid. Even though those technologies are ready to go, making them work smoothly without any interruption always seems hard to me. I believe, there has to be lots of tips and trick. ( in other words you have to be experienced with all them or I am very lazy) I was wondering if any common receipt exists or if there is someone who can answer my couple of high scalability questions.

raverbashing4• 14 years ago

Yes, see "heavily-modified Django"

(even though this is probably around scalability and connecting with other technologies)

Scalability and uptime is still hard

Still, Pinterest is laying on a great infrastructure (EC2, Elastic Map Reduce) and using it to the fullest.

kmfrk• 14 years ago

Pinterest has performed quite awfully and has been very, very slow up until now. They may have fixed the problem, but it's not necessarily a model to follow. It's nice to know that even a site like Pinterest can't get it quite right all the time.

teoruiz2• 14 years ago

According to the article Pinterest is spending on AWS EC2 >$30k to support 18M visitors/month.

Data: $52/h (peak time, let's say 18 out of 24 hours) and $15/h (night time, let's say 6/24).

Edit: as pointed in the comments $30k/month would only be the EC2 costs.

carson• 14 years ago

You are just counting EC2 cost. The AWS cost for 410TB of S3 storage is around $39k. You would need to add in BW cost on top of that.

It is also interesting that they seem to be using Akamai for a CDN instead of Cloudfront so not a completely AWS based solution.

I wish they went into what they are storing in S3. 410TB is a lot of storage. My initial guess was cached images but 80M objects breaks down to 5MB per object and that is a lot more than what is needed for image caching.

kitsune_• 14 years ago

From these figures they seem to burn over 100k per month on outsourced cloud services alone. Holy shit.

teoruiz2• 14 years ago

That's actually right, S3 costing at $39k/month seems completely crazy.

No wonder 37signals decided to switch to their self-hosted storage solution[1].

[1] http://37signals.com/svn/posts/2483-nuts-bolts-storage

whalesalad5• 14 years ago

It's nice to see that two companies valued at >= $1 billion are running a Python/Django stack. #teampony

lionheart• 14 years ago

Yeah, I'm actually in the process of switching to Python/Django from Rails, so its great to hear how well it scales.

whalesalad5• 14 years ago

I'm curious as to why you're switching. Care to share? I'm a long-time Django guy who's just been thrown into some Rails projects so I am going to be learning it shortly.

irahul8• 14 years ago

At 18m visitors, I don't think your choice of Django or Rails matters. DB tends to be the limit for even the normal applications, let alone something with that kind of volume.

pajju2• 14 years ago

Check this list out, Running Python - Django [1][3][4]:

-Instagram[0]

-Pinterest

-Disqus

Disqus.com - Disqus serves over 3 billion page views, and more than 500 million unique visitors a month on it's Django stack. As far as we know we are the largest installation out there.

-Mozilla[2]

addons.mozilla.com and support.mozilla.com

250k+ add-ons, 150 million views per month, 500+ million api hits per day (firefox checking for updates!

-Justin.tv(they moved to Django from Rails)[3]

-NASA

-National Geographic

-Canonical

-Bitbucket.org

-Discovery Networks

-Intel, AMD, HP, IBM

-Lexis-Nexis

-The Library of Congress

-The New York Times

-Orbitz

-PBS

-Rdio: Huge traffic Radio site.

-VMWare

-Walt Disney

-The Washington Post.

-lanyrd.com

-OSQA Sites. OSQA is an Open Sourced similar copy of Stackoverflow, a QnA community. AFAIK some 5K sites are powered by OSQA Stack. OSQA is built on Django.

-Youtube, LinkedIN, Google, NetFlix, Amazon: Python Stacks.

-GMail, Google Calendar, AdSense, AdWords, Android MarketPlace to name a few of the heavy hitters from Google are on Python - Google App Engine.

[0]http://techcrunch.com/2012/04/12/how-to-scale-a-1-billion-st...

[1]http://jacobian.org/writing/django-community/django-communit...

[2]http://reinout.vanrees.org/weblog/2011/06/06/large-mozilla-s...

References-

[3] http://stackoverflow.com/questions/886221/does-django-scale

[4] http://www.quora.com/Django/What-is-the-highest-traffic-webs...

tferris2• 14 years ago

Thinking also for a while to switch—what was the trigger that you switched from Rails to Python/Django?

bsg75• 14 years ago

> Sharding is used, a database is split when it reaches 50% of capacity, allows easy growth and gives sufficient IO capacity

Nice, compared to the usual, "The database was at 100% capacity, then we tried to shard/partition, and it did not go too well."

ankimal• 14 years ago

Small, cohesive goal oriented teams with clear well defined goals can achieve so much.

al_james2• 14 years ago

Does anyone else feel that 410TB of user data seems quite a lot? If I have my maths right, even if all the 80 million objects are user data (as opposed to, say, logs) thats 5.3mb per object. Considering that most pinterest photos are from the web, that seems quite big.

moe4• 14 years ago

Yes, that number is ridiculous.

For reference, that's about four times bigger than the iTunes music catalog (20mio MP3 files * 5MB average filesize = 100TB).

skeletonjelly2• 14 years ago

5MB seems a bit small, they'd also have lossless copies right?

tomwaddington2• 14 years ago

Original image plus two (or more) scaled thumbnails. Also, I don't think they're doing any reuse (upload the same pin for 2 different S3 objects of the same image). 5.3Mb / object is still really high though.

al_james2• 14 years ago

Wow, if they are not doing any reuse that really really dumb. I can't back this up, but would have thought at least 50% of all pins are repins.... if that 410TB figure is correct, this oversight is costing them $18k a month. It would be simple to have a separate asset id and pin id...

neovive• 14 years ago

Interesting. I'm migrating some old PHP apps over to Python and have been learning Flask + SQL Alchemy. Why would a site with so much traffic choose a full stack framework like Django that required so much modification?

skeletonjelly2• 14 years ago

Because when they started they didn't have the traffic, quicker to get to a running product. Probably easier now to cut back/change Django functionality than rewrite.

Pinterest Architecture Update - 18M Visitors, 10x Growth, 12 Employees, 410 TB

Related Stories