Lessons Learned While Building Reddit to 270 Million Page Views a Month

http://highscalability.com/blog/2010/5/17/7-lessons-learned-while-building-reddit-to-270-million-page.html

by rjim86 • 14 years ago

133 66 14 years ago

ndemoor3• 14 years ago

"Instead, they keep a Thing Table and a Data Table. Everything in Reddit is a Thing: users, links, comments, subreddits, awards, etc. Things keep common attribute like up/down votes, a type, and creation date. The Data table has three columns: thing id, key, value."

I hope they introduced some NoSQL sweetness by now.

encoderer8• 14 years ago

Exactly.

They turned their RDBMS into a NoSQL database that's still slowed-down by all the relational machinery.

Though I'm sure this was an informed choice at the time, I seriously hope nobody finds this advice actionable anymore. Use Cassandra. (Or your NoSQL DB of choice)

rfurmani• 14 years ago

They in an ad hoc way turned postgres into a key-value store, but in reality they have Cassandra running as a "permacache" in front of it and, in practice, everything really just hits Cassandra. It looks like postgres could at some point be phased out.

Source: ive hacked at the codebase to produce arxaliv

jebblue• 14 years ago

The only problem with NoSQL is what happens when one day _you_need_to_relate_data?

masklinn4• 14 years ago

> I hope they introduced some NoSQL sweetness by now.

Postgres has good support for being used as a key/value store. http://blog.reddit.com/2012/01/january-2012-state-of-servers... indicates they're still using Postgres for the main backend, and Cassandra next to it as both cache (it replaced memcached back in 2010) and to store data for some features (moderation log and flairs)

citricsquid• 14 years ago

Article is from 2010, if I remember correctly their architecture has changed substantially since this article.

chaz• 14 years ago

Their most recent infrastructure blog post was from January 2012, and shows that they're using Postgres 9, Cassandra 0.8, and local disk only (no more EBS). I'm curious if the recently-announced provisioned IOPS would enable them to go back to EBS.

http://blog.reddit.com/2012/01/january-2012-state-of-servers...

mdellabitta2• 14 years ago

Probably the better move would be to go to SSD-backed High I/O instances. Netflix did: http://techblog.netflix.com/2012/07/benchmarking-high-perfor...

ecaron2• 14 years ago

Only disagreement (although I feel like I'm arguing w/ Linus about git) is don't memcache session data (lesson 5.) Memcache's 1mb max-block (exceeding that removes too many performance perks to be considered viable) introduces a "I need to constantly worry about my sessions getting too big" mental overhead that isn't worth it.

Go with Redis for storing session data.

batista7• 14 years ago

Why would you need 1mb session data? Or 200K for that matter?

ecaron2• 14 years ago

Because it happens. Typically in places where it shouldn't (e.g. bad programming) but sometimes it does (e.g. enormously complex active system state.)

The point is that there are solutions offering equivalent speed without that barrier, so why select a technology that has limitations?

masklinn4• 14 years ago

FWIW Reddit replaced memcached with cassandra in early 2010 for caches and transient stores.

mdellabitta2• 14 years ago

Wow, now I'm glad I read comments before the article. This article is really out of date. They've come a long way since this was written.

encoderer8• 14 years ago

Even better: Don't use sessions. If there's data you're caching in a session, pre-compute it and store it in Redis (or other suitable NoSQL database).

Going statless wherever possible almost always wins. (And I only say "almost" because I'm not smart enough to know if it really always wins. But for me, it has)

irahul5• 14 years ago

> Even better: Don't use sessions. If there's data you're caching in a session, pre-compute it and store it in Redis (or other suitable NoSQL database).

What do you mean by data you are caching in a session? Session is for maintaining state viz. logged in or not, language preference etc.

How is storing in Redis(or any other data store) any different from storing in Memcache?

> Going statless wherever possible almost always wins.

Using sessions with a data store which can be accessed from multiple machines(no files or in-memory stores) is stateless. The simplest implementation will have a signed cookie with a session id. Your application will have a hook to load the user session from the backing data store before processing the request.

surferbayarea• 14 years ago

that's just ~12 queries/sec..not that huge a deal..

jonknee• 14 years ago

There are 2,592,000 seconds in a 30 month day so actually that's more like 104 page views per second. But that also does not count peaks (peak is likely 250+) and that each page view requires multiple queries.

Either way, no need to hate on the traffic figures of what everyone knows is a very large website.

masklinn4• 14 years ago

Also, it was back in 2009[0]. In December 2011, they were up to 2.07bn page views (~750 pages/sec on average)

[0] according to http://blog.reddit.com/2012/01/january-2012-state-of-servers..., December 2010 saw 829 million page views

arkitaip• 14 years ago

I love reading about how companies scale their BigHuge data but it bothers me that we still haven't reached the point where scalability is a commodity instead of a patchwork of technology that everyone actor solves in their own way.

joering2• 14 years ago

That's because 99% of internet websites will never see 200 million requests.

jeremyjh• 14 years ago

Much fewer than 1% will.

NelsonMinar• 14 years ago

Scalability can only be a commodity if all big sites were built the same way. But they never are. A scalable read-only system is entirely different from a system with a mix of read/write which is different again from a message routing system like Twitter or Facebook.

batista7• 14 years ago

That doesn't mean they cant take advantage of ready-made "functional architectural components" for each of their needs.

So they can pick and match from the "scalable read only" commodity system, the scalable "message routing system" etc.

nostrademons5• 14 years ago

It's because "how to make a website scale" depends heavily upon which website, what it does, and how big it needs to be. Making a messaging queue like Twitter or G+ scale is very different from making Google Search scale. Hell, making the indexing system of Google search scale is a very different problem from making the serving system scale.

You can't really avoid having a patchwork of technology, because it's a patchwork of problems. Instead, there're a bunch of tools at your disposal, a few "best practices" which are highly contextual, and you have to use your judgment and knowledge of the problem domain to put them together.

batista7• 14 years ago

>It's because "how to make a website scale" depends heavily upon which website, what it does, and how big it needs to be. Making a messaging queue like Twitter or G+ scale is very different from making Google Search scale. Hell, making the indexing system of Google search scale is a very different problem from making the serving system scale.

For most websites is not THAT different.

Actually, most have pretty similar needs, and you can sum those up in 3-5 different website architectural styles anyway.

There is far more duplication of work and ad-hoc solutions to the SAME problems than are "heavily different" needs.

ksec• 14 years ago

Well because every site are made different, and uses different tech, with different bottleneck.

But with SSD, RAM becoming a commodity, ( both prices has been dropping shapely ) I/O are much easier to deal with.

And with every release of PostgreSQL / MySQL, Easier Database Replication and more common practice of scaling we are much much better at it then say 2 - 3 years ago.

petercooper• 14 years ago

It's pretty cool how nowadays Redis can cover several of those concerns quickly and simply (open schema, caching, replication..)

dotborg3• 14 years ago

How are PHP, RoR, node.js, Perl/CGI etc. engineers supposed to build offline processing? Crontab?

hythloday• 14 years ago

There are worker queues for at least Perl & Ruby. I have no idea about PHP and JS but one assumes solutions exist.

CWIZO2• 14 years ago

We use gearman (and PHP, but it works with many other languages). It's great.

dotborg3• 14 years ago

yet another piece of software to maintain?

benologist3• 14 years ago

NodeJS is insanely easy to do that with, it's one of the best reasons to use it in my experience:

    var queue = [];

    // request handling
    module.exports = function(request, response) {     
        queue.push("bla"); 
        response.end("bye"); 
    };

    // out-of-request work
    setInterval(function() { 
        // do something with your queue either locally
        // or moving it to an external job queue that can
        // leverage the same script on a different thread
    }, 1000);

dotborg3• 14 years ago

what about concurrency?

rabidsnail• 14 years ago

Node has AMQP and Redis drivers. PHP does too, but I think it would be a mistake to write your workers in PHP.

Lessons Learned While Building Reddit to 270 Million Page Views a Month

Related Stories