Steve Huffman on Lessons Learned at Reddit

http://carsonified.com/blog/dev/steve-huffman-on-lessons-learned-at-reddit/

by giu • 16 years ago

202 31 16 years ago

kingkilr• 16 years ago

Joe Stump gave a similar talk at EuroDjangoCon last year (the topic is officially: dealing with a fuckload of data, but a lot of the same lessons). Unfortunately I have no idea what became of the recordings.

sumeeta• 16 years ago

There’s also Scaling your Python application on EC2 from PyCon 2010: http://pycon.blip.tv/file/3257303/

clemesha2• 16 years ago

Their strong usage of both Memcached and Memcachedb makes me think of my favorite 'all-in-one' solution: Redis.

philjackson• 16 years ago

Cassandra (which they moved to in the end) is also an excellent all-in-one solution. It even supports Hadoop since 0.6.

jim-greer2• 16 years ago

Did they say what forced them into emergency read-only mode this week?

mr_justin2• 16 years ago

He says (transcript is below the video) that he left Reddit in the Fall of 2007 but continued working as a contractor until the Fall of 2009 so he doesn't really address the current events of Reddit.

jim-greer2• 16 years ago

Thanks, sorry I didn't RTFA.

kn0thing• 16 years ago

That transcript is misleading. Steve stayed with reddit until Fall of 2009 when our contracts ended. He didn't leave to work as a contractor.

One thing I should also mention is that our deal was structured in such a way that either one of us could've left reddit before our contract ended without too much pain (missing some carrots, granted) but we were still rather attached to reddit, the community, and our friends (team reddit). Personally, I did also feel a compulsion to fulfill my side of the contract and make sure Condé Nast got its money's worth. That may make me a tool in the eyes of most Hacker News folks, but so be it.

akkartik2• 16 years ago

The approach he recommends - no scheme, extreme denormalization, lots of redundancy - is a strong case for nosql.

bh23ha• 16 years ago

Shouldn't lots of redundancy be a fundamental part of which ever db you are using.

As to extreme do-normalization, colour me sceptical, why not parallelise and throw more hardware at the problem?

dalore• 16 years ago

Because it's more cost effective to de-normalize then to parallelise in this instance.

jbellis2• 16 years ago

Turns out it's difficult to parallelize without also denormalizing...

alexro• 16 years ago

Reddit in fact now runs on Cassandra - the top nosql solution for a social website

metachris• 16 years ago

That's interesting. The guys at reddit previously compared various database systems including Amazon's SimpleDB, and found that PostgreSQL was the fastest (see 'Scaling your Python application on EC2' from PyCon 2010: http://pycon.blip.tv/file/3257303/ )

Appearently Reddit switched over to Cassandra in March 2010 (http://blog.reddit.com/2010/03/she-who-entangles-men.html)

akkartik2• 16 years ago

*schema

bryanh• 16 years ago

So many lectures on scaling are full of best-practices that are somewhat removed from reality and time/budget constraints. This lecture on the other hand has a lot of common-sense advice and solutions that may be obvious to some, but I must admit, this is going to help me (if I could get the traffic to worry about scaling...).

0nly1ife2• 16 years ago

Interesting. I'm an indie Facebook developer. My most popular app has over 8 million monthly uniques and 350 million pageviews. I didn't realize how substantial said traffic was until reading this article. I scaled my app to that size on my own over the course of 12 months. I know of other FB devs who scaled bigger apps working solo. If average hackers like us can do it, anyone can. The hard part is building a popular app. Articles about scaling are a means for founders to brag about their success without giving away useful information about what made their app popular in the first place.

staunch• 16 years ago

Alexis Ohanian has talked about what made Reddit successful many times. Steve Huffman is talking about the technical side of things. It doesn't come off as bragging to me at all.

0nly1ife2• 16 years ago

Bragging is the wrong word, but it is the first that came to mind. And I was excited to brag about the size of my app when I first read his article.

I just think we spend too much time talking about scaling and not enough time discussing virality.

percept• 16 years ago

Care to blog about your experiences?

clemesha2• 16 years ago

Wow, it sounds like your app is extremely popular - that sounds like huge traffic, especially for "scaling to that size in 12 months". Right? I am naive, or is this "fairly common" with decent Facebook games?

What is your application? Are you profitable? Very profitable? (Oh god I hope you are very profitable, or some of my apps are screwed - but then again I guess it really depends on the business model and usage characteristics)

strlen2• 16 years ago

Different application have different scalability characteristics (how much can be cached? how much integration is there?). A certain number may look very impressive for one sort of application, but not so impressive for another.

E.g., some of Yahoo's non-search properties receive tremendous traffic but their scalability stories (while still requiring a great amount of difficult engineering work) aren't as interesting as those of Facebook or other Yahoo properties e.g., Search, Flickr.

niktech• 16 years ago

Direct link to an iPhone watchable video: http://vimeo.com/channels/carsonifiedtv#10506751

dimarco• 16 years ago

a note for Mixergy: this is how you provide a video transcript.

AndrewWarner• 16 years ago

Checking it out now.

Thanks Mark.

evgen• 16 years ago

Let's see:

- crash and let a supervisor restart you

- isolate components via specialist processes and systems

- keep schema/types flexible

- be stateless

- cache & denormalize

- be redundant

- let other processes handle non-realtime tasks in the background

Sounds like he spent a long time learning the hard way that reddit should have been written in Erlang in the first place :)

Hexstream• 16 years ago

Well, they use (or used to use) RabbitMQ, which is written in Erlang.

Steve Huffman on Lessons Learned at Reddit

Related Stories