A Detailed Five Step Twitter Scaling Plan

http://whydoeseverythingsuck.com/2008/05/detailed-five-step-twitter-scaling-plan.html

by pchristensen • 18 years ago

25 59 18 years ago

axod2• 18 years ago

This has been solved many times, many years ago. It's not a hard problem. A one step scaling plan would be "write it properly this time".

chaostheory2• 18 years ago

many things are easier said than done.

axod2• 18 years ago

Of course. But there are solid examples from web1.0 and before that did more than twitter, and scaled to more users. Twitter is pretty simple in terms of functionality.

davidu5• 18 years ago

So wrong it's hard to know where to start...

Retric3• 18 years ago

How about:

1) It's not just a DB problem.

antidaily• 18 years ago

exactly.

hank77717• 18 years ago

Uh... yes it is. All scaling problems are database problems. ALL of them. They can be solved by different strategies such as caching, distributed design, etc. But at the end of the day the problem is always that you cant get to the data fast enough. That is a database problem.

hank77717• 18 years ago

Its fascinating funny how you are so smart you can't even deign to share your wisdom on this. If I am wrong, tell me where. Given that I have spent the last several years studying/developing distributed databases if I just don't know what I am doing I'd certainly like someone as smart as you to tell me.

nostrademons5• 18 years ago

I don't use Twitter, but as I understand it, it's a massively many-to-many messaging service, where it's not unusual to follow hundreds of people's tweets and have thousands of people follow yours. Can you imagine the performance if every message required hundreds of disk writes as the "received" copies get spread across a hundred shards?

cstejerean• 18 years ago

"Given that I have spent the last several years studying/developing distributed databases..."

That doesn't make your opinion of what's wrong with Twitter any more valid than the crap TC keeps printing unless you happen to also have lots of internal details of the Twitter architecture.

davidu5• 18 years ago

To do the writes requires a lookup. To do a read requires a lookup. That's how sharding works. If not a lookup, then a hash.

This doesn't even get into the complexity of scaling your model from 1,000 or 10,000 users where your brute-force methods seem like they will work and applying it to 10,000,000 users where your model will die a horrible death, kinda like Twitter is today.

KirinDave3• 18 years ago

Hank, your design is so high level as to be almost useless to anyone who'd actually benefit from a proposed design. It's not clear what issues you feel Twitter has, or how exactly this architecture would solve them. Usually your blog posts are more cooked than this one. I'm a regular reader, and at first I honestly thought your post was satirical.

As I read your design, one of the biggest problems is that to build up the infrastructure I think (and infer based of my experience with designing these kinds of systems) you are describing is that it's a massive engineering effort in and of itself. You basically end up rewriting a realtime-optimized BigTable or HBase, and that's much to much to expect of Twitter's small engineering team.

People seem to forget that scalability is a tradeoff, like any other aspect of a service. You devote an amount of resources to the problem that makes sense to your business. Despite twitter's downtimes, people don't seem to be leaving the service, despite several competitors. The service is down, but it isn't down so much that it's unusable.

For a site that's grown at such a rapid pace and has such a small engineering team, I think Twitter's scaling plan is probably very reasonable. They're growing their capacity at a rate they can afford, and a rate that they know will retain most of their userbase.

andrewparker• 18 years ago

This is post is little more than link bait (and I guess I'm falling for it by commenting).

hank77717• 18 years ago

With all due respect andrew why dont you share your technical analysis of what exactly I am saying that is wrong here. Btw, have you ever actually worked on any database scaling problems? Just checking.

davidu5• 18 years ago

I have. I have managed systems that handle millions of clicks a day in databases and billions of dynamic (PHP) pageviews on a daily basis.

I am currently managing a system that serves over 100,000 DNS queries per second, each of which is dynamically evaluated as to how we respond to it. And each of which is logged, amounting to over 7,000,000,000 log lines a day, all which get processed, parsed and in many cases handed over to a database store. Yep, we're dealing with BILLIONS of rows of MySQL per day.

bluelu• 18 years ago

His approach isn't so good, as it creates a copy for the same message sent to multiple users.

Better safe the messages in a central store by id and only write the ids to the different pages.

http://news.ycombinator.com/item?id=188080

hank77717• 18 years ago

I considered this, but given that the messages are small and the primary issue is scaling not storage this, it does not seem to me is a good solution. The entirety of everything ever written on twitter is probably not big enough to worry about storage size. The issue is performance. By the way, this architecture is essentially how email works. All your inbound messages are stored in one place separate from outbound messages from the sender. This allows for infinite horizontal scaling.

mtw• 18 years ago

imho, it's rails. rails (and any other ruby framework) is not suited for a real-time communication platform like twitter.

also you forgot about messaging queues.

KirinDave3• 18 years ago

Why would you say that? It's not like the Ruby code is handling that many messages. Even Rails sites don't use ruby for their concurrency and messaging. They push down to a database or to another concurrency level like ActiveMQ. Above them lives another piece of code or hardware to do load balancing.

This isn't even about Ruby, it's about the Rails architecture.

xirium3• 18 years ago

The suggested plan seems reasonable. Yes, if you denormalise users inbound and outbound messages then you store the data twice. Yes, shards give almost limitless scalability. However, could it be that during peak load, the number of messages to be written plus indexes to be maintained exceeds the total available disk seek operations?

wmf3• 18 years ago

So shard more, or get more disks, or get flash or whatever. (Yes, I don't know what I'm talking about.)

xirium3• 18 years ago

> So shard more, or get more disks, or get flash

Your intuition is excellent. These are all valid techniques to alleviate disk seek limitations. They are likely to be successful. Unfortunately, they're expensive options.

You can partially overcome disk seek limitations by performing database batch inserts. So, rather than inserting messages indiviually and updating indexes for each message, you insert 100 messages and perform less disk seeks updating the indexes for each batch. Unfortunately, it can be hard to reliably merge data into suitable batches.

nostrademons5• 18 years ago

You don't store it twice - you store it hundreds of times, once for each recipient. Twitter isn't person-to-person messaging, it's broadcast.

menloparkbum3• 18 years ago

Paying attention to half-baked advice like this is half the reason Twitter is fucked.

edw5193• 18 years ago

These design techniques could very well make a nice impact on performance. Then again, they might not.

Why? They are proposals to solve a problem that has not yet been defined.

Fact is, you don't know what the problem is.

So proposing a database solution to what could just as easily be a memory management or throughput problem is premature.

hank77717• 18 years ago

This is true. Though I can say that if one were building twitter and you didnt either have a design like the one I suggested, or a design with some kind of distributed caching then you are going to have exactly the kinds of scaling problems they have had. In other words if you use tradition techniques for something like twitter you are screwed. However you are absolutely right that (as I said in my piece) they may already, for example have thought of all of this and either implemented or dismissed it because they have a better design, and I have no way of knowing.

menloparkbum3• 18 years ago

This isn't true. You don't need a distributed sharded setup in order to scale. (You probably do if you are limiting yourself to MySQL) I built financial applications way back in the day that had a lot more reads and writes than twitter, and we just used ONE gigantic, super expensive database server.

sutro• 18 years ago

Hank, good article, and good defense of your article here on HN. Regarding the knee-jerk negative responses you've received here, I can only say that you are a more patient man than I. Best of luck with your current project.

A Detailed Five Step Twitter Scaling Plan

Related Stories