Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

https://github.com/twitter/twemcache/issues/2

by diwup • 14 years ago

73 37 14 years ago

alttab2• 14 years ago

I enjoyed the jab at twitter's "we are the only website on the planet to have scaling issues" holier than thou attitude. Rails is still trying to get over the character assassination by twitter when they failed to scale it. I know first hand that rails can scale very well. Do bad carpenters blame their tools?

timaelliott2• 14 years ago

Twitter's attitude is laughable. Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging.

achompas5• 14 years ago

Yes, you have a decent amount of traffic but you're also only dealing with ~160 uncompressed bytes (plus whatever overhead) per event. The hurdles you've overcame aren't that particularly amazing nor challenging.

>42M uniques last month.[0] Are you really going to assert Twitter hasn't dealt with amazing or challenging hurdles in getting this far?

[0] http://siteanalytics.compete.com/twitter.com/

EDIT: this ignores that twitter.com is not the only Twitter client--they served 15B (!!) requests/day (!!!) as of a year ago.

Not to mention metadata, instrumentation for services, logging, DB backups, and managing configuration of all of those distributed resources. Are we still talking about the ease of 160B?

http://www.readwriteweb.com/hack/2011/07/twitter-serves-more...

taligent• 14 years ago

Decent amount of traffic ?

Sorry but the only thing laughable is that comment.

InclinedPlane6• 14 years ago

What does content size matter? The challenge is that every single page of content except for each individual tweet is utterly unique for every user. That defeats the vast majority of straightforward caching implementations. You can't cache fully rendered pages ever because the chance that one random timeline view at a given time will be identical to any other view (even by the same person at a different time) is pretty much as close to zero as possible. Every view is dynamically generated content from up to several hundred or thousand different streams of data and needs to be put in order and have all of the per-user metadata set correctly.

Once you start looking into the actual mathematical constraints of the problem of twitter you realize that it's a scaling nightmare. Hundreds of millions of updates per day and tens of thousands of views per second (billions per day). There's only a few people in the world who have the right to look down on stats like that.

jasonwatkinspdx5• 14 years ago

Your assumptions are laughable. Tweets have considerable metadata that pushes them far beyond 160 bytes. Fragmenting this into a secondary object is counterproductive due to the constant factor of 2 requests vs one larger payload.

Someone who's been around the block a few times understands that it's difficult to make pronouncements without informed observation. That you are not willing to extend twitter's engineering staff the benefit of the doubt considering your lack of visibility into their measurements speaks loudly.

InclinedPlane6• 14 years ago

Admittedly I am not well versed in Rails at present. However, the question in regard to twitter isn't so much whether Rails can scale but whether it could scale when twitter needed it too, which was a fair number of years ago. Did Mongrel or Unicorn even exist back in, say, 2007?

jasonwatkinspdx5• 14 years ago

Mongrel did.

aaronblohowiak• 14 years ago

I believe the main earlier problem was that M:N broadcasting in mysql didn't scale well, but I could be mistaken.

kevinh• 14 years ago

There may be a difference between scaling well and scaling 9th on alexa well.

Sometimes bad carpenters blame their tools, but sometimes the tools weren't planned for something of that scale.

SoftwareMaven• 14 years ago

Regardless of the Twitter/Memcache controversy, bad carpenters do blame their tools. Good carpenters succeed in spite of their tools.

krakensden• 14 years ago

That response from manjuraj was kind of evil

floydprice• 14 years ago

Has open source changed? I remember when people didn't fork projects, rebadge them as there own and then promote them over the original.

Now don't get me wrong, I think its amazing that Twitter are opening up these enhancements to the community, but it feels like a kick in the teeth to the memcached folks to slap a twitter badge on it, why isn't this a collaboration that benefits the whole community? you know, like open source used to work.

I know at 34 I'm a dinosaur in this industry but I do try to keep up with the new way of doing things... This just feels wrong to me.

zxypoo2• 14 years ago

I think open source has changed a bit with the rise of the GitHub era. IMHO, I think @mikeal did a good post regarding this change "Apache considered harmful" http://www.mikealrogers.com/posts/apache-considered-harmful....

Even in the old days, it's easier to fork than work with upstream. I think these days it's just easier share those forks with services like GitHub. It should help spread ideas and improved solutions IMHO so downstream consumers actually benefit.

In Twitter's case, they are planning to do what works for them at the moment: "While we initially focused on the challenging goal of making Memcached work extremely well within the Twitter infrastructure, we look forward to sharing our code and ideas with the Memcached community in the long term."

zxypoo2• 14 years ago

Just to clarify things since I helped write the initial blog post about twemcache... in our original post, we had an error regarding the slab calcification problem we mentioned, this problem ONLY applied to our v1.4.4 fork of memcached. After speaking with the upstream maintainers, we learned that recent memcached versions have addressed some of these problems. These are the type of conversations we want to have.

At the time we adopted memcached, that's the version we went with and made sure it worked well in our production environment as we scaled as a company. We also open sourced twemproxy [https://github.com/twitter/twemproxy] which is a lightweight proxy for memcached which has worked well for us in combination with twemcache and may work well for others too.

We just want to reiterate that twemcache has worked well for our unique environment and any teams evaluating memcached should try all their try all their options, just like any other piece of software you adopt in your stack.

One of the reasons of open sourcing our work was to share our ideas with the memcached community to see what worked well for us and help everyone. For example, this is also how we treat our work with our MySQL fork [https://github.com/twitter/mysql] which we maintain in the open and have signed an OCA with Oracle to help get work pushed upstream so everyone benefits in the long run.

khangtoh• 14 years ago

Apparently another Twitter engineer says it's doing 23M/s

"23 million queries per second with zero fucks given"

https://twitter.com/timtrueman/status/222793786345013248

forgotusername• 14 years ago

tl;dr your memcache branch fragments my consulting clients, you should use my memcache branch instead

Twitter says they fixed the Memcache calcification problem. Dormando disagrees.

Related Stories