250x Speed Improvements with Microcaching (and No New Code)

http://fennb.com/microcaching-speed-your-app-up-250x-with-no-n

by taybenlor • 15 years ago

346 72 15 years ago

jbyers• 15 years ago

Nice technique. The number of distinct dynamic pages you expect to get hammered still must be regenerated within that second. With a longer window, some wp-admin or logged-in-user detection, and a third-party comment service, I could see this being a standard nginx wordpress configuration.

mhd• 15 years ago

What comment systems are you thinking about? With Disqus the rest of the page shouldn't really matter at all, whether it's dynamic, cached, static etc…

Depending on one's error tolerance, maybe dynamic configuration would help (check if you're hit heavily and/or check if you're on HN/slashdot/reddit, then modify the config accordingly).

IanMikutel• 15 years ago

What about the Facebook comment system?

michaelbuckbee• 15 years ago

This seems pretty similar to using Varnish or Squid as a reverse proxy (though likely easier to setup).

taybenlor• 15 years ago

Similar sort of approach. A lot less work to set up, especially if you're already running nginx.

wheels2• 15 years ago

> This is fine, up until the point where you get on HN and Reddit at the same time

Incidentally, you don't actually need much to handle that. Our web server is a wimpy 256 MB VPS and we've had (Wordpress) blog entries hit the front page of HN and Reddit simultaneously and weather the storm without missing a beat. An appropriately setup Apache + Wordpress SuperCache does the trick just fine. (Hint: The default Apache configuration isn't "appropriate".) You're not going to hit anywhere even close to 2k requests per second on the front page of those two.

timdorr• 15 years ago

It depends on the content. If it's something that will be visited just by HN/Reddit users: Sure, you're not going to get 2k/sec directly from them. But if it's something that's going to get shared/tweeted/retweeted/blogged/reblogged/etc., then you will be looking at huge traffic numbers. It just depends on how viral your content is and how far it spreads. It's the long tail that matters.

wheels2• 15 years ago

You've really got to hit the jackpot to get up to 2000/request per second (assuming you don't already have a high traffic site). Assuming about 20 requests per pageview, that's more than a quarter million pageviews per hour.

I've had things go moderately viral (thousands of tweets / retweets) and you don't get anywhere near that. Pre-tuning for hitting the jackpot is in most cases going to be premature optimization.

phillco• 15 years ago

Hell, I've been on shared hosting and handled getting frontpaged by HN and Reddit just fine. <3 WP Supercache.

jmtame• 15 years ago

I'm on a 512 Linode, and my Wordpress would freeze when I got on the front page of HN. I ended up turning Apache's Keep-Alive to off and installed W3 Total Cache, haven't had any problems since then.

mopoke• 15 years ago

I like this solution and am definitely tempted to give it a go.

Anyone got any thoughts on the best way to do this on a page with personalisation? (and this is really simple personalisation - one section of the page changes depending on whether you're logged in or not).

My solution would probably be to have the personalised section load as an async request but then you'd need to make sure that the async request can handle the same load as the microcached content.

Any other ideas?

splitrocket4• 15 years ago

That's how I've done it in the past. Depending on how many personalized requests you are doing, compared to the resource utilization savings you'll get from caching the rest of your content, you'll probably come out ahead.

_phred• 15 years ago

You can put any nginx variable into the proxy_cache_key setting, including cookies and query strings. I've used this to cache localized versions of each page on a site based off of a "language" cookie. Async is a good approach for what you need, but it's good to know what nginx's caching mechanism can do.

splitrocket4• 15 years ago

I've been doing essentially the same thing with my wordpress install. Went from a few hundred reqs/second on a cheap linode to over 4k reqs/second. (I think the limit was the benchmarking tool, not nginx) I've got the nginx config if anyone is interested.

dbalatero• 15 years ago

I'd be interested in this, yeah! My GitHub username is @dbalatero if you want to send it via that.

splitrocket4• 15 years ago

Here it is: http://alchemi.st/nginx-wordpress-network-and-fastcgi-cache-...

You can change the caching ttl to meet your needs.

brown9-2• 15 years ago

Is it fair to compare those ab benchmarks when the concurrency value is different for each (4 and 500)?

Would be curious to see how the original config handled 500 concurrent requests.

splitrocket4• 15 years ago

My experience doing something very similar is that the original config would not handle 500 concurrent requests. It's a fair comparison insofar as the uncached app probably couldn't serve that many concurrent requests in a reasonable amount of time.

patio11• 15 years ago

A national radio campaign is likely to bring hundreds or thousands of visitors spread over hours, not hundreds of thousands of visitors spread over seconds, so I probably would not take any particular action to harden a site in anticipation of it. It is a poor use of engineering resources and adds technical risk with no corresponding benefit. (n.b. Pasting code you got from a blog post, particularly code marked as kinda broken, is not a risk-free endeavor! I love nginx, don't get me wrong, but paste in snippets from two different blog posts and watch the sparks fly if you don't understand how nginx handles, e.g., location priority.)

adgar3• 15 years ago

Not to mention that he's compromising on accuracy. Yeah, 1 second is tiny, but why give it up if you don't have to? Especially those of you reading who want to try this on a potentially more dynamic site.

Edit: downvoted with no explanation, cute. And readers here think they're above Reddit standards.

There's no reason to compromise the integrity of a system to any extent before you have to. As others have demonstrated, that need was clearly not met here.

maratd• 15 years ago

> Not to mention that he's compromising on accuracy. Yeah, 1 second is tiny, but why give it up if you don't have to?

While NGINX may use very little CPU and have minimal hard drive access, PHP (or Java/Python/Ruby/etc.) will hit those resources hard.

Look, I'm not one of those environmentalist wackos who pisses his pants when somebody forgets to turn off a light ... but ... if you're keeping your BLOG server a hundred times busier than it has to be just so that your users don't experience a staleness of 1 second ... you're an asshole. A military or medical application, fine. But a blog? Sorry, no.

gruseom3• 15 years ago

downvoted with no explanation, cute. And readers here think they're above Reddit standards.

Downvoting with no explanation is arguably acceptable here. There's disagreement about it, but people do it all the time, including me when I don't have time to type in a comment or feel like doing so would add more noise than signal (like, probably, right now). Complaining about being downvoted, though, definitely is not.

buro93• 15 years ago

> I love nginx, don't get me wrong, but paste in snippets from two different blog posts and watch the sparks fly

This week where I work I have witnessed much the same with Apache2.

The problem: They have a new Wordpress install, want to redirect an existing site path to Wordpress, and want to ensure that the performance rivals that of the existing site.

Before I could get them to just change the Netscaler rules and add a simple Wordpress add-on that creates static files from dynamic, they had cargo-culted their way to a glorious solution that involved leaving all of the routing in place but reconfiguring Apache using mod_proxy, mod_rewrite, and proxyhtmlurlmap such that requests for static items get munged through the static web server, sent off to the Wordpress server, re-written on the way back, and then served up in-place of the existing file.

They literally have an Apache conf that is the sum of 5 different blogs. And they don't understand how it works, and how it is in other ways broken (they can't login to wp-admin anymore, but hey... it's OK, they can copy the Wordpress database from staging).

This has also taken them 2 weeks to achieve.

When asked why they didn't just change the routing rules, add Varnish and configure Wordpress to know what domain it's on (so they don't have to rewrite anything), they simply shrugged and explained that it's working now.

PS: If anyone wants a technical co-founder in London, or an Architect/CTO level person who can hire good people and build good teams. Please speak to me. I'm not sure I can take the madness of working for other people much longer.

pornel• 15 years ago

I see WordPress blogs down all the time. You don't need radio campaign to kill it, sometimes a tweet is enough.

Uncached WordPress instance in the article could handle no more than 100k visitors spread over 3 hours (and that's assuming no spikes).

Of course adding such config last minute is a bad idea, but nobody should be setting up uncached WordPress for more than a few friends in the first place.

crikli• 15 years ago

Is this using nginx in lieu of Apache or nginx in front of apache, acting as a reverse proxy or similar?

nirvana5• 15 years ago

"If you have personalized pages (ie: majority logged-in users) this approach isn't going to work. "

I've considered this problem, and am working on a solution for nirvana[1]. The biggest challenge to this project has been to take a language (coffeescript) that is sequential and run it in a distributed environment, without the programmer having to know distributed programming. One of the techniques I'm applying is making a response (in this case, a web page) the result of a collection of components, which are rendered separately in the same context. (EG: The context is the headers of the request, plus the user record if the user is logged in, etc.)

So, the request comes in, the components are loaded from the cache, they are executed (in parallel) all with a copy of the state, their results are aggregated and that result can run thru templating to produce a webpage that is returned.

The idea then becomes, instead of executing the code for every component in every request, if the component has no context specific requirements (e.g.: it is the same for every user, it's a static element, or it's dynamic, and but doesn't need to be generated every time) .. then it can be flagged as cacheable. The caching would also have a staleness factor (Eg: 1m, 5m 10m).[2]

My hope is that you can have pages that are custom per user, but that also contain heavy impact results (say a graph produced by an expensive operation), where the results come form cache, the static components come from cache, but the user specific parts are dynamically generated each request.

This component approach not only lets the code be rendered in parallel, and often not even rendered, but instead pulled from cache, but it should allow for more convenient re-use of common elements and features across a site.

I hadn't considered caching for just 1s, though. Will have to think about that.

[1] Nirvana is CoffeeScript web development backed by erlang and Riak. Instantly distributed coffeescript. It will be open source, hopefully soon. Follow @nirvanacore on Twitter if you're interested in being notified.

[2] Planned. There are some implications of this that will require tradeoffs, so initially it may just be a flag of Yes/No for "Cache for up to 1 minute." or some value like that.

aidenn0• 15 years ago

Something I have done in the past is put all the "loaded by everyone but not static" stuff in memcached. The application pushes to memcached on startup, and any POST that would change it causes the application backend to update the memcached entry(ies). This is a slightly different approach from the microcaching, but it has the advantage of being always consistent.

I only do this when the stuff starts to obviously become a bottleneck, and I haven't done much with cloud hosting, where RAM seems to be more expensive than cpu cycles, so I don't know how well it would work there.

sciurus• 15 years ago

To relate this even more directly to the article, I believe W3 Total Cache can be configured to do this.

http://wordpress.org/extend/plugins/w3-total-cache/

nirvana5• 15 years ago

Effectively, a Riak in-memory database is memcached, but distributed over all the nodes, with zero administration to grow or shrink it. So, if I've got 10 nodes, and each has 30MB set aside for this caching (or whatever) effectively its a 300MB cache. Adding another node makes it a 330MB cache.

I'm avoiding the logic of deciding whether a POST would change it or not, by letting the components handle it. They can set a default TTL, and they can also, if they process any relevant data, force an update to the cache.

phillmv2• 15 years ago

>Nirvana is CoffeeScript web development backed by erlang and Riak.

OH COME ON, I would almost bet money @nirvanacore follows the great traditions set by @rolfscale and @hipsterhacker.

That said!, good luck :D.

>The idea then becomes, instead of executing the code for every component in every request, if the component has no context specific requirements (e.g.: it is the same for every user, it's a static element, or it's dynamic, and but doesn't need to be generated every time) .. then it can be flagged as cacheable.

How are you going to win over everyone else's caching solution? Why are you going to be easier than

    cache[:foo] ||= Computation.lovers_do_it_expensively() 

?

nirvana5• 15 years ago

I'd love to find a place on the internet where I can talk about my project seriously with people. I'm the only engineer in our small startup, and building this platform to decrease the cost of development for myself going forward. I'm open sourcing it because there isn't a solution like this out here (or I'd just use that instead.) My focus is actually on zero-operations-dudes-on-payroll scalability up to medium sized services, not "it's gonna be huge man" scalability. I'm not using MongoDB because its "wicked fast!!!", I'm using Riak because "no ops guys needed, just add nodes."

Seriously, anyone know such a place? I guess I'll have one when I make the first release, since I'll make a mailing list and issue tracker, etc. But right now, before the first release, when I'm thinking about serious architectural issues, that's the time I could use someone off of whom I could bounce ideas and get feedback, positive or negative. I mean, constructive feedback.

That's why I made the comment on this post, because it's reasonably relevant to what I'm doing.

In answer to your question, I have no interest in winning over anyone else's caching solution. I'm not building a web cache, I'm building a web development platform. This makes web development a whole lot easier for the kind of development I do. I expect it will be useful for others, but even if it isn't, the time invested in it is almost nearly recouped developing our MVP.

buro93• 15 years ago

Varnish supports ESI. What you've described is ESI.

Just put Varnish in front of your servers and used the Expires headers of the responses to cache stuff accordingly. If you include user information in the cache hash of the personalised chunks of the page (i.e. in the URL or cookie) then you can cache that for a short time too.

I do apply short term caching to a lot of things, mostly to protect against double requests for the same thing when there might be an expensive query behind it.

nirvana5• 15 years ago

Thanks for the pointer. There's two advantages to what I'm doing over using Varnish in this way:

1. The component writer knows the needs of the component and can set the caching policy for it there. Other developers can then just include the component without knowing what its needs are. In a way, this means they don't have to design the page for caching.. they kinda get it for free.

2. There are several optimizations I can make using this method, and since it is intrinsic to the need to be able to run things in parallel (components have a clearly defined line between them another components) the caching ability is just a few extra lines of code and so relatively low cost for me to add.

Cherian• 15 years ago

I believe nginx has something similar called SSI

http://wiki.nginx.org/HttpSsiModule

tobyhede2• 15 years ago

THAT SOUNDS AWESOME.

Really interesting idea, I've been working with Riak and CoffeeScript and love the model. Adding distribution is ... inspired. Mad like a fox!

mkjones• 15 years ago

Cool approach - sounds a lot like what we call "Pagelets" at Facebook. Check out https://www.facebook.com/note.php?note_id=389414033919 for a writeup about how we use them.

nirvana5• 15 years ago

Thanks, reading that now. Riak has a pipeline service called riak_pipe. One of the things I've been thinking about is having applications define a pipe, and then sending requests thru the pipe. The pipe is inherently sequential, so it fits well with a sequential language like javascript (you'd just write fittings in the coffeescript/javascript and then at the app level define the pipeline as a series of fittings.) On the other hand, this seems potentially less parallelizable and so I won't know which is more performant until I can run some tests (And I'm not yet to the point where I can do that.)

Anyway, reading the page you linked to now, will update this comment with what I learn from it.

EDIT TO ADD: Have read a good chunk of that page now. THANK YOU! It's very interesting. I'd considered doing something like this already, but had put it off for later. At this point I'm working on just the server side, though I've saved the article and will check it out again when I can had some higher level functionality to nirvana. Thanks!

riffraff• 15 years ago

I believe this is somewhat similar to how Snippets work in Lift, where each snippet renders by itself and coordination is managed by actors.

troels• 15 years ago

I'm doing client side includes for personalised content. So, the main page is served with no personalised content and with all the heavy caching that this allows, and the html has place holders for the personalised content. Then, upon page load, an ajax request is fired that loads the user-specific content and injects it to the page. It works quite well and since this is specifically personalised content, I'm not too afraid of having to support non-js clients (Such as crawlers etc.).

rymedia• 15 years ago

Very impressive.

simonw2• 15 years ago

I really like the idea of setting a cookie that bypasses caching for a few seconds - I've heard the same technique used by Facebook, who set a cookie that ties you to the MySQL master server rather than the slave after you perform a write action so that you'll see your update without waiting for replication lag.

It relies on using the Max-Age Cookie argument though, and I was under the impression that IE doesn't implement that correctly. Anyone know what the status of IE and max-age cookies is?

Fenn• 15 years ago

Hey Simon,

Hmmm, you're right about the IE issues (http://mrcoles.com/blog/cookies-max-age-vs-expires/). I haven't looked into how possible/hard expires would be in nginx config, but you could probably hack it in with http://wiki.nginx.org/HttpUserIdModule

simonw2• 15 years ago

The problem with the expires argument is that you have to set it to a specific date, but I don't think you can be sure what timezone / clock setting the user's browser has - which means that it's pretty much impossible to reliably set a cookie that expires in a couple of seconds time.

Hopefully I'm wrong - I'd love to know if there's a workaround for that issue.

todsul4• 15 years ago

The big issue here is the application type. This only works well for a very particular type of application. That is, a highly dynamic site that is NOT dependent on user logins.

1) If the site is only moderately dynamic, you can just use plain Nginx and set fastcgi_cache to a few minutes or hours. Much less load on the server. I like to keep things simple, I wouldn't even bother with Apache. Porting rewrites to Nginx is super simple.

2) If the site is customised on a per-user basis, 'microcaching' will break the site and have disastrous consequences. Every user will see the system customised for whichever user primed the cache.

My primary website is user based. That means this 'microcaching' concept wont work at all. It would be catastrophic.

That's where Varnish comes in with ESI. I really wish I didn't have to use Varnish. It's slower than Nginx, it adds another layer of complexity, and in testing, it seems slightly flaky. But what Varnish+ESI allows is caching of parts of my page that aren't user specific. I.e. header, footer, etc.

If you want to see my test results of Nginx vs Varnish, see http://todsul.com/nginx-varnish

chlee• 15 years ago

I thought about incorporating Varnish into my side-project as well. However, Varnish seems to be a [potential] hassle when you account for VCL, ESI, and etc. It also adds an extra layer or indirection and complexity as you've mentioned.

With regards to caching user specific parts of your page, have you considered template caching [1, 2]? This way you can apply different caching policies to different parts of your pages w/o using varnish and ESI.

With that said, why would one choose Varnish + ESI (or reverse proxy in other words) over template caching + memcached/redis/riak? Can someone explain the pro's and con's of both approaches?

[1] http://www.makotemplates.org/docs/caching.html [2] http://jinja.pocoo.org/docs/extensions/

slyall• 15 years ago

1. A reverse proxy will usually be faster since it doesn't have to execute an "code" (even a cut-down framework).

2. Some sites will be "read-only" and your really can serve everyone with exactly the same content.

3. Legacy codebase/programmers may not be able to use more scalable techniques so a reverse proxy is the best solution.

todsul4• 15 years ago

Yes, we previously used template caching because it was built into Symfony1 (an MVC PHP framework). Symfony2 now leaves caching to the HTTP Standard, plus integrates Akamai's ESI. Symfony specifically recommends Varnish for live projects.

Memcache certainly has its place, but for me, Varnish is a plug-n-play solution that doesn't require code changes. Even with ESI, if you've already set HTTP Cache, you just need to set s-maxage per component. That's it. Then of course there's all the Varnish load balancing features that I haven't used.

If my site didn't use logins, I'd definitely just go for Nginx + fastcgi_cache. People argue that Apache can do the same with the right tuning. But there's something to be said for a 5 minute VPS build by a complete novice that can serve 20,000 requests per second using about 10 lines of config code.

I think this all becomes much more complicated for seriously large websites. That's where memcache/redis et al. join the party, potentially with Varnish.

megaman821• 15 years ago

In my own use and benchmarks I have seen seen Varnish is faster than Nginx for full-page caching. Varnish with ESI will be slower of course. I think ESI is better for Facebook levels of personalization for light personalization ('Welcome, Bob!') using a cookies/javascript is better since it allows you to use full page caching again.

klaruz• 15 years ago

I agree, something seems wrong with that test. Varnish performance should not top out with the exact same performance no matter the concurrency. Here's an older article that shows with some very basic changes you can get 27k/s on an old single core machine: http://kristianlyng.wordpress.com/2009/10/19/high-end-varnis...

todsul4• 15 years ago

How can that be? If Nginx serves static files much faster, and full page caching is just a static file, how can Varnish be faster?

bbrizzi• 15 years ago

Anyone know of an equivalent in Apache?

mike-cardwell• 15 years ago

The few times I've been on the front page of Slashdot it has eclipsed the traffic that I've had from being on the front page of Reddit. Hacker News barely causes a blip.

jbarham• 15 years ago

Given that I'm currently personally setting up a caching cluster to host www.melbournecup.com (aka "The Race That Stops a Nation"), being able to gracefully handle a huge spike in traffic is something that is very much on my mind! :)

The site itself is developed in Django and so far I'm just planning on putting a bunch of Varnish caches (behind a load balancer) in front of the Django server. I'm using the very nice Django Varnish app (https://github.com/justquick/django-varnish) in the Django instance to automatically purge pages from the cache as they're updated.

I'm deliberately trying to keep the setup as simple as possible, but the goal is to have a fast site and fresh content.

Tips from others who have handled similar traffic loads would be very welcome!

ck2• 15 years ago

The only way to make wordpress fast and responsive is to bypass wordpress entirely.

You don't need to do anything complex - just install wp-super-cache, set a long timeout, and most importantly add the .htaccess rules it generates to bypass wordpress entirely and serve the cached static files directly.

zzzeek• 15 years ago

Microcaching at the page level is of course a great idea for a dynamic app, but only works if the content being served is identical for all users - in which case why not just use static pages. Oh right, because we only know how to use Wordpress. Ditching wordpress for a static generator should be the preferred route, if possible. (use disqus or similar for comments).

The vast universe of truly dynamic apps that we write in Rails or Python or whatever usually have page elements that are specific to the user's session - "Welcome John Smith" and all that (edit: oh i see he mentioned that at the bottom). So page-level caching isn't feasible there, unless like in the case of disqus you're using javascript to inject personalized content from another server. But for a really interactive web application where coarse grained solutions like this aren't feasible, I'm still a proponent of page-component level caching, something you normally do in your app layer, not the web server layer.

eli• 15 years ago

I think I'd rather just install Varnish than try to reinvent it in my ngnix config. As a bonus you'll get faster serving of static assets too.

250x Speed Improvements with Microcaching (and No New Code)

Related Stories