Researching a new form of HTTP caching optimization

http://blog.phusion.nl/2015/01/06/researching-http-caching-optimization/

by ohcoder • 12 years ago

57 49 12 years ago

blog.phusion.nl

samarudge2• 12 years ago

the Varnish HTTP cache has been used very successfully to speed up WordPress. But Varnish doesn’t help a lot with logged-in traffic This is caching that Varnish and other “normal” HTTP caches (including CloudFlare) could not have done

Varnish supports the ESI (Edge-Side Includes) standard, which allows it to cache fragments of a page, and for the cache server to build them again. It also allows you to completely bypass the cache for certain fragments. This is also supported by a number of CDNs (Fastly, Akamai). I've used the ESI technique several times and have been able to achieve a >98% cache hit rate on Fastly for a site with dynamic per-user content. Even the cache misses are only responsible for rendering a small component of the page

stephenr2• 12 years ago

Exactly.

I'm not really sure what situations this built-in cache would be more effective than the likes of a well-tuned Varnish.

FooBarWidget7• 12 years ago

Good to know. Using edge side includes may be easier than trying to change the app to a semi single page app. But that only solves half of the problem. The other half is varying the response based on the value of a specific cookie.

I've updated the blog post with information regarding edge side include.

samarudge2• 12 years ago

Varnish also supports custom cache keys via the VCL function `vcl_hash` as documented in https://www.varnish-software.com/static/book/VCL_functions.h...

I couldn't (quickly) find documentation on how to get the value of a specific cookie, but the server could send a user ID in a header or something Varnish can easily access to be used in the above function.

sandstrom• 12 years ago

I understand that you want to offer something that 'beats' varnish, and it shines through in the article.

But I don't think it matters if your cache is better than every other cache. Rather, as long as you offer a convenient, easily implemented cache, built into the webserver, that's great in itself. We're using Passenger on all our production servers and are most satisfied, because of its ease of use.

Perhaps you could just write "this could be accomplished with Varnish, which has a lot of benefits for advanced cases, but we think our cache will be useful for those that prefer not to manage a separate caching tier."

cheald3• 12 years ago

Varnish also supports plugins for extreme flexibility. For example, I wrote a plugin for our Varnish install which performs HMAC validation of a specific signed cookie and then sets a header which is used downstream in the caching rules.

Varnish is mature, powerful, and fast as hell. It would take a lot of work to reach a point where I'd swap it out for something else.

stephenr2• 12 years ago

The part about inability to speed up authenticated page loads fails to take into consideration things like ESI.

If the majority of your page is still the same for logged-in users, but they see some pieces of content personalised for them, breaking them out into individually request-able components means you can let software like Varnish or a CDN rely on it's cache of the main page content, and make a very small (and ideally simple to process on the backend) request for the per-user content.

It took me some hunting to find it (no docs for v5 yet apparently) but if your stack already includes a caching layer (e.g. Varnish or a CDN) you may want to disable this extra cache using the config described here: http://blog.phusion.nl/2014/11/25/introducing-phusion-passen...

rcarmo• 12 years ago

Hmmm. I might be missing something here, but I routinely clean out cookies in "public"/unauthed URLs in Varnish, as well as hashing the cache based on _part_ of a specific cookie (the bit that defines, say, the site's theme, or a generic user role).

They do mention that they didn't investigate how to do this in Varnish, but I recall having picked up the basic technique from one of the author's posts.

michaelmior• 12 years ago

This was my first thought too. You can vary by any subset of cookies in Varnish if you simply normalize the Cookie header by stripping out irrelevant cookies. Another trick I've used is to parse the cookie to look for the user ID and set a User-Id header on the backend response and then set Vary: User-Id.

chris_overseas2• 12 years ago

I had a contract at a company about 13 years ago where I was working on a web-based CMS that had been built entirely in-house. Because each rendered page's content was built up in a hierarchical manner, I added a caching layer that allowed arbitrary portions of the rendered page "tree" to be cached all the way up to the entire page and HTTP response if possible. Each cached portion could be located quickly based on its dependencies (template ID, content ID, CSS etc). If any edits were made to the site, only the relevant portions needed to be flushed and re-rendered. User-specific portions could be cached in the user's session rather than site-wide. (Thinking about it now, each portion could have been rendered in parallel too, though this was back in the days when multicore machines weren't very common and it wasn't something that occurred to me.)

I built this without giving much thought to whether anyone else had attempted something similar. Presumably they had though I do remember being disappointed when researching the various open source caching libraries, they didn't offer much help at the time. Overall I was pretty happy with the way it all worked and the performance boost it provided was like night and day.

Sadly the company is no longer operating, presumably the CMS codebase is long lost.

jacquesm• 12 years ago

Most major frameworks include something like this but it's usually harder to use, 'partials' caching as it is called is a pretty effective way to speed up the server side of things when you need that.

Varnish has a similar facility (which allows you to abstract it out of the framework code entirely).

A dependencies ('make') like automatic approach to this would be quite nice to have, maybe your approach could be retro-fitted onto one of the existing CMSs?

chris_overseas2• 12 years ago

Thanks, I hadn't heard of the names "Russian doll" or "partials" before. I haven't worked with a CMS (or even web apps at all really) for over a decade now but I'm surprised you seem to imply that automatic dependency management isn't standard stuff these days already. It was certainly very useful to us, though admittedly tricky to implement robustly.

megaman8212• 12 years ago

I have seen this referred to as the "Russian doll" caching strategy. You can find several examples using Rail's partials or Django's cache template tag.

cordite• 12 years ago

I believe Stack Overflow does some of this too. Though I can't seem to find the source that convinced me of such.

est• 12 years ago

uWSGI has a powerful cache mechanism, and it's clustered.

ddorian43• 12 years ago

and you should be using it:

http://cramer.io/2013/06/27/serving-python-web-applications/

Argorak4• 12 years ago

Fragment caching of non-anonymous things on a cache is nothing new. Varnish supports it with ESI, other caches support that with other strategies.

I've successfully build SPAs using that strategy. It's basically Rails russian doll-caching on a dedicated process.

WimLeers3• 12 years ago

What if the cache can parse cookies and vary the cached response by the value of a specific cookie, not the entire header?
This is caching that Varnish and other “normal” HTTP caches (including CloudFlare) could not have done.

This is false. I'm not at all very familiar with Varnish, but I know this is easily possible, and has been used for many, many years.

E.g. for Drupal + Varnish, i.e. to only keep Drupal's session cookie, I found these examples, in less than a minute of googling:

- https://www.varnish-cache.org/trac/wiki/VarnishAndDrupal

- https://www.lullabot.com/blog/article/configuring-varnish-hi... (grep for "inclusion")

Everything in this article has been well-known for at least half a decade, yet is being presented as major technical breakthroughs. Too much marketing, IMO.

jgrahamc• 12 years ago

Yes, and we (CloudFlare) offer different caching based on cookies.

FooBarWidget7• 12 years ago

Can you link to the relevant documentation?

FooBarWidget7• 12 years ago

The first example looks nothing like parsing a specific cookie. It merely sanitizes the cookie headers a little, but it doesn't extract out a specific cookie to use as cache key.

And both examples you link to remove cookies. That's not what we're after. We're after the extraction of a specific cookie without removing anything.

It's also not marketing for a commercial product. The research is for an open source project, and the code is public and open source. The entire point of the blog post is to call for research participants who could not only test our ideas, but who could also point out anything we might have missed.

WimLeers3• 12 years ago

The first example looks nothing like parsing a specific cookie. It merely sanitizes the cookie headers a little, but it doesn't extract out a specific cookie to use as cache key.

You say you don't parse a specific cookie, but doesn't extract a specific cookie as a cache key? The blog post says the opposite:

We modified Passenger to parse cookies, and to vary turbocache responses based on the value of this user_id cookie. We invoke Passenger like this: passenger start --vary-turbocache-by-cookie user_id

In other words: parse the user_id cookie.

The mentioned commit even adds a ~250 LoC file (ext/common/ServerKit/CookieUtils.h) to do cookie parsing: https://github.com/phusion/passenger/commit/a760649cd79fde43...

---

Anyway, what you're getting at is that Phusion only parses a certain value from the cookie and then uses it as a custom Vary header, whereas the examples I linked to clean up the Cookie header and then varies on that cleaned up Cookie header. That boils down to exactly the same thing.

P.S.: downvoting? Not very nice.

hannibalhorn• 12 years ago

Even ignoring the fact that edge side includes are the way to go when confronted with this problem, it sounds like they're basically saying they'd like to include the userid in the Vary header, but cannot because the Cookie header includes a bunch of other stuff.

Instead of parsing the Cookie header and doing a bunch of additional work in the cache layer, why not just add a separate custom header (X-User-Id, say) and Vary on that?

WimLeers3• 12 years ago

You're talking about response headers. But this deals with sending responses, you only get requests. Request headers can't contain a custom header that you control (unless your JS is making the requests of course). Hence you have to transform the Cookie header into whatever you need.

megaman8212• 12 years ago

I almost always configure Varnish not to cache on Vary Cookie and pair that with some simple changes on the back-end to keep most my pages cacheable.

One, don't blindly set Vary Cookie on every single page. Two, when a page only needs minor variation like a username; store the username in a cookie and use JavaScript to display it on the page.

lstamour4• 12 years ago

Yeah, the biggest advantage ESI offers is not requiring JS. If that's important to you, configure it and do extra rendering on the server-side. If not... :)

sandGorgon3• 12 years ago

Sorry to barge on this thread - but does have a reasonable varnish config that works for a wordpress site behind https nginx and W3 Total Cache ?

It seems strangely hard to configure something like this.

lstamour4• 12 years ago

I used (and modified) one based on Dreamhost's https://github.com/dreamhost/varnish-vcl-collection/blob/mas... but there's a bunch of others on Github if you search for them. Note that this isn't a shortcut so you don't have to learn VCL... and that the way I did it was to put Varnish on port 80 and the web server on another port. I've seen other configs where a load balancer asks Varnish when it knows varnish should have the answer and otherwise sends traffic directly to the web server. And sometimes your config will vary based on asset type or domain name -- e.g. static.example.org assets might be cacheable forever while your site's pages might only require a 2 minute cache. You can invalidate cached-forever pages with commands for Varnish or by varying the URL using a "cache buster" hash or string in the asset filename. Oh and don't forget to set the storage type Varnish uses to malloc instead of file.

sandGorgon3• 12 years ago

thanks for this !

I guess you dont have an SSL termination point in front. What I was thinking of was nginx -> varnish -> nginx.

codingjester2• 12 years ago

I think it's been said enough here but Varnish can certainly do almost everything that they are saying it can't do, including some other storage optimizations like storing the gzipped response and serving a non-gzipped when requested (as of Varnish 3.0).

You can use Vary for tons of caching optimizations via varnish, such as caching mobile web-pages vs non-mobile web pages or just a particular header. It's all about just flexing a little bit of VCL (which I'll admit sometimes can throw people off).

They had me until this part:

This is an HTTP cache built directly in Passenger so that it can achieve much higher performance than external HTTP caches like Varnish.

And since they have no benchmarks to really back up these claims, I'm skeptical they did much research against Varnish to tune or set it up. I'd love to see the numbers on varnish vs their turbocache. Without numbers, I have to take a lot of it with a grain of salt.

Either way, seems like it could be an extra handy thing to have in your toolbox, as long as it fits your stack.

FooBarWidget7• 12 years ago

You're reading too much hostility in it. The article does not claim to be better than Varnish. Performance is achieved by not implementing many features. For example, Varnish is a full-blown programming language and has an infinite size. The Passenger turbocache has almost no configuration options, does not support any sort of custom programming and only supports 8 entries at most. The fact that the max size is so small allows it to be implemented in very compact data structures, but its usefulness is also severely limited. It's merely designed with a different set of tradeoffs than Varnish.

It isn't that difficult to make a web server that's faster all the production servers out there. For example H2O is faster than Nginx... because it has infinitely less features. Ditto for Passenger's turbocache.

The rest of the article describes some ideas, some of which cannot be implemented using pure Varnish and require cooperation from the app. In my opinion you're missing out on a lot of interesting ideas if you stop reading only because you think Varnish is being slammed.

codingjester2• 12 years ago

Fair enough! I can definitely see how I might have taken it too hard (or it came off in my writing of the comment)

I was just more or less intrigued by the actual claim and would have loved to just see some numbers on "higher performance than varnish" and the scenarios where it could reach those performance pieces. It just piqued my interest for future architecture considerations.

Thanks for enlightening me more on turbocache. It's on my list of things to go through tonight.

Edit: I did want to point out my bad choice of words. "They had me until here.", did not mean I didn't read the article in it's entirety. Just that I continued with a bit more skepticism.

Argorak4• 12 years ago

I think the complaint is just that the approach is in no way as novel as you make it seem.

It can and has been implemented with Varnish and a few lines of VCL (with support of the app as well, of course).

What is "pure varnish" anyways? Proper varnish use always needs a tuned VCL, that's the whole core of the product.

Also, if I might say, I find your edits very handwavey. Please prove why that is impossible in VCL.

markcampbell• 12 years ago

I've left some comments on the Disqus thread on the blog, but I'll reiterate my concern about the security of the cookie being set.

The cookie being set is unsigned according to the documentation[0] on rails, so a user could modify it and send it back to get a different cached response. Say that they saw that the user level was being set in there (like in the blog post) and they change the value to the 'staff' value to get the staff cached response. Probably not a good idea!

With that said, I don't think that this technique is adequate right now when you have user access level concerns and you're relying on that piece of unsigned information to not be tampered with.

[0] http://api.rubyonrails.org/classes/ActionDispatch/Cookies.ht...

marvy• 12 years ago

Switching to a single page app seems invasive. Why not just have an iframe for the username and stuff?

MichaelGG• 12 years ago

Normally, non-cacheable page fragments would make every page uncacheable.

AFAICT, ASP.NET supported partial output caching since 2003. And, it can store multiple versions of a cached item via a user-defined parameter (a property on the cached control class). It seems it might be possible to simply create a property that returns the user ID and let the cache vary on that.

Of course, this uses the Web Forms controls-based system, which isn't very popular. But calling this type of caching new seems a bit of a stretch.

http://msdn.microsoft.com/en-us/library/k4he1ds5%28v=vs.71%2...

Edit: And here's a link talking about VaryByCustom, complete with example code that uses the version of the browser to generate unique per-browser-version caches.

http://msdn.microsoft.com/en-us/library/5ecf4420%28v=vs.71%2...

jcase• 12 years ago

When you mentioned discourse serving 19k req/sec, does it hit the ruby stack at all? If no, serving 19k req/sec of cached HTML doesn't seem that impressive. What am I missing?

Researching a new form of HTTP caching optimization

Related Stories