In search of performance – how we shaved 200ms off every POST request

https://gocardless.com/blog/in-search-of-performance-how-we-shaved-200ms-off-every-post-request/

by Sinjo • 11 years ago

87 15 11 years ago

dap2• 11 years ago

This is a great write-up of a tough technical problem, and I definitely agree with their conclusion that "it's worth taking the time to understand your stack." I especially liked that it wasn't preachy. It just explained their problem and how they solved it. (It seems a lot of write-ups end up offering very specific technical advice that grossly overgeneralizes from their experience.)

My only question is how they got to haproxy as the root cause so quickly. In my experience, comparing what's different between production and staging is a long shot because there's so much that's different. Obviously workload can matter a lot, and so can uptime and time since upgrade. So I'm curious if haproxy was the first thing they saw that was different or if they just didn't write about the dead ends.

rocky1138• 11 years ago

It might not be all that different if they're using Docker.

edwhitesell2• 11 years ago

With proper tracking it is not difficult to see different versions of packages installed. Checking run books/OPs logs would also lead to a quick understanding that Staging was updated and Production wasn't. Either of those two things would make it relatively easy to find the discrepancy in versions. Then, as they noted, a review of changelogs would have led to the patch they found.

If you don't have processes and documentation to quickly point out those kinds of differences between environments, you're doing something wrong.

dap2• 11 years ago

It's not that you can't find _some_ differences that way. As others have you pointed out, there should be little difference between the software deployed in staging and production. But there are can be differences in performance and hardware configuration (unless you can afford to mirror your production deployment exactly). Most unavoidably, there are problems you only hit after cumulative amounts of uptime or load[1], which is nearly impossible to simulate in a preproduction environment.

[1] For an example, see our recent outage related having seen 200M PostgreSQL transaction: https://www.joyent.com/blog/manta-postmortem-7-27-2015

Sinjo• 11 years ago

Thanks! Glad you enjoyed it.

The main thing that narrowed it down was thinking about the request path from the app server to Elasticsearch. That pointed us at the load balancers (we tried a request directly from app -> Elasticsearch in production to verify they were the problem).

You're right that HAProxy wasn't the first place we looked once we were there, though. If I remember right, we started by diffing `sysctl` output to see if Ubuntu had tweaked something between versions.

developer1• 11 years ago

To pitch in some preachy advice: with some time and effort, staging and production environments should be essentially identical. Sure, production is generally going to have more servers handling each service (more web nodes, more db/storage nodes, multiple load balancers, etc.), but the setup and deployments should not be any different.

All too often, we think of staging as nothing more than a clone of the development environment on a remote server where QA can get at it. If you run your load balancer (haproxy), web server (apache, nginx), database (mysql, postgres, mondo, elasticsearch), caching (memcached, redis) all on a single server for staging, you're eventually going to stumble on this kind of hard-to-diagnose problem.

One of the main disservices you are doing to yourself with a single-server staging is that all of your traffic is going to be travelling over localhost or, worse, over unix sockets. You're not even testing basic network latency or performance.

Staging should really be considered a first-level production. Its configuration and maintenance should be handled with the same attention to detail as production.

throwaway64908• 11 years ago

"We use Ruby."

Ono-Sendai• 11 years ago

That was their first problem.

aikah• 11 years ago

That was their first problem.

Where do you think you are, on reddit?

markbnj• 11 years ago

> To make things worse, Net::HTTP doesn't set TCP_NODELAY on the TCP socket it opens, so it waits for acknowledgement of the first packet before sending the second. This behaviour is a consequence of Nagle's algorithm.

I think rather than a consequence of Nagle's algorithm it is the situation that the algorithm is intended to optimize when an app generates many small packets.

geofft2• 11 years ago

What are the cases where you want Nagle's algorithm enabled? It seems like the primary use case is telnet and telnet-like services (like SSH), but even there, Mosh does much better at balancing performance against congestion. For non-interactive protocols, do you ever want Nagle's algorithm?

toast02• 11 years ago

Nagle's algorithm would actually help in this case, if there were a bit more data to send, since ruby is passing data bits at a time, buffering it until you get a full packet would be nice. It's just the first bit is sent right away, and the second bit doesn't fill the packet, and there is no third bit. If you were pipelining reuests, the algorithm would be helpful.

geofft2• 11 years ago

But the client code already has information about whether it's planning to write some more or not, so you could just depend on that, instead of using heuristics about the peer's behavior. Even regular stdio-style buffering plus an explicit flush would help more than Nagle's algorithm.

In search of performance – how we shaved 200ms off every POST request

Related Stories