Getting the most out of HAProxy

https://www.twilio.com/engineering/2013/10/16/haproxy

by kevinburke • 13 years ago

66 16 13 years ago

falcolas2• 13 years ago

Want to know what else is cool about haproxy? The stats unix socket. With it, you can query the existing configuration at whim, and with haproxy 1.4 or greater, you can configure haproxy on the fly.

    show stat -1 4\n
    disable server xyz\n

What's not so cool? Disabling a server does nothing to kill off existing connections to that server, and idle timeouts can cause problems if your processes are slow to respond.

With care, though, it's a great component of a HA solution, in addition to a being a great load balancer.

nasalgoat• 13 years ago

If you're running multiple haproxy processes, the unix socket commands only work on the process you happen to randomly get when you use the socket.

I've asked about plans to fix that and it's not on the current roadmap for 1.5.

I need multiple processes to allow SSL offloading.

falcolas2• 13 years ago

Perhaps start each up with a separate configuration file for each instance? Or run as a cluster, so changes to one are propagated to all (untested)?

    haproxy -f /path/to/alternate/cfg

nl2• 13 years ago

As we convert more and more Twilio tools to a service-oriented architecture, these logs have been invaluable at debugging failures

I'd love to know more about how Twilio does SOA (and no, the linked document doesn't expand on it).

Do they use an ESB or do they rely on individual services connecting directly to each other?

I'm 100% convinced of the ideas behind a service-oriented architecture. I'm less convinced about the need for an ESB, but I'm happy to be talked around.

Experiences/Opinions/War Stories eagerly sought.

kevinburke• 13 years ago

Internally each FE runs HAProxy to round robin between all of the shared backends, no ESB involved. Hoping to write another blog post about this soon.

Agree that the link is kind of not-good, this might help a little... http://www.slideshare.net/twilio/highavailability-infrastruc...

I wish I could say more about our infrastructure and experience with SOA, but I'm a little limited in what I can share. We are hiring though :)

nl2• 13 years ago

Thanks for the response & link.

What's a "FE"?

jtokoph• 13 years ago

Another cool HAProxy option is setting up backup/failover backends. So if all of your real backends die or go offline, haproxy can serve a friendly maintenance/downtime page from the failover backend.

zerop• 13 years ago

where haproxy scores above ngingx..I am curious to know, as we use nginx....

zh• 13 years ago

NginX is very good WEB SERVER - so it shines in serving content - web pages, caching images etc. But in proxy mode, the load-balancing between backends is pretty basic. HAProxy is born to be very good LOAD BALANCER - you can have different balancing algorithms, not only round-robin. HAProxy have also TCP mode, so you can balance your MySQL backends, for example. Also for me, personally, HAProxy stats rock.

ZoFreX• 13 years ago

Thanks for the explanation, it's very enlightening to someone (me) who's never used haproxy. For front-end load balancing, I've been using Varnish. Is HAProxy also better at http load balancing than Varnish as well as nginx, do you know?

vidarh• 13 years ago

Regarding your point about MySQL:

We used haproxy to do close to zero downtime migration of our postgres servers (and other stuff) from one data centre to another recently when we moved out of our old data centre: Set up a slave in the new dc, set up haproxy with the slave as backup, point all the clients to haproxy. Shut down the master, and as soon as the master stopped responding, haproxy would shove clients to the slave in the new data centre instead. Then touch the trigger file to make the slave recover and switch to master mode.

It meant we were in read-only mode because requests where going to the slaves for about 2-3 seconds (and unlike MySQL Postgres does not allow writes to a slave until it's been promoted) before the slave started handling updates - other than that everything just chugged along.

dantheta• 13 years ago

I was also pleased to find that in version 1.4 a MySQL login health check was added "option mysql-check".

gopalv• 13 years ago

More fun HAProxy bits include pinning a user persistently to one machine (which fails over when that machine fails a health-check)

Particularly useful when you've got a web+mc combo in one machine which can get the most out of such a setup - http://notmysock.org/blog/hacks/haproxy-user-repinning

petercooper• 13 years ago

I don't do anything heavy enough to need load balancing, but I've always found HAProxy to be ideal for situations where I'm limited by IPs but I want to run a variety of HTTP oriented services on the same machine.

It's so easy to filter by hostname and send requests off to different internal daemons and HAProxy is solid as a rock unlike other mechanisms I've tried.

Getting the most out of HAProxy

Related Stories