A self-killing web site requested by a customer (2011)

https://rachelbythebay.com/w/2011/06/28/sp/

194 65 6 years ago

I found it interesting because such a simple task (requiring at least a number of on-line servers before the load-balancer starts serving requests), required a custom binary controlled the webserver and had to cross-monitor each server.

For example with HAProxy (my favorite load-balancer and HTTP "router") this can be easily achieved by using `nbsrv`, creating an ACL and only routing requests to the backend based on that ACL. Based on the documentation bellow:

* http://cbonte.github.io/haproxy-dconv/2.1/configuration.html...

One can write this:

    ~~~~
    frontend www
    mode http
    acl site_alive nbsrv(dynamic) gt 2
    use_backend dynamic if site_alive
    ~~~~

[This article was linked from the original article described in (https://news.ycombinator.com/item?id=23099347).]

jefftk• 6 years ago

Things are a lot easier these days, mostly because there's been more time for people to have problems and code up good generalizable solutions

ciprian_craciun4• 6 years ago

I just wanted to mention that I didn't fault the original author for the proposed solution. It was 2011, they seemed to use some commercial load-balancer, and the customer didn't seem to want to actually solve the solution.

I actually found the original solution interesting, and I also proposed an alternative achievable with HAProxy as an load-balancer.

wrkronmiller• 6 years ago

That config looks like it kills the site if there are 2 or fewer back-end servers (similar to the setup in the article).

Wouldn't it make more sense simply to limit the number of connections per back-end server?

ciprian_craciun4• 6 years ago

Indeed if I were to configure the load-balancer I would limit both the number of concurrent requests and the number of queued requests. (In fact this is what I do in my HAProxy deployments.)

However I assume that the customer (the original author was referring to) didn't want to apply any limitations. (Why is beyond my reasoning...)

klausjensen• 6 years ago

Lovely hack and an example of how thinking outside the box can create solutions, that are order of magnitude cheaper than the "obvious" solution.

MaxBarraclough5• 6 years ago

The load now rebalanced to four remaining machines is just far too big, and they all die as a result.

Perhaps I'm missing something terribly obvious here, but why would that happen?

I can understand requests being dropped and processing-times worsening, but a full system-wide crash?

edit My bad, I'd missed this in the article:

they could have rewritten their web site code so it didn't send the machines into a many-GB-deep swap fest. They could have done that without getting any hosting people involved. They didn't, and so now I have a story

sneak• 6 years ago

System latency is not linear with load. At a certain threshold it starts swapping way beyond the capacity of the disk i/o and basically nothing gets done.

I would assume that if you left it alone for an hour or so it might eventually unfuck itself, but for production purposes, that counts as dead, especially when you can’t even ssh into it because the memory allocations for your ssh session are also in that gigantic queue for disk bandwidth via swap.

MaxBarraclough5• 6 years ago

Why doesn't the server just drop incoming requests until it's able to handle them again? As I mentioned in another comment, this is what routers do.

edit I hadn't noticed this at the bottom of the article:

they could have rewritten their web site code so it didn't send the machines into a many-GB-deep swap fest. They could have done that without getting any hosting people involved. They didn't

brabel2• 6 years ago

This will easily happen with systems that do not have any kind of upper limits on what they _try_ to do. For example: not limiting your thread pool size so that your server might try to create more threads than the system has memory for, dying due to lack of memory. Similar things can happen even if you're using async IO as you still need memory for all those stacks being run asynchronously.

MaxBarraclough5• 6 years ago

So there are two ways a server can be engineered to handle overload:

1. A throttling scheme of some sort

2. Ignore the possibility, and just crash disastrously if/when it occurs

Is that right? If so, surely that's what we should be talking about here, no? (I see ninkendo mentioned it in their comment too.)

Am I right in thinking the whole problem that the blog-post discusses - avoiding a small pool of servers getting overloaded and blowing up - follows from poorly engineered servers that go with Option 2?

Edit This is what I get for reading the article too quickly. She addresses this quite explicitly:

they could have rewritten their web site code so it didn't send the machines into a many-GB-deep swap fest. They could have done that without getting any hosting people involved. They didn't

Beltiras4• 6 years ago

It's like thrashing for memory. Once you can't keep up with requests it's extra strain which inevitably will bring down the system.

MaxBarraclough5• 6 years ago

With respect, that's a metaphor, not an answer.

Why shouldn't the system be able to just drop incoming requests until it's able to handle them? That's what routers do.

aasasd• 6 years ago

In the article, swap is mentioned. Apparently either processing each request requires some amount of memory, or new processes are fired up for concurrent requests (the Apache model). So, what I've seen happen in similar circumstances is that once swapping starts, some of the processes run 10x slower, while new load keeps coming in and piling up, running the server even deeper into the swap. Soon you have a thousand processes, load-average >100, and you can't run anything in the terminal.

As mentioned in other comments, a proper defense against this would be to have a front-line system that anticipates such a situation and actively manages it: puts outstanding requests in a queue (throttling the load) and then errors out on new requests if there's still too many of them coming in.

nicbou3• 6 years ago

This is called a cascading failure. It's also a problem with the electric grid, and more terrifyingly with global finance.

https://en.wikipedia.org/wiki/Cascading_failure

Beltiras4• 6 years ago

I'd imagine this is the next couple of years in finance due to COVID. Shocks and reverberations for a long long while.

nicbou3• 6 years ago

Travel, immigration, entertainment, restaurants etc are severely affected. A lot of people have lost their income, and spend less.

There is also some feat that this will exacerbate problems with an already stressed financial system. Banks are massively increasing their cash reserves, and many organisations have a grim outlook on the upcoming months.

joncrane• 6 years ago

This is a really interesting failure pattern and has led to the development of so-called "circuit breakers" to avoid the cascade.

(Many of the recent (last ~5 years) AWS failures have been due to cascading failures and it's neat to read their postmortems).

DaiPlusPlus2• 6 years ago

The cascading failure of NYC’s electricity grid in 1965 was the background of episode 1 of Connections.

nitrogen2• 6 years ago

That show's first season gives an excellent picture of how interconnected technology is and how ideas spread and combine. The surprising connections are analogous to the cascading failures we see in other systems. Well worth a watch.

londons_explore• 6 years ago

A better solution would be to simply configure the loadbalancer to have a maximum number of requests per second per endpoint and then to drop any requests over that.

An even better loadbalancer will poll a load endpoint, representing CPU load, queue length, percentage of time GC'ing, or some similar metric, and scale back requests as that metric gets too high.

MichaelApproved• 6 years ago

Maybe their “load balancer” was just a simple round robin service which wouldn’t have those abilities.

ninkendo2• 6 years ago

Wait, why are the servers “crashing” when under too much load in the first place?

If there’s some sort of natural limit to how many simultaneous connections they can handle, why can’t they just return some 4xx error code for connections beyond that? (And have clients implement an exponential back off?)

Or if that’s too difficult, the load balancer could keep track of some maximum number of connections (or even requests per second) each backend is capable of, and throttle (again with some 4xx error code) when the limit has been reached by all backends? This is pretty basic functionality for load balancers.

You’re going to need actual congestion control anyway, when the number of client connections is unbounded like this. Even when your servers aren’t crashing, what if the client apps whose clicks you’re tracking becomes suddenly more popular and you can’t handle the load even with all of your servers up?

laurentdc2• 6 years ago

If there’s some sort of natural limit to how many simultaneous connections they can handle

Yup, it's pretty much a DDoS. It's simple to test. Spin up Wordpress + Woocommerce (a common ecommerce stack) in a Digitalocean 1 GB droplet.

Now ab -n 1000 -c 30 the home page, that's 30 concurrent clients

Watch MySQL die, Apache get killed because out of memory, and...

root@wootest01:~# reboot

-bash: fork: Cannot allocate memory

ninkendo2• 6 years ago

So, maybe fix that?

Why is Apache continuing to fork new workers ad infinitum? That’s a denial of service attack waiting to happen, and the answer surely isn’t “oh let’s just automate the rebooting”...

Edit: you’ve edited your comment to say it’s a DoS as well, so looks like we’re on the same page here, the software is garbage.

Neil44• 6 years ago

Exactly this, limit your apache processes to match the ram you have available so the machine returns 4xx or 5xx to excess requests but otherwise remains responsive and normal. Problem solved. The options are right there in the conf.

Start with a super low limit then slowly increase while watching reqs/sec. When reqs/sec stops increasing or you run out of ram then there’s your sweet spot.

jonhohle• 6 years ago

This. Least-conns load balancing with max-conns per host allows this type of failure to be handled as gracefully as possible without human intervention or apps needing to share state. It amazes me that these aren’t defaults on very popular load balancing platforms in 2020.

mgkimsal• 6 years ago

was it just a cost thing that would prevent people from just adding another server in to the mix? given that 4 was the magic number, add another server or two to add buffer to time between servers dying and 'it all breaks'? I'm realizing the cost factor may have been it, depending on size/location/etc. - would there be any other reason?

Beltiras4• 6 years ago

Might have been tried and not worked. Might have been a limitation in scaling (f.ex. only being able to do master-slave replication and not being able to add more master nodes). Remember: it's bad software to begin with.

jasonlotito• 6 years ago

Not necessarily the reason here, but to answer your question: "would there be any other reason?"

Another reason is backend constraints. Adding another server might simply cause more pressure on backend systems like a database. So they can't simply add more servers, as then the additional load on the database might cause that to crash, and result in even longer downtime.

Random_ernest2• 6 years ago

I am not a webdev, but isn't that a task for the loadbalancer in the first place?

ciprian_craciun4• 6 years ago

Unfortunately the load-balancer is not a magic-bullet curing every issue a system has. A load-balancer can be configured to do lots of things, like for example:

* limit the number of concurrent requests, and drop the others;

* limit the number of concurrent requests, but queue the others (with a timeout);

* distribute all requests uniformly (randomly or in round-robin fashion) to all backends;

* (any combination of the above);

However if the "customer" asks you to not drop or queue requests, then there is nothing the load-balancer can actually do...

Random_ernest2• 6 years ago

I took it from the article that dropping requests was permitted (since that happens when all servers go down). So my assumption is still that a better solution is that the load balancer allows only a specific number of requests per server and rejects or caches the rest. I would even argue that requests being rejected is more understandable for the user than the website simply not being there for a certain time.

jrockway2• 6 years ago

There is actually quite a bit of complexity with load balancing, but the good news is that a lot of the complexity is understood and is configurable on the load balancer.

I think what Rachel calls a "suicide pact" is now commonly called a circuit breaker. After a certain number of requests fail, the load balancer simply removes all the backends for a certain period of time, and causes all requests to immediately fail. This attempts to mitigate the cascading failure by simply isolating the backend from the frontend for a period of time. If you have something like a "stateless" web-app that shares a database with the other replicas, and the database stops working, this is exactly what you want. No replica will be able to handle the request, so don't send it to any replica.

Another option to look into is the balancer's "panic threshold". Normally your load balancer will see which backends are healthy, and only route requests to those. That is what the load balancer in the article did, and the effect was that it overloaded the other backends to the point of failure (and this is a somewhat common failure mode). With a panic threshold set, when that many backends become unhealthy, the balancer stops using health as a routing criterion. It will knowingly send some requests to an unhealthy backend. This means that the healthy backends will receive traffic load that they can handle, so at least (healthy/total)% of requests will be handled successfully (instead of causing a cascading failure).

Finally, other posts mention a common case like running ab against apache/mysql/php on a small machine. The OOM eventually kicks in and starts killing things. Luckily, people are also more careful on that front now. Envoy, for example, has the overload manager, so you can configure exactly how much memory you are going to use, and what happens when you get close to the limits. For my personal site, I use 64M of RAM for Envoy, and when it gets to 99% of that, it just stops accepting new connections. This sucks, of course, but it's better than getting OOM killed entirely. (A real website would probably want to give it more than 64M of RAM, but with my benchmarking I can't get anywhere close with 8000 requests/second going through it... and I'm never going to see that kind of load.)

I guess the TL;DR is that in 2011 it sounded scary to have a "suicide pact" but now it's normal. Sometimes you've got to kill yourself to save others. If you're a web app, that is.

rrmoelker• 6 years ago

While not actively developing infrastructure myself, I've always like the concept presented in the Hystrix package: https://github.com/Netflix/Hystrix

Even though it seems it is no longer maintained, the circuit breakers, fail over modes and all that are well documented.

And I don't know why Hystrix hasn't been adopted by a wide audience yet. It seems like a necessity in the micro service landscape.

jrockway2• 6 years ago

I have always preferred the library approach myself, but it seems like people are converging on "sidecar" proxies to connect up their microservices. Istio and Linkerd are the big ones. Istio uses Envoy which you can use without a whole "service mesh" to add things like circuit breaking, load balancing, rate limiting, etc.

rjkennedy98• 6 years ago

HA is definitely super tricky. Not many products do it well. One of the last NoSQL databases I used for instance was quicker to restart than for failover to be detected so DBAs would just restart the cluster instead of waiting for failover to happen during an upgrade.

hangonhn• 6 years ago

Oh boy! I had a similar cascading failure situation once with a Nagios "cluster" I inherited. The previous engineer distributed the work between a master and 3 slave nodes with a backup mechanism such that if any of the slaves died, the load would go to the master. This was fine when he first created it but as more slaves were added, the master was running at capacity just dealing with the incoming data. So each each additional slave node, the probability of one of them failing and sending its load to overwhelm the master increased. Sometimes a poorly designed distributed system is worse than a single big server.

I ended up leveraging Consul to do leadership election (only for the alerting bit) and monitor the health of all the nodes in the cluster. If one of them failed, the load was redistributed equally among the remaining nodes.

A self-killing web site requested by a customer (2011)

Related Stories