Misconfigured Circuit Breakers

https://engineering.shopify.com/blogs/engineering/circuit-breaker-misconfigured

by rbanffy • 6 years ago

178 39 6 years ago

engineering.shopify.com

klodolph2• 6 years ago

Consider randomizing error_timeout, to avoid a possible case where all clients retry at the same time.

smoyer• 6 years ago

We use this quite a bit but we actually have a constant timeout with random "jitter" added to it. You almost never want the timeout completely randomized because there's a chance you'll get a zero.

klodolph2• 6 years ago

The concept of “completely randomizing” something isn’t really well-defined, it’s more of a casual phrase that people use without thinking about it. If you have a constant timeout with random jitter added to it, you’ve randomized it. It is now random.

If you think about it, a “perfectly random” die roll on an ordinary die will never be 0. So, whether a value is random has no relationship with whether that value can be 0.

ed_balls• 6 years ago

you don't want it random. You want to add a splay .e.g

# add a random value (from 0 to X) to the timeout # This can be used to prevent the thundering herd problem splay = "5s"

speedgoose• 6 years ago

After reading your comment, I did consider to implement that in my codebase and then I remembered that I have so few servers that it would not matter. Even if my company has to scale up to 99% of the market.

londons_explore• 6 years ago

I suggest you do it anyway. It's a one line code or config change, yet if you do suffer the "stampeding herd" problem, it's hard to diagnose and hard to fix in the middle of another production outage.

derefr6• 6 years ago

This might be my bias as someone who mostly writes in actor-based concurrent languages, but: is there a reason to test the service’s back-to-life-ness by passing a real request through, rather than just having every open circuit trigger the creation of a periodic background “/health”-endpoint (or equivalent) poller actor for that backend within your service? To do the check within the client request lifecycle seems like it would needlessly increase client-request 99th-percentile latency for no real benefit, other than the single request which doesn’t time out, closes the circuit, and so gets served from the service.

ETA: you could even—presuming your system does solely idempotent stuff—take the exact request whose timeout caused the circuit to open, and pass that request (the particular query for SQL; the particular URI + POST/PUT data for REST) to the poller-actor you’re spawning to use as its health-check. Once that request doesn’t time out any more, you know you’re back online.

I’d recommend a real separate /health endpoint for the service you’re trying to contact, though, because 1. most applications do non-idempotent things, and 2. some queries time out because they touch edge-case bad/unscalable code paths, not because the remote-service-as-a-whole is down. If your `SELECT * FROM expensive_generated_report_view` SQL query times out, you really should not be reusing that query as a health-check against your RDBMS. (But nor should you just be doing `SELECT 1`!)

dtech5• 6 years ago

A /health endpoint or similar mock might be available and responding fast, while a real request might fail or time out.

derefr6• 6 years ago

Well, that’s not a “true” /health endpoint, then. A service’s /health endpoint should run through its regular non-trivial code paths and depend for its success on all of the dependent services normal requests to that service depend on. (Probably you’ll need to write it yourself, rather than using one supplied by your application framework.)

For example, if you have a CRUD app fronting a database, your CRUD app’s /health endpoint should attempt to make a simple database query. If you have an ETL daemon that pulls from a third-party API and pushes to a message queue, it should probe both the readiness of the API and the message-queue before reporting its own readiness to work. (Of course, it is exactly in the case where the service has its own circuit-breaking logic with back-up paths “around” these dependencies, that it gets to say it’s healthy when its dependencies aren’t.)

A test of the /health endpoint is, after all, what determines if a service is considered viable for routing by load-balancers; and, vice-versa, if the service is considered to be “stuck” and in need of a restart by platform auto-scaling logic. As such, your /health endpoints really should be configured in a way where they generate false positives—reporting being unhealthy when they’re really healthy—rather than false negatives.

If you’ve got a pool of instances, better to have them paranoid of their own readiness in a way where your system will be constantly draining traffic from + restarting them, than to have them lackadaisical about their readiness in a way where they’re receiving traffic they can’t handle.

TheRealPomax• 6 years ago

you don't mock it, you implement a real one that acts as a remote microservice. Sometimes it's a /health route on the same site, sometimes it's a status.example.com, etc. The only thing it should do is give you a static errors-or-not file that gets regenerated at some reasonable interval by an independently running process, off of all known dependencies that you control.

If the health route says everything's up and running, then you can try a normal request, and if that fails, you back off anew (and you log an error entry that can be used to update the health endpoint's data generator, because something, somewhere, is misreporting)

lostcolony• 6 years ago

So some of the responses to you seem to be missing the point. I agree with you, and with a health check being the wrong thing to circuit break on.

Not matter how complex a health endpoint is, the circuit breaker is more so. The health endpoint might make a legit request, checking all downstream services, etc etc, to confirm that everything is working...but the actual traffic is many different requests, with different code flows.

As an example, imagine service Bs health check actually makes a request. Great. Only, service B is misconfigured to talk to staging, whereas the upstream, service A, is in prod. The health check may always pass; after all, the staging request's data all gets resolved and handled appropriately. The upstream one, however, leads to constant failure, as that data doesn't exist.

Health check passes 100% of the time. Actual upstream calls pass 0% of the time. Don't have your upstream calls circuit break on health checks; have them circuit break on actual responses.

(And yes, you should have network isolation between staging and prod so that particular example can't happen. It is intentionally a contrived example)

myself248• 6 years ago

This is apparently a software function known as a circuit breaker, and has nothing to do with electrical current flow.

Fuckin' HN headlines, I swear.

bluedays• 6 years ago

yeah I got a little excited about how I might fix the circuit breaker in my house.

segfaultbuserr• 6 years ago

how I might fix the circuit breaker in my house.

Don't be so disappointed, here's my tip for you. In electrical installations, every subcircuit should have its own RCD/GFCI circuit breakers installed, rather than relying a RCI/GFCI breaker on the higher hierarchy. Otherwise the circuit breakers are misconfigurated - failing to do so would make troubleshooting leakage current (read "tripping breakers") extremely difficult.

But when it happens, there's a solution. I once fixed one at my home that always trips spontaneously, the secret is using a leakage current clamp meter. A RCD/GFCI circuit breaker trips when the leakage current is greater than its threshold to protect equipment and people. With a leakage clamp meter, it allows you to monitor the leakage current in real-time rather than hopelessly trying in the dark. You just clamp the meter across BOTH the live and neutral wire on the main breaker (if both wires carry the same current in the opposite directions, i.e. no leakage, net magnetic flux inside the sensor will be zero, so the meter reads 0 mA, and vice versa), turn off every subcircuit, and starts switching on different subcircuits one-by-one. If you see the leakage current substantially increases after you switched on a specific subcircuit, repeat the process by disconnecting all appliances from that subcircuit and connecting them one-by-one while observing the meter. Finally you'll find the smoking gun (if there's still significant leakage when nothing is connected, it means the wiring is at fault), mine was a light fixture with damaged insulation.

A leakage current clamp costs $200-$300, Fluke's meter will cost you $600. However, you can get a good enough, low-accuracy leakage clamp from China on AliExpress for $50-$100, just make sure the product description mentions "leakage" (the laws of physics are the same for all current clamp meters, but most are not designed to sense sub-10 mA leakage current).

In my experience, only a handful of electricians working with home wirings are aware of this diagnostic technique. I learned it from an application note of an industrial system.

spelunker• 6 years ago

Now let me tell you about bulkheads...

dtech5• 6 years ago

It's called an analogy. It functions similarly to an electrical circuit breaker because it disables systems before things go fully haywire/bad, it helps devs understand the concept by using familiar things

ew6082• 6 years ago

It's bad naming practice and only adds confusion to search.

hinkley4• 6 years ago

Although it behaves a lot more like a GFCI than a circuit breaker, but apparently nobody involved ever watched This Old House, or any other home improvement shows.

I would like these throttling tools to behave in a different way, one that more closely resembles a circuit breaker. But some rat bastard has already used that name so now I don't know what to call it.

jp_sc• 6 years ago

Seeing where the link was pointing to (Shopify), you could have expected it.

On the other hand, the same article from the Philips Engineering Blog... now that would be very confusing!

JensRex• 6 years ago

I'm an electrician. I was very confused for the first few paragraphs.

james_s_tayler• 6 years ago

It's a common pattern in software engineering to stop failures jumping from one area of a system to another. I don't get why you're mad about that. It functions analogous to the very thing it's named after.

hinkley4• 6 years ago

Better to remain silent and be thought a fool...

This term has been around for quite a while. I don't think it's a very apt term, but it's the one that was chosen, so whateryagonnado.

Welcome to the 10,000.

hhsuey• 6 years ago

Utilization, the percentage of a worker’s maximum available working capacity, increases indefinitely as the request queue builds up, resulting in a utilization graph like this.

Keyword indefinitely. Isn't this assuming the service worker doesn't have a timeout itself?

damianpolan• 6 years ago

Even with a timeout, the time spent waiting on the timeout is wasted utilization. Therefore, if the request rate stays the same, and each request is wasting enough utilization, the utilization required to process the request will be higher than the rate it is worked off.

freshbob• 6 years ago

The author seems to think completely in terms of "wasted utilization" when it comes to timeouts. I think they are missing the point of the timeouts and the retry logic to begin with. The effort by the circuit breaker isn't wasted, because it is exactly trying to establish whether a resource is responding or not taking into account occasional network hiccups. If every effort past the initial timeouts was wasted, then why implement this logic to begin with? I agree with derefr (https://news.ycombinator.com/item?id=22546241) in the sense that it seems illogical to increase latency for users simply to check for availability of a timed-out resource.

IMHO the worst-case assumption of all service instances failing simultaneously leads the author astray in their quest to reduce "wasted utilization".

Pretend the network switch rebooted and all services were unavailable for a short period of time, but your website is in high demand, so the error threshold of three errors per resource was quickly reached. Let's pretend the network switch needed 5 seconds to reboot, so 42 resources each failing 3 times in that time equals 126 requests/5 seconds, 25.2 requests/second. Now, instead of quickly recovering from that state after two seconds, the author advises to instead wait 30 seconds, so that's 756 requests---because your site is so popular---before the first service is retried. Then an additional 41 requests (~1.67 seconds) until all resources are marked available again. So now you made about one thousand people unhappy in case it's their browsing session that's constantly lost. Unless of course your were too optimistic when setting the half_open_resource_timeout, because then your services might be blocked for multiples of error_timeouts, e.g. minutes with a high error_timeout value of 30 seconds. That's a lot more than a thousand people unable to log in.

IMHO setting the half_open_resource_timeout way lower than the regular service_timeout value will just risk the services _never_ becoming available again after an internal network outage in your data center. That seems like a recipe for disaster.

ofek• 6 years ago

Good read!

This sounds similar to the work I did [1] on the Datadog Agent, especially regarding the concept of each resource having its own circuit breaker.

My implementation is a bit different though, instead based on exponential decay like BGP Route Flap Damping [2]. It matches our use case a bit better and is easier to reason about.

[1] https://github.com/DataDog/datadog-agent/pull/1458

[2] https://tools.ietf.org/html/rfc2439#section-2.3

Misconfigured Circuit Breakers

Related Stories