Chaos Monkey released into the wild

http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html

by timf • 14 years ago

235 36 14 years ago

techblog.netflix.com

coob• 14 years ago

Very interesting. However, this won't catch bugs that don't cause instance outages. I suppose the next step to is to somehow simulate misconfigurations/overloaded services/unexpected errors (I have no idea how this would work).

Zombieball• 14 years ago

Netflix actually has a whole Simian Army (http://techblog.netflix.com/2011/07/netflix-simian-army.html) which includes in addition to Chaos Monkey: Latency Monkey, Chaos Gorrila, Conformity Monkey, etc.

jedberg4• 14 years ago

We actually have those too. Right now in fact we're doing a series of latency simulations.

oonny• 14 years ago

Jeff Atwood had a good piece on this: http://www.codinghorror.com/blog/2011/04/working-with-the-ch...

Overall it's brilliant from an infrastructure standpoint but it's up to the developers to make your code monkey proof!

sabat• 14 years ago

It's up to the developers and you, the operations/devops people, to work together to make it chaos-monkey-proof. No engineering team is an island.

InclinedPlane• 14 years ago

This reminds me of something I read the other day¹, the idea that complex fault tolerant systems tend to end up running with faults as a matter of course (and sheer probability). This elevates that notion to another level, get rid of the idea of operating without faults and maintain a low-level of faultiness artificially to ensure that resiliency to faults in the system is always working.

¹ http://www.johndcook.com/blog/2012/07/13/fault-tolerant-syst...

ryusage• 14 years ago

Reminds me of biology. It's always intrigued me that we often want to make computer systems as "good" as humans, and yet our acceptance of human faults is far higher than what we accept from computers. Any small part of a body may have some problems, but the body as a whole continues to function well. I wonder if this kind of design will become a trend?

delinka• 14 years ago

We tolerate fewer faults from our computers because the underlying system is not complex enough to prevent those faults from affecting us in large ways. As you say, a small failing in the human body doesn't render the entire organism inert and useless; no, we can adapt our behavior during healing and move on. When a small fault occurs in your computer, it tends to have disastrous effects.

We will tolerate faults in our computer systems when then faults don't cost us our homework, our jobs, or our lives. And complexity is the only way to making that happen.

I, for one, welcome our new chaos-wielding overlord!

wanderr• 14 years ago

This is certainly a great idea in theory, but I wonder if it has caused them to build things to be a little too fault tolerant, by assuming there are faults when maybe things are just slow. On Xbox, my recently watched fails to appear about 30% of the time even though it appears just fine on the site. The only thing I can think is that it must be taking just a little bit too long to return, so they assume failure and move on.

samstave2• 14 years ago

The reason why this is critical for Netflix is that all the infrastructure is provided as a service and this completely out of their control.

By creating faults in the Netflix service which they can build around to stay optimally available, they completely or nearly completely eliminate the risk of being affected by faults in AWS services.

Now, clearly this has not been 100% true yet because some major outages have effected Netflix - though that's when the entire service fails, as opposed to single point of failure types of faults, which chaos monkey is really designed to accomodate.

(disclaimer: this is my outside perception based on purely interest in how Netflix has built themselves. I could be incorrect)

jedberg4• 14 years ago

You are mostly correct. The infrastructure isn't completely out of our control, but it is indeed provided by a 3rd party.

By creating faults it means we know we can handle those types of outages.

As we experience new types of faults, we build tools and systems to make us resilient to those types of faults, and where possible similar but yet unexperienced faults.

Randgalt• 14 years ago

FYI - we have a Latency Monkey as well

joelcox• 14 years ago

Webpulp TV recently did an interview with Jeremy Edburg from Netflix in which they discussed Chaos Monkey, albeit shortly.

http://webpulp.tv/episodes/how-netflix-one-of-the-largest-ec...

Zenst4• 14 years ago

Still think playing kerplunk with processes/systems whilst good is akin to pissing on your server and then seeing how quick you can repair it.

I personaly like to run stuff on a CPU/network crippled test setup as that can expose cracks normal testing fails to highlight and in a way that is indicative of what can actualy happen under some load spikes.

mikeash2• 14 years ago

If your goal is for your server to be urine-proof, then peeing on it from time to time to make sure would be a good idea.

Zenst4• 14 years ago

sadly my analogy failed most peoples IQ :facepalm:

technomancy• 14 years ago

Having the code behind the chaos monkey is not nearly as valuable as having the guts to run it in the first place.

jedberg4• 14 years ago

If this were reddit, I'd link to that meme about doing testing in production.

Since this is HN, I'll instead say that if you don't have the guts, you're only lulling yourself into a false sense of security. AWS or not, your systems will fail.

By running the monkey, at least you can make it fail on a schedule that is convenient to you, instead of happening when you're drunk on a Saturday night (or whatever your vice of choice might be).

freehunter• 14 years ago

At my work (large corporate office), we have random node outages. It's not quite as in depth as chaos monkey, but it goes towards the same purpose. Just pull the plug on the server. More than once, a random node outage has caught a novice developer making static links to nodes through the load balancer. We also have random pen-tests designed to DoS or otherwise disable services around the network. Controlled destruction of your infrastructure is the quickest way to highlight any faults.

But remember: what's the difference between hacking and pentesting? Permission.

mkopinsky• 14 years ago

Without chaos monkey, instances will not randomly die anywhere near as frequently as with. (Otherwise, why run it at all?)

Increasing the amount of failures in order to increase the percentage of nice (not 3am) failures takes guts.

Zenst4• 14 years ago

yes it's like knowing a mad axeman and then letting said person go wild in yoru server room.

waivej• 14 years ago

This is so cool, and I'm wondering if I can do something similar. It reminds me of "bug seeding" where you purposely insert bugs into your product and count how many are found through testing. (Of course, you track them so you can take them out later.)

TimJRobinson• 14 years ago

That's a really good idea, do you know if there are any tools out there for doing this? I'm thinking for something basic you could make it change random variables or numbers in the code base and run the tests after each change and report if they failed.

tomblomfield• 14 years ago

There's Heckle for Ruby

http://ruby.sadi.st/Heckle.html

herge• 14 years ago

Fuzz testing involves an automated process commenting out each line of your code one-by-one and running your unit tests to see what lines are actually tested.

photorized• 14 years ago

AWS is the chaos monkey.

Their outages are becoming more and more random.

acdha• 14 years ago

> Their outages are becoming more and more random

… or more loudly reported. Very few of the reported outages affect people who follow Amazon's guidelines for reliability - even the last one had most of the impact on cheapskates who didn't pay for redundancy.

jrydberg• 14 years ago

Usling the monkey, have you found that some design patterns are better than others? For example, queues vs RPC?

Chaos Monkey released into the wild

Related Stories