Netflix: Lessons We’ve Learned Using AWS

http://techblog.netflix.com/2010/12/5-lessons-weve-learned-using-aws.html

by jeffmiller • 16 years ago

256 62 16 years ago

techblog.netflix.com

wccrawford• 16 years ago

I want a Chaos Monkey, too!

Actually, that was my first reaction, but after thinking for a moment, that isn't really a reliable way to test. If you make changes to something, you don't know for sure if the chaos monkey hit while you were testing a certain thing or not. Proper unit tests would seem to be a lot more useful.

qjz• 16 years ago

If Chaos Monkey hits during a test, you just run the test again. Multiple tests are a good thing, after all, and the failure might expose a weakness you wouldn't have seen otherwise.

mickeyben• 16 years ago

How would you write unit tests to test against services not responding and routing issues between your different instances ?

jdludlow2• 16 years ago

That's what mock objects are for. You mock the object that would normally call the remote system, have it artificially fail, and use that to test clients of that object.

ghshephard• 16 years ago

SQLite [1] uses a test harness to do this:

"SQLite responds gracefully to memory allocation failures and disk I/O errors. Transactions are ACID even if interrupted by system crashes or power failures. All of this is verified by the automated tests using special test harnesses which simulate system failures. "

http://www.sqlite.org/about.html

joevandyk3• 16 years ago

It's not meant to be a reliable way to test. Unit tests wouldn't help the problems that Chaos Monkey is meant to solve.

Chaos Monkey is meant to force apps to go into failure modes "in the real world". If they see anything unexpected happening after a service is killed or degraded, they can investigate.

It sure beats having a service go down unexpectedly for the first time six months ago and not have tested scenario in production.

SpikeGronim2• 16 years ago

Unit tests and fault injection tests catch a different but overlapping set of errors. Chaos Monkey is a terrific idea and I commend Netflix for implementing it early on. If you're not willing to randomly kill hosts then you're not confident that your distributed system really works.

This excellent paper[1] by James Hamilton (then MS, now AWS) recommends never doing clean shutdowns on applications - just kill them. Unless you have a lot of persistent state to manage this is a great idea. If it takes hours to migrate multiple TB off your storage host, maybe not such a good idea.

1. http://www.usenix.org/event/lisa07/tech/full_papers/hamilton...

ergo982• 16 years ago

Reading both this entry and the one that explained why they went with AWS, I'm left confused about why they ever went to AWS in the first place.

jerf2• 16 years ago

Me too. This feels like "oooh, shiny!" from high levels, not engineering needs. You shouldn't outsource your core competency, and this feels like what they are doing.

I say "feel" advisedly. I don't have inner knowledge. It's just that the explanation up to this point doesn't seem all that compelling; for what seems like rather dubious benefits it seems like they've taken on an awful lot of risks they have little ability to manage themselves.

irons• 16 years ago

This came up in the first entry. Netflix would apparently dispute your take on their core competency.

http://techblog.netflix.com/2010/12/four-reasons-we-choose-a...

The problems [Amazon] are trying to solve are incredibly difficult ones, but they aren’t specific to our business. Every successful internet company has to figure out great storage solutions, hardware failover, networking infrastructure, etc.

Their third argument for virtualized infrastructure was "We're not very good at predicting customer growth or device engagement", which strikes me as a compelling problem to solve. Having to change their approach to managing service dependencies and failures doesn't seem like too high a price to pay.

jolan• 16 years ago

I think they alluded to it in the first post. When they rolled out Xbox 360 streaming, they severely underestimated demand. It was nearly impossible to get a full quality stream for 2-3 months after the roll out.

Now with AWS, they can spin up additional instances at will.

wmf5• 16 years ago

Note that Netflix does not use AWS for streaming.

recampbell• 16 years ago

Having built a non-trivial production system, I have a love-hate relationship with AWS. I love the possibilities and potential, but I'm deeply disappointed when it fails to live up to the hype.

It's just a matter of alternatives. There's frankly no other IaaS out there that can match the features of EC2 (eg, EBS, elastic IP). These guys are unstoppable, they come out with more features every week.

wmf5• 16 years ago

Yeah, EC2 is the best cloud but it's a shame that it's still missing features that ESX 2.0 had (and Amazon has shown no interest in fixing it).

byteclub2• 16 years ago

Sounds like the initial investment into reengineering their software to handle distributed environments well turned out to be quite large, which is why it does sound a bit like they brought the troubles upon themselves. But presumably they now have 2 advantages: ability to scale horizontally, and having their software adapted to distributed environments (which means that technically they are not limited to AWS, in case problems arise with Amazon's service). Sounds like the foundation of their house is stronger now, which was probably the whole point of the exercise.

tybris• 16 years ago

In short: Because restaurants don't farm their own crops or raise their own livestock. They cook.

ergo982• 16 years ago

Analogies merely confuse (they always have that superficial "aha!" moment, yet never bear scrutiny). If it must be, however, in this case it's more synonymous with a restaurant that farmed out their kitchen and their wait staff, but they carefully crafted the menu, chose the silverware, and did all of the marketing.

Jd• 16 years ago

Basically the gist is: You need to be prepared for anything to stop working at any time.

The tone of this post indicates to me that the criticism and problems experienced by Netflix with AWS are understated, which I can understand given their position as a flagship AWS customer, etc.

SpikeGronim2• 16 years ago

You need to be prepared for halting failures and variable performance. Moving from higher end, dedicated hardware to lower end, shared hardware means an increase the failure rate and in the variance of your latencies. In exchange you get elastic scaling, you convert capital costs to operational costs (no more building datacenters), and you hand the "undifferentiated heavy lifting" to AWS. The posts aren't, IMO, understating problems but demonstrating the tradeoffs that come with AWS' core value proposition.

petercooper• 16 years ago

Including your whole AWS account, if Amazon doesn't like you for some reason, or if you attract too many attacks and disrupt their network.

aristus• 16 years ago

I'm pretty sure "session-based memory management" should be "memory-based session management", ie they kept user session state in memory.

joevandyk3• 16 years ago

Maybe they rewrote malloc to make a http call to a session provider which keeps track of the allocated memory on the machine? :)

c2• 16 years ago

Sounds like the AWS architecture caused Netflix to write better code ( read: more durable, more fault tolerant ). Less assumptions baked in the code, and it will be easier to port it to a new data center/cloud architecture if AWS doesn't meet their needs.

As Netflix continues to scale, these changes will make managing that growth much easier.

A lot of you seem to take this post as being negative against AWS architecture. I take it more as a good collection of common things that you need to watch out for in distributed environments, specifically the dangers of assumptions within your current infrastructure which may change dramatically as you scale.

sabat• 16 years ago

I'll bet other companies (e.g. Heroku, Dropbox) that use AWS/EC2 would have similar things to say.

I did have this one question, being a guy with an IT background: they expected stability? Really? I always expect host/app/system failure, and am pleasantly surprised when it doesn't happen.

Travis• 16 years ago

My take on their experience was that they had only sort of abstractly thought about stability. Then they severely underestimated the variance in performance. Before moving to Amazon, stability was something you worked at in the lower levels, e.g., hardware, power, etc. Once you move to amazon, stability cannot be controlled at a lower level, so you have to make your application code more robust.

With your own IT infrastructure, you'd logically put the focus into improving the reliability on your hardware stack, rather than just focusing on handling those issues in software.

easp2• 16 years ago

One difference is that Heroku and similar are fairly young companies, whose infrastructure was largely built in "the cloud" and for "the cloud." Whatever preconceptions the builders had about the charactaristics of the underlying environment, they started learning the reality and adapting to it relatively early in the project.

Netflix is a sizable and successful business. It's more than a decade old. Their streaming business launched only 4 years ago, more than 6 months before EC2's general availability, and almost 2 years before EBS, and it would be even longer before any of it was a reasonable choice for a company of their size, launching a new service building off their existing customer base, that had to succeed to allow the company to navigate a major transition that the company had been planning for for years.

The similarities and differences in their experiences would be interesting and probably informative, particularly if considered against the vastly different starting points.

briandoll3• 16 years ago

This reads like the 'fallacies of distributed computing' paper (http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Comput...).

While the likelihood of failure (or added latency, impacting upstream changes, etc.) is greater in large-scale distributed environments for which you do not control vs. your home-grown datacenter, those scenarios are just facts of life in distributed environments.

An awesome side effect of hosting an app in a cloud environment is that you must face up to those fallacies immediately or they'll eat you alive.

wmf5• 16 years ago

Paying large engineering costs upfront is "awesome"?

briandoll3• 16 years ago

Ignoring the fallacies of distributed computing in web applications is akin to technical debt. You're going to have to pay it eventually, and as the metaphor suggests, it's cheaper to pay it off sooner rather than later.

easp2• 16 years ago

Huh?

Netflix is a going concern, and has been for quite some time. Even the streaming business is a few years old.

It may be an up front investment since the full benefits may take some time to accrue, but it is an investment made based on a great deal more information than a typical startup has.

Further, it sounds like an investment that is providing them with an immediate and valuable benefit, they now have a foundation for ramping up their streaming business. This is going to be a big growth area, and it's is one that they are well positioned to succeed in. Or, you could flip it around, failure to secure a strong position in the streaming business will be the death of the company, and a squandering of the business equity they have been building, since their founding, to be well positioned for this transition.

And finally, if you have doubts about the value of making up-front engineering investment in order to reduce forecastable operating costs (and avoid what is probably an even larger up-front investment in datacenter buildout), well, what are you doing here?

dochtman• 16 years ago

Lesson they have not yet learned: including a HTML title tag in their Blogger templates.

jemfinch2• 16 years ago

Their "Chaos Monkey" approach reminds me of an excellent paper on "Crash Only Software": http://goo.gl/dqDII

The best way to test the uncommon case is to make it more common.

squidsoup• 16 years ago

It would be fantastic to see Chaos Monkey open-sourced, but it's probably a very domain specific app that only works in the context of their infrastructure.

Some great ideas to be gleamed from the paper you've provided - thanks!

gfodor• 16 years ago

It'd be incredibly easy to implement a v1, a small shell script. AWS has "security groups", which generally are used to split up the roles of machines. (They're used for firewall rules, but in practice having these serve as role metadata is quite useful as well.)

Chaos Monkey is just a script that runs "ec2-terminate-instance" commands on node in certain security groups at a certain rate, and boots machines at the same rate.

Of course, the devil is in the details, but at a high level this wouldn't be too specific to the domain the infrastructure is built around.

apu• 16 years ago

Interesting paper, but for future reference, please post non-shortened links here. Thanks!

aw3c2• 16 years ago

Please do not use URL shorteners when they are not needed. They are obfuscating.

Actual URL is http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.90.... and that is a direct link to a PDF.

jemfinch2• 16 years ago

When pg implements proper hyperlinking that doesn't clutter up my text with an ugly URL, I'll stop using a shortener. I understand the positives and negatives of using a shortener, but the positive impact on readability seems to outweigh the negatives when I'm confronted with an ugly link longer than the text I use to introduce it. Sorry if that offends you.

I should have noted that my link was a PDF, though. My apologies for that oversight.

byteclub2• 16 years ago

If you do decide to adopt your very own pet Chaos Monkey in your next project, make sure you ARE able to gracefully degrade your service in case of failures. Otherwise your customers will see the monkey in action, manifested by "we'll be back shortly" messages. It's easier said than done, since a lot of the time all of us forget to write (or feel lazy, or have no idea how to properly handle) the "else" statements in case of errors/unavailable services/unreachable databases.

Otherwise, good idea. It forces you to think about the perils of distributed environment from the very beginning, as opposed to leaving it to be an afterthought.

kondro• 16 years ago

Hardware is always going to fail eventually. Moving to AWS caused NetFlix to write better code to deal with these failures.

Failures were always going to happen, even in their own datacentre. What they have now is a more fault-tolerant system which should have less downtime overall.

wglb• 16 years ago

Interesting: The Chaos Monkey’s job is to randomly kill instances

Another way to say "If it ain't tested, it's broken".

jasonkester• 16 years ago

I love the idea of setting up a fully working system on AWS, then repeating all traffic from your live site over to it to see how it stands up under load.

No need to simulate traffic for testing purposes. Here's our actual traffic. All of it.

Nice.

Netflix: Lessons We’ve Learned Using AWS

Related Stories