Continuous Deployment at IMVU: Doing the impossible fifty times a day.

http://timothyfitz.wordpress.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/

by TimothyFitz • 17 years ago

172 73 17 years ago

timothyfitz.wordpress.com

patio11• 17 years ago

One of the greatest lines I have ever read on a blog:

It may be hard to imagine writing rock solid one-in-a-million-or-better tests that drive Internet Explorer to click ajax frontend buttons executing backend apache, php, memcache, mysql, java and solr. I am writing this blog post to tell you that not only is it possible, it’s just one part of my day job.

ntoshev2• 17 years ago

These are automated functional tests. It would be interesting to know if the author uses unit test too and how he writes tests in general. Also some code:test ratio metrics would be indicative of how much weight does all this testing add.

jmathes3• 17 years ago

I work at IMVU as well, know the author personally, and can tell you that yes, he unit tests too. In fact, like all of us, writes all code test-first.

For test strategy in general, you need a whole seminar. There's a good blog on testing over at http://googletesting.blogspot.com/ <- go there!

As for code:test ratio, I'm not sure what metric you want, but using just php files, there are ~1k active test files and ~7k php files total. So if you take into account that 1k of that 7k is the test, and some is test-setup and obsolete files we no longer use, it's maybe about 5:1 code:test. However, a lot of the code is third party OS software packages. In-house code is about 1.5:1 test:code, in my experience. These, by the way, are very rough estimates.

plinkplonk• 17 years ago

"writing rock solid one-in-a-million-or-better tests that drive Internet Explorer"

I find this unparseable. (English is not my native language). As far as I know "one in a million" means something like "very rare". Help?

smhinsey• 17 years ago

i am a native speaker and it's not entirely clear to me either but i would feel safe in venturing to guess that it's a reference to the false positive rate

ivank• 17 years ago

He means the test is broken if it fails more than once in one million test runs.

timf• 17 years ago

In the article it talks about tests that fail only one in a million times (referring to the quality of the test, not the test case being exercised), I think that is what is meant.

ivey• 17 years ago

When I say reliable, I don’t mean “they can fail once in a thousand test runs.” I mean “they must not fail more often than once in a million test runs.” We have around 15k test cases, and they’re run around 70 times a day. That’s a million test cases a day. Even with a literally one in a million chance of an intermittent failure per test case we would still expect to see an intermittent test failure every day.

He means that test failures are not acceptable in their culture, and they have a LOT of tests.

briansmith• 17 years ago

"rock solid" is "very reliable" "one-in-a-million-or-better tests" is "tests which fail less than one in a million times".

Our Internet-Explorer-based tests are very reliable; they fail less than once per million executions.

edw5192• 17 years ago

not only is it possible, it’s just one part of my day job

A necessary, but not sufficient, requirement for a nimble start-up.

mechanical_fish3• 17 years ago

Don't be too disappointed if a single submission gets a lukewarm or confused response on HN. The upmods and comments on here are a lot less consistent than what you're used to. ;) Just keep writing. It's really valuable.

Also, it's clear to me why your daily routine might sound like science fiction to the median HN reader: A lot of programmers have never seen a system like this. As those of us who were online during a specific half-hour period a couple weeks ago can attest, even Google doesn't have a system that's remotely as reliable as this: It appears to be possible to break all of Google search, worldwide, in ten minutes by misplacing a single character in a text file.

jeremyw2• 17 years ago

Hmm, on your Google point, we know that they use partial-cluster deployments extensively, and several presentations point to sophisticated testing of these momentary guinea pig users. I wouldn't hold a one-time lack of a sanity check against their total uptime history. Tests ain't perfect.

mechanical_fish3• 17 years ago

I agree that we shouldn't extrapolate too much from this one incident. But it's not like Google's super-secrecy policy gives us much choice. If anyone from Google wants to tell us about their deployment infrastructure and explain why this one incident really was a nigh-impossible black-swan one-in-one-billion-hour freak of nature -- or why Google has sensibly traded away a certain amount of uptime in exchange for a more flexible architecture (or, perhaps, more cash to spend on tasty gourmet pizzas) -- I'm sure we'll all listen with rapt attention. Until then, we get to tease them mercilessly. ;)

Meanwhile, I'm sure that the original submitter would agree that tests ain't perfect. If you read the link at the top of this blog post:

http://timothyfitz.wordpress.com/2009/02/08/continuous-deplo...

...you'll find that this isn't merely an article about automated testing. Automated testing is just a part of the mighty continuous-deployment ecosystem being described here. It isn't even the real heart of that system: The heart is a planned, well-designed, semi-automated routine for rolling back changes in production. They roll out a change to a subset of their servers, monitor for statistical anomalies in the usage patterns of real, live users, and only continue the rollout if there are no anomalies. If they run into trouble, back they go.

edw5192• 17 years ago

lukewarm or confused response

I have often posed questions here about things I've been doing for years just to see what others are doing or if there is a better way. Invariably, someone tells me it won't work when I already know better.

This tells me 2 things. I've encountered someone who speaks when they should be listening (what else is new), and, more importantly, that I'm pushing the envelope enough to make otherwise knowledgeable people uncomfortable. Good.

TimothyFitz8• 17 years ago

I definitely wasn't disappointed. In fact it was refreshing; I didn't realize how hard of a problem other people considered Continuous Deployment (and in fact, how little other people had considered it at all).

I also didn't mean to rag on news.yc, on average the responses here are better than the original posts which is an unparalleled level of quality.

I left the technical issues unspecified in my first post on Continuous Deployment, and the comments on that post already had started discussing the path of solutions we ended up building ourselves!

I still have another post to write about this, because we also ship a native windows client. We ship daily prereleases of it, and roughly biweekly full releases (offered to all users). That's close to, but not quite as impressive, as the update system that google chrome uses. Google definitely still has us beat in certain categories.

JabavuAdams• 17 years ago

> As those of us who were online during a specific half-hour period a couple weeks ago can attest, even Google doesn't have a system that's remotely as reliable as this: It appears to be possible to break all of Google search, worldwide, in ten minutes by misplacing a single character in a text file.

How can we conclude that the same isn't true of IMVU? The fact that such a rare event hasn't happened to them yet tells us very little.

DannoHung• 17 years ago

I tell people that we should aim for this sort of automation and they pat me on the head and say, "No, no, that will never do."

I think there's an idea that if something goes wrong because you let an automated system do it, it's somehow much worse than if something goes wrong because there was human error. I don't really understand the reasoning.

TimothyFitz8• 17 years ago

Exactly. Drew Perttula put it better than I'll be able to: "IMHO, manual testing has only two advantages: it’s the easiest thing to [try to] do; and it has a lovely accountability chain. You can always blame the developer, and non-technical people will easily accept that this is the “inevitable cost of software engineering”."

mst2• 17 years ago

The thing I'd be really interested in is how you deal with UI changes - I've never found a satisfactory way to test "is this ugly/confusing" other than letting a few users bang on it on a staging server.

akronim2• 17 years ago

There's a difference between automation and how often your customers see something going wrong. I'm all for the automation. But lets say the error rate is so low that just 0.1% of automated releases go wrong. Rolling out 50 times a day means you'll expose an error every 20 days. Compare that to a monthly, weekly or even daily cycle and you can see you're exposing yourself and your customers to problems without much corresponding gain.

Retric• 17 years ago

If you're doing 10 releases a day and someone notices a bug then:

A) You can quickly fix it reducing the number of people that see that bug.

B) You can probably release the fix without breaking other things.

Is it better for you to release 5 bugs that 5% of your users sees for a few hours or release 1 bug that 100% of your users see for 2 weeks?

jacquesm3• 17 years ago

Any chance of more detail than you are giving in your posting ? This is extremely interesting stuff, I'd really like to know a lot more about what goes in to achieving this.

eries3• 17 years ago

what do you want to know?

jacquesm3• 17 years ago

Everything :)

No, seriously, I'd be much obliged if you could tell what tools go into your setup, how much of it is created in house - and thus unavailable - and how much of it is off the shelf, preferably open source. I'd very much like to spend time on recreating what you've done there.

lgriffith4• 17 years ago

Interesting idea but....

Looks like meeting that goal would constrain you to write code to be used by a robot and not by a human. There may be many cases where this is both doable and acceptable to the end user. So no problem with that.

I am greatly challenged to see how this could be done for a highly interactive, visually oriented, subtle pattern generating response to user input, type application. Computers are still not as bright as earth worms when it comes to generalized pattern recognition. Which means we programmers are about as bright as earth worms when it comes to writing such code.

How then could computers automatically test all the software reactions to the wonderful and totally unpredictable behavior of mere humans as they interact with your software? The test cases would expand to consume all the resources available for development. All you would get done is writing all but impossible test cases. At least you wouldn't ship bugs.

This does not consider the explosion of combination and permutations of inputs that prohibits exhaustive testing that no matter how many systems you run tests on.

It would be much easier and cheaper to go out of business. Your certainty of being free of shipped bugs would be much better than one in a million.

mechanical_fish3• 17 years ago

As I noted elsewhere in this thread (http://news.ycombinator.com/item?id=475391), this article is not merely about automated tests. The author says that his company is using continuous deployment because it lets live, human end users bang on the code, as quickly as possible, in bite-sized chunks that can more easily be rolled back and fixed.

lgriffith4• 17 years ago

Then why do such an exhaustive automated test?

Why not have your local tests, automated or not, cover the common cases and error conditions to catch programmer stupidities? Then let the actual humans do the strange corner cases.

If your design is even close to correct, testing repeatedly tested code is pointless. If your design is corrupt and your implementation is sloppy, no amount of testing is going to save your ass.

I do very rapid turns and I am a one man team. I can turn my system in less that 30 minutes and have the user testing it in a live situation on the other coast. If I want 10 turns a day, I can easily do it. Low coupling, high cohesion, clean correct design, and disciplined implementation makes it possible.

I agree that doing things in small chunks is a great way to do it but doing the equivalent of a weeks worth of global automated testing for each small change seems like a silly exercise. That is except for the server hardware salesmen and system admin people.

The sales commissions and payroll look rather good. The production of real value is questionable. Bang for the buck is as important for testing as it is in any other part of product development.

amix3• 17 years ago

From what I have read Facebook use a similar method: commit and deploy often and rollback if something messes up. We also use this method on Plurk.com and have done so for about a year. Thought, IMVU's case is pretty extreme :)

The major problem is rolling back client side changes (that are located in scripts or CSS). This is pretty costly to rollback, because of browser cache - we solve this by having real versioning of the static files so we can force a refresh of browser cache (real versioning = script_{timestamp}.js and not script.js?v={timestamp}).

delano2• 17 years ago

What do you mean by real versioning?

The major advantage to using "script.js?v={timestamp}" is that it maintains a consistent URI for the resource. Whereas with "script_{timestamp}.js", everything that points to it needs to be updated every time it changes.

You could create a symbolic link or rewrite rule that directs requests for "script.js" to the latest "script_{timestamp}.js" but it's more convenient to use a URI parameter.

amix3• 17 years ago

The problem with script.js?v={timestamp} is that it's ignored by some browsers while script_{timestamp}.js isn't. And with script.js?v={timestamp} you can't set good cache headers.

Also, if you ever move to a CDN, then you are forced to use real versioning (at least with Amazon Cloudfront).

The versioning scheme we use is `md5 hash of name + file contents + file extension` (and not timestamp).

willwagner• 17 years ago

I've worked at places that have done this sort of thing too. We basically dropped css and js files in new directory with the svn rev number which made it very easy to deploy and break through the client side cache.

The urls would be something like:

/static/r12345/foo.js and /static/r12345/foo.css

abstractbill2• 17 years ago

commit and deploy often and rollback if something messes up

This describes pretty well what we do at Justin.TV too. These days I push new code about 5 times a day.

emmett• 17 years ago

Although to be fair, we don't have that totally sweet "immune system" thing. I have to admit, it sounds pretty cool.

TimothyFitz8• 17 years ago

I've read your post on unit tests, and I didn't understand what you were trying to say.

Were you saying don't write automated tests that test your code, instead focus on monitoring the actual production invironment?

Or were you saying that specifically the "unit test" class of automated tests are not worth their time?

I can imagine a system that monitors the business metrics well enough to prevent defects from slipping into production (it's a stretch, metrics are soft and squishy moving targets), but I can't imagine using only those metrics to find every bug you ever slip into production. Metrics are so distant from the bug that caused their downturn; you'd waste so many cycles debugging. The gap between writing the code and finding the problem would be much larger than if unit tests found them; that has to slow things down as well.

zhyder• 17 years ago

That makes sure the code is consistent if the user refreshes the webpage or just visits it the first time. But what happens if the user just keeps the AJAXy web-page open for hours (as I do with Gmail for instance)? If you deploy too often and both frontend+backend code are in flux, you're more likely to end up with an inconsistent code state.

I guess you could make the frontend code aware of the code version, include it as a param with each XHR request, have the server check versions and return a "version mismatch", and then produce some alert on the browser asking to refresh the page. But this would tradeoff far too much usability.

mst2• 17 years ago

Last time we ran into this one, we made sure as much page state as possible was pushed into the fragment portion of the URL (for bookmarkability as much as anything else).

Then when the AJAX stuff saw a version mismatch, it would wait until the user completed any operation that -wasn't- stored in the fragment and put up an "updating, gimme a sec" box, and refresh itself.

It was a hell of a lot of work but -extremely- slick (which I'm allowed to say because it wasn't me who wrote that part ;)

akronim2• 17 years ago

deploy/rollback is probably ok for a consumer site. But not everything is a public website (no really...) - if you're deploying a service with an SLA with dollar penalties for downtime you might want to stick to a more traditional release cycle. I sure hope the phone network, the stock exchange and my bank aren't using deploy/rollback and releasing 50 times a day!

gfodor• 17 years ago

This is great stuff, thanks.

pj• 17 years ago

Continuous deployment is good, but the comments are valid.

There is a certain non-zero probability for errors to occur during deployment. Binaries have to be reloaded, database connections have to be reconnected, sessions have to be restored, etc, so the more you deploy, the larger the coefficient before this probability in the "will something go wrong" equation.

So, what we do is break up our system into deployment groups where some handful of users gets updated a few times an hour sometimes. We test the deployment on this small set of users, usually they know the change is coming and are ready to test the change in real time.

Sometimes we repeat this process using different deployment groups. Test in this one, then test in that one, until we get a final small errorless deployment and then we roll out to the masses.

If it is successful, we roll it out to the masses.

Your site doesn't have to be /all/ beta or /all/ production. You can have batches of users in different groups.

shiranaihito3• 17 years ago

Why do all changes have to end up in production immediately?

teej• 17 years ago

Because having thousands of real users running your code gives you insight that automate tests simply cannot match.

shiranaihito3• 17 years ago

Yes, but they might give you a hard time too if you put out something silly before thinking things through.

kragen2• 17 years ago

When something breaks in production, it's easier to figure out what it was and fix it if you only changed a few things since the last time you updated production.

pskomoroch• 17 years ago

I was thinking of doing a basic Django + Selenium + Hudson continuous integration how-to blog post, but this blows me out of the water :)

TimothyFitz8• 17 years ago

I would love to see that post. The first question most people ask me is "How do I get there?" and I don't have a great place to point and say "start here"

A well written concise introduction to continuous integration / constant testing would be a boon to this community.

forkqueue• 17 years ago

I'd still like to see that post, this blog post gives the overview, yours should give the details.

inerte• 17 years ago

I think the original article misguided some people. It all looked very simple, update the code and put it in production. That _is a horrible idea_, as some have noted.

What's not horrible is having thousands of tests, on dozen of machines, 9 minutes to-live, with selective updating of users, and rollbacks, as this article has explained.

The original post was too light on details, I guess. Its intention was not to be comprehensive anyway, the focus was why recently changed code should be put in production ASAP. But it looked like the author was simply FTPing after commit. And the whole "SOMEONE IS WRONG ON THE INTERNET" thing kicked in.

TimothyFitz8• 17 years ago

Honestly I think it's a gradient.

I'm also one of the developers on a hobby project called http://TIGdb.com (Jeff Lindsay is the other, and has written the majority of the website) We don't have a big Continuous Deploy infrastructure, but we also don't have the users and business requirements of IMVU.

We started with the usual, completely manual deploys and hard-to-setup sandboxes, and have been iterating towards a fully automated setup ever since. The entire time we've been doing this, we've been committing and deploying often. Our users are patient, because we're giving them something they can't get elsewhere and we're giving it to them for free. As we do introduce regressions, we'll post-mortem them (probably using the 5 why's technique) and we'll slowly evolve a system to prevent regressions. If the site is a success, we'll have evolved a world class deploy system. If the site never makes it that big then we won't have wasted time on infrastructure. It's classic lean startup thinking (even though TIGdb is really just a hobby project).

sbraford3• 17 years ago

Just curious - who maintains the Selenium tets, and how big is the development / "QA" team?

I've never worked in a team big enough that it could devote resources to maintaining all of the following kinds of tests: * unit * functional * AND acceptance * plus writing the actual code

IMHO, a neutral third-party group like QA should be responsible for writing & maintaining acceptance tests.

s3graham• 17 years ago

Whoa, I love the sound of this as far as development process... But what really blew me away is: 3D Chat makes $1M/month? Really? Or did I find the wrong IMVU?

TimothyFitz8• 17 years ago

Yep, that IMVU. Here are a few more staggering statistics that we've published: http://www.vator.tv/news/show/2009-01-22-recession-not-affec...

mhartl• 17 years ago

I don't recall the last time an article linked from Hacker News so quickly and dramatically expanded my notion of what is possible in software development. Bravo!

Continuous Deployment at IMVU: Doing the impossible fifty times a day.

Related Stories