How to make breaking changes and not break all the things

http://matthew.mceachen.us/blog/how-to-make-breaking-changes-and-not-break-all-the-things-1315.html?hn

by mceachen • 12 years ago

40 12 12 years ago

matthew.mceachen.us

lsh1232• 12 years ago

One important point is missing from the article: make sure to have unit/functional tests to run to make sure your changes are indeed are not breaking anything.

mtillman• 12 years ago

for the record @mrm, the above comment was not from me. :) solid article though as usual.

mceachen3• 12 years ago

Absolutely. Thanks. I've added Step 0 to reference this, and added an attribution at the end of the post.

One thing that surprised me when I started at Twitter was the requirement for every line of code to be reviewed by another engineer—and if you change or add implementation, you can't ship if there isn't corresponding test changes or additions. More tests isn't a panacea, of course, though. There's definitely a spectrum of good, explanatory tests and tests-that-just-cement-the-implementation-details.

lsh1232• 12 years ago

The automated tests are not panacea of course. Yet, you need to have enough tests to give you confidence that the change indeed is breaking nothing. It is one story to perform a deep refactoring on a C++ or Java code with 95% tests code coverage where either compiler or tests will pickup things for you. And it is a completely another story to do a refactoring on a PHP frontend code w/o any compiler/static analyzer and w/o any tests where your best tool is search-and-replace. Even the "depth" of safe refactoring will be very different in those two cases.

rjbwork• 12 years ago

I've found that integration/functional tests are far more important for these sorts of base structural changes to applications, especially when it involves fundamental changes to how data is physically stored.

jakozaur• 12 years ago

Very good approach. Also having some controls like feature flags per customer/user is very useful. Sometimes migration takes a while and it is useful to move users gradually through those phases with ability to rollback.

mceachen3• 12 years ago

Absolutely! I didn't reference it originally, but saw that it was already out in the wild, so I added a short paragraph about Twitter's solution, "Decider."

MaxGabriel• 12 years ago

When we were switching a contained part of our codebase from Mongo to MySQL, we ran both in parallel and stored a key in Redis to let us turn off the MySQL backend (we re-used our feature flag utility for this).

This turned out to be quite useful. If we were throwing exceptions in the new MySQL code, or even just suspected the new system of causing problems, we could switch it off. We also switched it off so that we could easily run migrations without taking down the site.

quest88• 12 years ago

Thanks for the article!

mceachen3• 12 years ago

I actually wrote it for some coworkers at Twitter—they were struggling with this, and it's a surprisingly non-trivial problem to tackle.

polemic• 12 years ago

This is a good starting point. One thing I'd add is the need to consider the effects of table locks during changes. It's a nasty surprise when you've tested your multi-stage process in dev, only to find during production updates that a long running select is blocking, or that your change causes request to queue until you exceed connection limits.

How to make breaking changes and not break all the things

Related Stories