When Feature Flags Go Wrong (2016)

https://www.infoq.com/articles/feature-flags-gone-wrong

by scaryclam • 9 years ago

25 4 9 years ago

rdtsc2• 9 years ago

Feature flags can be dangerous because even an extra boolean flag (on/off) means doubling all the possible configuration states. With just 10 flags you'd have 1024 different combinations. Do you test all of them?

Sure, in practice not all features interact (so you can be sure on module A in the backend is orthogonal to changing color of the UI buttons), but sometimes you can't be sure and feature combinations and interactions never tested will appear in production and break unexpectedly.

One common thing I noticed is that feature A and feature B are both tested by toggling one on, doing a test, toggling it off. The test passes, everything looks good, product is shipped. Customer then turns A and B feature on and product goes up in flames.

I like the idea of auditing configuration changes (not just features others config options as well). I had implemented that across a few products and it saved the day during debugging. For example we noticed customer has been toggling feature combinations we never tested, or they reported "product stopped working" but looking at the they've been tweaking the hell out of it, which made it stop working as we had configured it.

moonman272• 9 years ago

Could you tell us more about how you did your auditing? Did you add more logging?

rdtsc2• 9 years ago

Yap it was done using the logging facility with a separate rotation, log forwarding rules, and such.

arwhatever• 9 years ago

Once worked on a project where the feature flagging utility code was named FeatureMasking.IsSuppressed("feature name"), which would lead one to wonder whether it was the feature or the masking which was suppressed.

Worse, rather than cleanly blocking off full sections of code, it was common to find feature flag checking embedded within multiline expressions like

if (a>b) && c.IsNotEmpty && FeatureMasking.IsSuppressed("whatever") && d <= e ...

When Feature Flags Go Wrong (2016)

Related Stories