Deployments Not Releases – Get Good at Delivering Software

https://blog.mangoteque.com//blog/2023/03/13/deployments-not-releases/

by kiyanwang • 3 years ago

44 18 3 years ago

blog.mangoteque.com

carlmr2• 3 years ago

Feature flags are not a panacea for deployment issues, they can even create many issues.

If there is any interaction between features you start getting combinatorial explosion of possible code paths, and you all of a sudden get issues that only affect specific customers due to their set of feature flags.

Feature flags have intrinsic complexity. Keeping the code harmonized between customers can cause extra effort when you need to redesign pieces of the system to not need alternative code paths.

Both of these costs should be weighed against each other.

re-thc2• 3 years ago

Yes, I've seen not-so-big product with 100s of feature flags. It's impossible to test for. Just have to hope for the best. No 1 can keep track of them or know what are any more.

To make matters worse it can be dynamically enabled based on cookies / headers per customer.

dboreham• 3 years ago

They should be removed after the feature is deployed and stable. Some folks have a CI task that checks for stale FF and alerts.

dgb23• 3 years ago

There are two similar things that can easily be conflated as they have some overlap, or can at least be used interchangeably in some cases.

For one we have feature flags, which should be used to disable features for certain environments, possibly AB testing and so on. You use those to reduce risk and friction in an ad-hoc fashion. They are meant to be simple booleans in code and short lived.

Then we have feature scope, registration, authorization etc. which can be more wide reaching. They should be part of the data model and treated like everything else that is data driven.

When we conflate the two, we get into trouble like you describe.

tyingq• 3 years ago

It feels like the article should mention the other side of feature flags. The benefit is obvious, but it does also create other problems. Like, 8 boolean feature flags means 256 true/false combinations that might not all have been tested.

jacquesm• 3 years ago

Feature flags can work if you are religious about retiring them as soon as possible otherwise you will end up with just another layer of complexity. Ideally by the next release they should be gone unless a feature flag is the gate to an upsell. And then the flag should be on the upsell, not on any particular feature!

Featureflags should never be used for privileged operations assigned to particular users.

emilecantin• 3 years ago

I guess this might work in a pure SaaS environment, but our product has an on-premise version (think Jira data-center), so what keeps us from releasing more often is the maintenance burden that imposes on our customers.

JohnFen• 3 years ago

what keeps us from releasing more often is the maintenance burden that imposes on our customers.

You guys rock. Almost no software companies pay any attention to the fact that every new release imposes a burden on their customers, and rapid release schedules impose that burden rapidly. In a way, it's offloading some of the development burden onto your customers.

This is true for pure SaaS products as well, although to a lesser degree.

I am very happy to hear of a company that remembers this and takes it into consideration.

gnat• 3 years ago

We release new on-prem versions every fortnight and it's up to the customer to decide if they want to upgrade to it or not. We ack the maintenance burden but also that if they want a new shiny then they need to accept that it means an upgrade.

Downside, of course, is that different customers have different versions of our software on their premises. Support's job is harder, as they need to know "oh, you're on 2.14 and this is a problem with preorders, that bug was fixed in 2.18" for a bunch of historical versions and problems.

roel_v2• 3 years ago

Maybe this works in the type of work this guy does, but I've been horribly burned by feature flags. How do you test for the combinatorial explosion of feature interactions? Even if you separate into 'silos' that pinky promise do not affect anything beyond some 'defined' boundary, interactions are still too numerous to test as different products very quickly. And now you have person A using 'feature flag combination' A' and person B using feature flag combination B', and they ask you a question about how this or that is supposed to work, and you won't even know until you are able to exhaustively reason through what the effect of every feature flag this person has en/disabled is - if you even know that at all.

Yes in the end it's all in the process, but feature flags are not the end-all be-all these few paragraphs posit them to be.

capableweb3• 3 years ago

They are supposed to be removed quickly. The biggest issue I've seen with implementations of feature flags in products are that they are not removed quickly enough. They should be removed as quickly as possible, to avoid what you're saying. In some places, I've even advocated and implemented strategies for not considering something actually done until the feature flag has been removed. So it can be deployed and activated in production, but in terms of project management, the overall task of the feature shouldn't be considered done until cleanup has been done. This process has been the most successful.

Another strategy that kind of worked but not as well, is requiring the cleanup of the feature flag to happen in the next sprint. But it tends to be de-prioritized, and sometimes even pushed to the next sprint as other things are always more important.

Another strategy that I've discussed and thought about but never actually implemented or advocated for, is limiting the max amount of active feature flags in the specific project to N numbers (ideally 1). So if you want to activate another feature flag, you have to first cleanup the previous one.

capn_duck• 3 years ago

If they are limited to 1, then they are not appropriate to use as a deployment mechanism. Do you only have one change being worked at a time? On my team, there's typically at least have a dozen different stories being worked in parallel.

roel_v2• 3 years ago

Yah 'supposed', but that's (to me) in the same bucket as 'different features behind flags aren't supposed to interact' and 'we're supposed to have detailed documentation of who is/was using what set of features at time x'. That's also what I meant with 'it's all in the process' - if there is good management of all of this, it may work. And I'm sure that for some people, it does actually work. Meanwhile for me, I'm staring at build scripts with 15+ year old feature flags that at this point are realistically never going to be removed, and people still sometimes suggest 'yeah but this one little thing we can put in a custom build until it works for everyone else too...' So now not having feature flags is a hill I'll die on.

Deployments Not Releases – Get Good at Delivering Software

Related Stories