Kanban can not save you from the engineering death spiral

http://blog.aha.io/index.php/the-engineering-death-spiral

by bdehaaff • 13 years ago

60 64 13 years ago

blog.aha.io

hacknat3• 13 years ago

In people' experience(s) how likely is it that a company can salvage a situation like this? I'm starting to experience this at work and I've been thinking about leaving (for other reasons than just this), and I wonder if anyone has actually seen a turnaround in an engineering team's culture.

bdehaaff27• 13 years ago

I have seen a turnaround, but it usually takes significant change. And typically that means management change. How long have you been experiencing the problem? The longer it goes on the harder it is to escape.

JimboOmega5• 13 years ago

I was at a company where my entire (too large) reporting change turned over, and yet we were still displaying all the symptoms. It was horribly demotivating to always be blocked, and to have even the simplest of fixes wait months to get deployed.

In our case, the biggest problem seemed to be siloing. Developers were not allowed to touch ANY systems, even the QA and Dev environments. If we wanted to deploy a fix for QA to verify, and the SysAdmin team was busy, it simply didn't happen - often for weeks. The same was true of database problems - if there was even a small flaw in the SQL, I had to sit around and wait for them to come up with something new, put it through our system, and then I could copy and paste it.

The problem was it had been going on for far too long, and was too entrenched. Any new manager who got hired just slid into the current "way we do things" - there was too much momentum.

The various fixes that were attempted were all either counterproductive or pointless. The most common was to "slow down" the process to reduce the number of bugs that got pushed out, which was maddening. There would be other fads (A Kanban board nobody used was one of them)... we also tried to switch to estimating time using fibonacci numbers for no reason anybody could figure out.

We also had sprints that the manager-du-jour would try to revive. They were never meaningful. Instead, when the sprint was over, we'd go through the tasks assigned to us and bump them to the next sprint in Jira.

We went through a couple different managers (and VPs of Engineering/CTOs/etc). The biggest practical difference between them was how much they cared about the trappings of work - that is, how much they cared that developers were at their desks at time X, how angry they got when they saw a non-work website open when they walked by, etc.

None seemed interested in fixing the deeper problems. None seemed able to change the fundamental cultural preference for silos, gates, and an ever slowing pace. None seemed incredibly concerned that a developer who spots a bug and wants to fix it would be embarking on a journey that would involve almost a dozen people, countless "gates", and probably months of time. (First QA needs to verify the bug, then requirements needs to specify what fix they need, then DB team needs to make any SQL changes... and of course managers need to specify priority...).

The company was ultimately sold off at a steep discount.

Xylakant• 13 years ago

From what I've seen it depends on how long the situation has been lingering on. Teams have ups and downs, phases where things just hum along and phases where matters are more difficult. If it's just a phase and a fairly recent development it's possible to turn things around, especially if there's still people around that are motivated and want to change things. A clear diagnosis of what's wrong would be the first step. However, it gets harder to change things the longer the problem has been around - everybody retired to his own little world, blames the rest of the team for failures, too much blood shed between the individuals. It's very hard to climb back up from this hole.

bdehaaff27• 13 years ago

Well said. I totally agree. Sometimes it's just a blip and sometimes there is distrust and structural dysfunction that is going to need a lot of work to fix.

RogerL2• 13 years ago

I have seen a turnaround, a dramatic one. I forget the number, but this was for something in the range of a $60MM contract. I will not identify the company or product for various reasons.

How was the turnaround accomplished? By firing everyone in charge, and replacing them with people that could actually make reasonable decisions. Some lower level people were also let go; we had some people without the chops to do the job. But that was not the main problem. The problem was consistent management focused entirely on 'process' without regard to what we were trying to accomplish. In the form of "MIL-STD-1234 says we have to produce documentation in the form of X. Therefore, produce X". Documentation does not result in a great product. Documentation can only, wait for it, document what was done. That was just one sliver of it, but entirely representative. Methodology X, process Y, 5 levels of sign offs all focused on what color ink you used or if your document's headers use the right capitalization. Just shuffling requirements along an endless document chain, without actually trying to figure out how to meet those requirements.

A really smart systems engineer who had the bosses ear was able to say 'they will never make it', he eventually listened, she was put in charge, our best engineer was finally put in charge of the SW, and from there we recovered.

In the interim we hired a bunch of outside consultants to come in. I don't see that they did anything of value. Lot's of interviewing, lots of 'do x, y, z'; meanwhile the people that had successfully produced this sort of project before were still being ignored.

That is one story. The problem was mid-level management, and only once they were all fired (actually, there were several rounds of getting rid of person A, and replacing them with an exact clone, and of course that person had identical results) did we recover. This probably doesn't apply to any place where the problem is not middle management (say, a team full of cowboys, or a PM that sends you off on wild goose chases, or whatever).

bdehaaff27• 13 years ago

Exactly. Put the folks who can get the work done in charge of actually getting the work done. Give them the responsibility and allow their courage to flourish 9and remove barriers when needed). Celebrate their accomplishments.

programminggeek• 13 years ago

It seems like from what people are saying you either have to get the people in charge to change or change the people in charge.

bdehaaff27• 13 years ago

Yep. Big change is needed one way or another because the ground is approaching faster than most teams in the death spiral think.

pkananen• 13 years ago

The article has some good points, but I'm not really sure what it has to do with Kanban.

bdehaaff27• 13 years ago

Thanks. The point is that applying simple tools like Kanban or trying to "shock" the system by "becoming" agile typically does not work because these tools do not address the underlying issues.

cjensen• 13 years ago

Kanban is the silver bullet of the day which sinking enterprises reach for. I've been a software engineer since 1989, and the entire time has been filled with process/method fads "which will fix everything." In 1989 it was Yourdon's institutionalized bureaucracy. Today it's Kanban.

Doesn't make Kanban broken; it just means that it is the non-solution for today's desperate souls.

bdehaaff27• 13 years ago

Very true. There is always some new "fix it." The problem is often that the engineering tools do not improve product management in the least and if the product strategy is bad and customer understanding is missing, no engineering team or tool can solve that.

bayesianhorse• 13 years ago

One problem with this article is the absence of any reasoning process.

How do you know there is such a thing as a death spiral? How do you know these symptoms apply to the "death spiral" condition. How do you know your solutions actually alleviate the symptoms or abandon the condition?

bdehaaff27• 13 years ago

Personal experience after being an early employee or founder of six software companies and knowing or speaking directly with folks in hundreds of others. Simple experience and pattern recognition. Not science.

VLM• 13 years ago

Whatever metric you use for "working" goes up, and whatever metric you use for "accomplishing" goes down. Perhaps if you fix the problem those trends will reverse.

Doesn't need to be hyper game-able beancounter make peoples pay depend on fudging the numbers. IN fact that will create negative creativity. You get better results if you write down and graph the departmental ratios of currently open bugs vs lines of git commits and for god sakes don't let anyone know or the gameification is on...

bdehaaff27• 13 years ago

Yes, I think you are saying that the focus needs to be on results. Not effort. I agree, but is that what you are highlighting with your comments?

seiji• 13 years ago

Death spiral: When you keep creating products with no demand in the real world [1]. You create things, release things, nobody uses it, but by then, you're working on the next version nobody wants either.

Death spiral: Working on components of a larger project with no hope of ever integrating all pieces into a coherent final release. Everything gets incrementally improved in isolation, but nobody can figure out how to cut a product out of all the pieces.

[1]: Plenty of demand exists in the delusional minds of executives, PMs, and clueless employees though. It just doesn't match the real world.

bdehaaff27• 13 years ago

True. That's one of the ways that you end up in this engineering death spiral -- bad market / product fit and thrashing strategy.

3pt14159• 13 years ago

The article's own link to the authors product is broken.

bdehaaff27• 13 years ago

Thank you for pointing that out. Grateful for the call out. Best.

bdehaaff27• 13 years ago

It's been fixed.

ecesena2• 13 years ago

A friend of mine, CTO at Elfo [1] (a 60-ppl Italian software basically-unknown company) is successfully using Kanban for software development since almost 2 years.

Each team/project (6 to 10 developers) has it's own Kanban board, each person has no more than 1-2 tasks/phase and they focus on keeping the phases balanced, i.e. with the same amount of tasks. They're happy since its a light weight way to have a full picture of what's going on.

His original notes (in Italian, sorry) [2].

[1] http://www.elfo.net [2] http://www.robertocappelletti.com/2012/01/kanban/

bdehaaff27• 13 years ago

Great. Thanks for sharing this success story. It would be interesting to know if the engineering team is motivated and understands the why they are building what they are.

ecesena2• 13 years ago

As far as I got a lot. The main reason is that, with limited effort, you see your progress (tasks move). Moreover the management is really enthusiastic of the approach, so they're often showing to guests (like I was) and promoting the activity as something unique of "their guys", so I guess this also makes people proud of what they're doing...

lmm2• 13 years ago

Huh? I'm pretty sure Kanban helps with all of these.

Why are we building this feature? With a strict limit on work in progress, someone must have explicitly decided that it was more important than all the other features in your backlog. Sure, you can still disagree with their weighting, but at least it forces them to prioritize, rather than delaying the product with a kitchen sink of everyone's ideas.

We can't release until quality is better? If you keep cards in progress until they're released (and you should), you very quickly reach a point where you have to release, or it's simply impossible to do any more work. In fact this is pretty much the core problem Kanban was meant to solve.

I'm blocked? A blocked card is filling up a valuable work-in-progress slot, and a manager should notice very quickly. If particular issues keep blocking several cards, that means the manager is seeing those same issues every day; sooner or later they should take notice.

bdehaaff27• 13 years ago

Kanban does nothing to solve the product strategy and market fit questions and is often applied as a fix-all for much deeper rooted problems. I would also argue that Kanban has the tendency to turn engineers into widget builders and is demotivating. Here, take this tiny piece of work and when you finish it I will reward you with another. Keep your head down and just trust me.

mprovost• 13 years ago

I would say Kanban is the opposite of keeping your head down - you have to come up for air every time you finish something and priorities can change each time. Waterfall is the classic head-down approach, work on something for months/years and then release it and see if there is still a market for it. I think there are strong tendencies towards waterfall in many tech personalities - "I'm just going to disappear and come back with something brilliant at some unspecified point in the future." People don't want to feel that they're building widgets when they could be making a cathedral but that's often what is needed. Agile and Kanban fight this tendency and are therefore not always popular.

acdha2• 13 years ago

You can't solve social problems with dogma. Every single thing you mentioned above will work if it's done right but if you're in a toxic, poorly-managed environment each of those assumptions is wrong.

It is absolutely the case that you will find managers who make everything priority 1, ignore problems for far too long, keep piling features into dreadnaught-sized releases, etc. The only thing which can change that is a different manager – either because the previous one is sacked or because the development team threatens to resign en masse unless they are given enough control to do their jobs responsibly.

bdehaaff27• 13 years ago

Great insights. Give the folks who can get the work done the freedom and responsibility to get the work done. Managers should remove barriers -- not impose additional ones ever time something goes wrong. It's hard enough to build great software.

ChikkaChiChi• 13 years ago

Excellent comment/reply. Kanban is a tool, but its not a solution to a cultural problem.

lmm2• 13 years ago

It is absolutely the case that you will find managers who make everything priority 1, ignore problems for far too long, keep piling features into dreadnaught-sized releases, etc. The only thing which can change that is a different manager

But Kanban doesn't let them. You are required to prioritize upcoming cards in a linear order, not equally. Limiting work in progress is literally the definition of Kanban. If your manager won't do that, it's not that Kanban isn't solving your problems, it's that your manager isn't adopting Kanban.

wwweston• 13 years ago

Yep.

Productivity frameworks can be helpful in a context with people who are smart enough to process the central insights and genuinely interested in playing a longer game than an immediate problem or narrow personal/career agenda.

But in some hands, the important insights either don't get absorbed, or they're wielded selectively to poor effect. Bad decisions can actually be justified with situational appeal to some guiding principle -- but when it cuts the other direction, the principle gets played down and the other side of any tension present gets played up.

coryfklein• 13 years ago

Warning: title not reflective of content.

bdehaaff27• 13 years ago

Thanks for the comment. The point is that using Kanban or implementing "agile" is not a savior from the engineering death spiral. The post includes that exact point in the first paragraph.

karlkfi3• 13 years ago

The real silver bullet behind Kanban has nothing to do with Kanban. It's the desire for continual improvement and willingness to start where you are and improve from there.

Anytime you attempt to make a huge process shift you are tempting fate and risking failure. The short term drawbacks to huge change can often stifle the process before it reaches maturity. The trick is to work through the rationalization with the team based on current observable problems and lead them towards self-realization and analysis of their own problems.

Most people have ideas of how to make things better, but if you lead them towards the desire for slack, optimizing for the bottleneck, rather than maxing out everyone as if they're all the bottleneck, then you can easily get from there to Kanban-like WIP limits. From there you can adopt other solutions as the problems they tackle are realized.

Smart people frequently arrive at similar approaches whenf aced with the same problems, but not always. Your team may find some novel approach that works better for your context, or they may go looking for options and find Kanban. But the goal should begin with being able to communicate process problems, ask how they can be improved, test the potential solutions, and adopt the best ones.

Unfortunately, the biggest hinderance to gradual process improvement (once your team is on board) is that software tools tend to offer a "whole solution" and don't have the ability to grow with your team's maturity, unless they're writing it themselves.

bdehaaff27• 13 years ago

Thanks for the detailed thoughts. I am not sure though that Kanban or an other agile tool can solve the problem when it is one of leadership or direction. Strategy and management do not rise out of the incremental planning bits.

karlkfi3• 13 years ago

One of management's jobs is to enable and empower the workers. One of the principals that enables Kanban development is that management should not dictate worker process. If management doesn't know that they may be tempted to fix the development problems with more process, which actually disempowers the workers. Adding more dictated process may work in the short term, but it's a tactical fix, not a strategic one. A well run business keeps tactics and strategy in balance, and hopefully delegates the tactics as low as possible, to retain empowerment. Good management acts as a multiplier to worker productivity. Bad management cripples what it's trying to manage.

So yes, you need good management, but good management alone does not guarantee success. Management cannot be the source of the solution to every problem.

The other bit to take into account is that management can benefit just as much from its own cycle of metrics, feedback and retrospective action. All levels of an organization need to be able to improve, not just the developer teams. However, that's usually a completely different set of tools and skill sets.

jonathandart• 13 years ago

Can anyone here recommend some references for someone wanting to learn about Kanban?

karlkfi3• 13 years ago

http://www.amazon.com/Kanban-ebook/dp/B0057H2M70/ Note the subtitle: Successful Evolutionary Change. Don't try to change everything all in one go.

And if you already know Scrum, here's some differences: http://www.infoq.com/news/2009/05/kniberg-kanban-v-scrum

bdehaaff27• 13 years ago

There is a good overview here for Kanban for development: http://en.wikipedia.org/wiki/Kanban_%28development%29

msaspence• 13 years ago

The key word (that is admittedly missed from the title) is "blindly": 'blindly applying "agile shock therapy" or introducing Kanban will only suck you further into the abyss'.

Which is a bit of truism: do anything blindly and your chances of succeeding are based entirely on luck.

Depending on the specific circumstances Kanban, or any other agile methodology, may or may not be part of the solution to the so called "death spiral".

Kanban can not save you from the engineering death spiral

Related Stories