Agile and the long crisis of software

https://logicmag.io/clouds/agile-and-the-long-crisis-of-software/

by gumby • 4 years ago

176 169 4 years ago

It was going so well until it turned into “old white men created Agile and they don’t understand POC’s and women”.

The ‘weirdos’ in the first paragraphs were primarily nerdy men. The landscape has changed tremendously since then - Agile is terrible for many things, but it does not incite racism or bias. Perhaps the only bad thing it does not do.

I was hoping for a better ending to this read.

mr902102• 4 years ago

I am African, and I cringe hard whenever somebody refers to a group of people by their skin feature, regardless of where they come from.

I personally don’t think it’s ok to refer to a person like that, it’s a rather gross oversimplification of a human being.

We ought to be more sophisticated than that.

odshoifsdhfs• 4 years ago

No, you don't understand (/s), now he is in the 'in group' and gets brownie points, and because he used that logic, you can't criticise him or you are a racist/something other.

discreteevent• 4 years ago

"The mind is the standard of the man" - Martin Luther King. (Just in case: in those days 'man' meant 'human')

planarhobbit• 4 years ago

I love it when people who have no idea about our profession start doing this race baiting bullshit. I’m really hoping (too optimistic, I know) that HN will collectively one day start flagging articles written by clueless people who spend an awful lot of words to say less than nothing.

Embarrassing.

mr902102• 4 years ago

Talking about agile, I subscribe the ideas from Allen Holub (holub.com), especially because he doesn’t talk about a one-size fit all formula, instead he focuses in collaboration between peers, and short feedback cycles aiming to solve business challenges.

nonstickcoating• 4 years ago

I do not think the article argues that Agile somehow incites racism and bias, it argues that it doesn't account for them while it should.

asplake2• 4 years ago

Agile's crisis (which is real) is because despite the generativity of the manifesto, the frameworks it spawned largely ignore how change happens in a healthy and sustainable way. It is deeply ironic how much the Agile industry leans into 1990's models of change (solution-driven, imposed, etc).

That explains an awful lot. When people think that it's enough that they're right, bad things happen. Agile doesn't actually need a theory of society to understand that, just a little humility.

mojzu• 4 years ago

It's also only a couple of paragraphs in a long post and offers some evidence of parts of the agile methodology (retrospectives and code reviews) being used to harass people so it seems unfair to knock the article for those reasons in my opinion

BlargMcLarg3• 4 years ago

That's a problem with top-down management in general. If your management is largely old white men who pull the strings, yes obviously it's going to account for their opinions the most.

Author misses the part where the majority of the working force is also "white men" who see no benefit from this either, because it turns out, the decisions of management aren't for "white men" or even "old white men". The decisions are for "our management".

meerita2• 4 years ago

It's the gender and race wars agenda. Everything has to have a gender/race problem.

pif• 4 years ago

No, it's short-sighted people from US not realising that the internet is broader than the US.

Please, if you are writing an US-centric article, state it in the title. Example: "Agile and the Long Crisis of Software in the USA".

piva002• 4 years ago

Agile as per the Agile Manifesto isn't bad. At all.

I saw the transition of the industry from a complete waterfall process to the overhyping of agile, the pushback and now the land where agile as per the manifesto is more-or-less a given. The old ways were so much worse...

The bad sprouts from agile are the multitude of processes and bullshit methodologies that tried to ride the hype and create "The Agile Way". Scrum and others are a side-effect of this and I'm very glad to see Scrum being shunned for the past 4-6 years and instead engineers being attracted towards much more lightweight processes.

The problem with agile is that it only works when people had experiences, bad and good, and a team can tailor how their agile process should look like given their constraints. Throwing a bunch of juniors under an agile methodology and telling them what's right or not only creates friction and useless rituals, you need experienced people (PMs, EMs and at least one engineer) to cut the bullshit and stick to the points that matter: collaboration, communication, re-assessment of priorities, etc.

pydry2• 4 years ago

It's vague as hell though. Everybody from the agile consultants to the CEO down to the lowliest engineer projected their desires and intrinsic meaning on to the agile manifesto.

If you ask 10 people what agile "really" means you'll get 10 different answers, many quite incompatible with each other.

ericbarrett• 4 years ago

Fully agree with you, Agile as first proposed was like opening a window in a house in the desert in July; such a breath of fresh air. In the early 2000s you were ahead of the curve if you could manage a point release every six months. Saying "continuous development" would have gotten you laughed out of the building.

throwaway65326• 4 years ago

Agile as per the Agile Manifesto isn't bad. At all.

If you actually read it it really is just a bunch of words that doesn't specify much of anything at all. To me it reads as something incredibly vague and hand-wavy and how on earth modern Agile has been inferred out of the text of the agile manifesto is somewhat beyond me.

wodenokoto• 4 years ago

I felt like the justification for saying that agile was not good for underrepresented groups came very late, but it is there and it actually is not a bad argument.

We’ve long known that eliminating bureaucracy, hierarchy, and documentation feels great, until you’re the person who needs rules for protection.

I think the article eventually makes good arguments for this, at least if you are willing to say that code review and pair programming are things that spring out of the spirit of the manifesto.

Basically they author is claiming that the goal of the agile manifesto is to remove oversight from developers so that they can focus on "the project". The downside is that you are losing oversight of bad behaviour, which gives space for work-place bullying. (and the other downside argued is that developers, who are also stakeholders, do not have a voice in questioning the utility of the project overall).

r_c_a_d2• 4 years ago

I never read it as "removing oversight", more like "removing excessive admin". And it explicitly says to value "individuals and interactions". Part of that means you need a facilitator in retrospectives who can detect conflicts between individuals which may not be being addressed in that forum - and ensure that they are addressed elsewhere.

richliss• 4 years ago

95% of articles on Agile are incorrect and this is no different.

1. Mike (Miguel) Beedle was latino.

2. "Al Tenhundfeld, one of the manifesto’s authors" - NOPE.

I could go on but what's the point.

If you want to learn about agile I recommend reading Alistair Cockburn, James Coplien and Craig Larman. They're old school and know the history of this stuff amazingly well.

bitwize3• 4 years ago

We are in an era of mythologizing that old white men are at the root of all problems, so thinkpieces like this have to mention old white men as a matter of course.

That said, however, the article is correct to assert that Agile, as implemented in real organizations, addresses the corporation's needs and only those needs. Otherwise, it wouldn't have been adopted at all, but dismissed as the snivelings of whiny insubordinate programmers who forgot just who signs the checks.

throwaway65326• 4 years ago

We are in an era of mythologizing that old white men are at the root of all problems, so thinkpieces like this have to mention old white men as a matter of course.

That we've gotten to this point must also be old white men's fault.

eesmith3• 4 years ago

Of the 9 previous times this was posted this month, here are the five with comments, and either a pull quote or my summary of the discussion.

1 comment at https://news.ycombinator.com/item?id=31299834 ("The way it is described in the article highlights a subtle problem - the ability to change requirements as you go", followed by a discussion of that tangent.)

1 comment at https://news.ycombinator.com/item?id=31145030 ("doesn't match my understanding of the history")

2 comments at https://news.ycombinator.com/item?id=30988096 (both concerning the high number of re-posts)

1 comment at https://news.ycombinator.com/item?id=30967432 ("Neat historical overview followed by bizzare conclusions")

1 comment at https://news.ycombinator.com/item?id=30963468 ("Sigh. Another article on "Agile" that makes it read like "SCRUM" == Agile (Sprint, Stand-Up are Scrum terms, not Agile).")

drewcoo• 4 years ago

So you're saying we favor frequent submissions over in-depth conversations?

Hey - that's not in the manifesto!

ellen364• 4 years ago

Interesting idea from the article: that Agile methods “compartmentalize features” and risk creating “an assembly line in which no one, at any point, takes full responsibility for what the team has created”.

The author meant responsibility for the product’s impact on society, I think, but it could also apply to responsibility for technical decisions.

Jensson• 4 years ago

Sounds like ouija board driven design.

mnd999• 4 years ago

I haven't read the article because TLDR; but agile, particularly in it's earlier incarnations like XP is the least bad methodology I've used for most of the problems I've worked on in non-tech firms and small tech firms. I never worked in a FAANG or a Microsoft so at that scale things may be different.

The usual rules apply, there is no silver bullet, nothing is all bad. Pick the most sensible process for the job and adapt as necessary.

mikewarot• 4 years ago

TLDR

Agile was a response to management practices that didn't respect the profession of software development.

Management didn't get the joke when the Waterfall model was used to show exactly what was wrong with those management practices.

Agile was a response to the increasing unrealistic expectations of management, and the widespread (and highly predictable) failure of the Waterfall model.

Somewhere along the line, women were pushed out of the mainstream of programming, and became a minority.

The men who started the Agile movement were a bit creepy, and those in charge now have acknowledged past sins, at least somewhat.

Consultants and Trade shows have commodified the dissent that was Agile programming, slowly turning it insane by forcing it to be a Waterfall again.

As always, people who make powerful tools share some responsibility of how they are used.

Dave3of52• 4 years ago

Just as a note to the author what you saw the engineers do was SCRUM not agile. Most argue that SCRUM is agile I'm not 100% sure of that.

SCRUM was a framework that one of the original agile proponents built to make money off developers. It gained popularity not from engineers but from project managers and the like which meant suddenly every project started to use SCRUM. The main reason was that they sold a course which gave a certificate and if there something that a project manager like more than micromanaging it's certification.

dgb233• 4 years ago

Why do you think that is?

sirmarksalot• 4 years ago

I think the Agile Manifesto is an exercise in subversively asking management for a very reasonable-sounding thing that is completely impossible: autonomy. Every line poses a threat to a well-known corporate best practice, and the natural reaction is to co-opt the message and bring the items in the left column back into the right column.

In the case of Scrum, it's that "individuals and interactions over tools and processes" line. Scrum did start out as a popular Agile methodology among developers, but because it was the most prescriptive and process-oriented of the methodologies, it got adopted and twisted by management and turned into a tool to reassert authority. Now it reasserts that tools and processes are, in fact, more important than individuals and interactions, but this time with the "Agile" label.

I'd love to try working on a Scrum project that doesn't rely on JIRA. Just notes on a corkboard with a quick "need anything?" conversation in the morning.

a_imho• 4 years ago

Well, the first point of Agile Manifesto says

Individuals and interactions over processes and tools

and I would argue Scrum is a process and a rather heavyweight one. Thinking in terms of Scrum instead of your team/members and their preferences is getting it backwards.

Juliate4• 4 years ago

Indeed, but in the industry, I've seen no company that embraced "agile" without embracing some predefined framework (scrum, kanban, xp) to "apply it".

I've yet to find a growing/grown company that still embrace chaos as a start to find and nurture their own agile working methods.

It's very much like how companies implement "innovation strategies", being oblivious to the fact that innovation starts with rebellion against the very structure in which it appears.

Uncertainty is not something people embrace easily.

azangru6• 4 years ago

Does this quote from the article:

They were all working hard, that was clear, but there was a fatal flaw: no one really knew what the project was ultimately supposed to look like, or exactly what purpose it was supposed to serve.

sound like scrum though? No product owner, who would have a vision for what the product's purpose was or what it was supposed to eventually look like. No product backlog outlining a path to get to the vision. No sprint reviews with the product owner and stakeholders to inspect whether the product is on the right track. At least not that I saw in the article while quickly skimming through.

What she is describing is some techniques picked from some agile methodologies; but lacking the proper spirit of the game.

sgt1014• 4 years ago

no true scotsperson?

Cthulhu_• 4 years ago

I agree that SCRUM is a framework built on top of agile, but I think it's a valuable handhold to start off with, because agile itself is too free and open to interpretation. I don't like the huge agile training, certification and consultancy market though, even though I've met some really well-versed project management people that operate in that market.

MattPalmer10869• 4 years ago

Many developers embraced scrum when it first appeared. It was a lot more developer focused than the pseudo waterfall type practices that existed previously.

Can't code until the architect has written the spec. The deadline estimated from a Gantt chart by the PM. No feedback from stakeholders during development or the ability to alter requirements as they become better understood. So usually delivering code late that doesn't actually meet the customer's needs.

Scrum was better.

I hear it is now horribly abused (I'm out of the game now) and that's a shame.

EDIT: I saw the most resistance to Scrum from project managers themselves, who did not understand a world without Gantt charts, or changing requirements as you went along.

bdefore3• 4 years ago

Waterfall is a mostly manufactured pejorative by proponents of new processes. In what form it did exist, its practices were far from ubiquitous and mostly limited to government or very large corporations. Scrum arrived to slay a mythical beast.

Dave3of52• 4 years ago

I suspect with your comments you've been out the game a while. Most devs that I interact with hate SCRUM. Your Gantt chart is still there btw it's just not shown to devs but product are still creating all that but overlaying SCRUM over it.

Your 6 month project now has to be finished every 2 weeks now working shipped bug free in 2 whole weeks. The scope has been reduced for sure but now everything you do has to be in 2-3days at tops.

kkfx• 4 years ago

Honestly SCRUM is an abomination, a tentative to adapt a mass-production technique (Toyoda Kanban for mechanical industry of the '900) to R&D/intellectual activities.

The abomination is refusing to understand that creative activities can't be done well in constrained setups, while mass-production work very well with a certain rigidity. Toyoda NEVER tried to make Toyota R&D a kanban activity, just the factory part was kanban. Modern neoliberal managers have decided that the same technique they like can scale, while it can't. As a result we see a kind of quality-rise in listing terms, while the final outcome is crapwares monsters no one can understand not master, witch a big batch of side techniques to keep them artificially alive to a certain extent.

Classic sw development have flaws, Agile manifesto was a tentative to mitigate them, but have not much to do with SCRUM nor with what we call agile today.

kps• 4 years ago

Scrum is collectivised micromanagement.

jeffbee• 4 years ago

It feels like they could have just called Agile "The True Scotsman" to short circuit both sides of this endless repetitive and predictable argument.

jghn2• 4 years ago

One thing I've noticed is POs are almost always from the Product Management side. It makes sense, they both have Product in their name. I think this is part of the problem and that at the scrum team level the PO should be an engineer.

The issue is that the granularity of a system that should have product management is often larger than the granularity of a single scrum team. You can have several teams working together on a cohesive product. Yet often each team will have it's own product manager acting as PO. This leads to a lot of unnecessary horse trading and suboptimal results.

luciusdomitius2• 4 years ago

"long-haired weirdos who hung around university labs after hours" How is someone's appearance relevat to the management technique? I've never seen it as a criteria.

https://www.linkedin.com/in/miriamposner/ - OK, now it is clear - the author has "immense" experience in both management and software engineering and that's why she tries to describe the situation using the knowledge she has available to herself. What a miserable failure though.

The real question is why people with actual experience in both would read such an article...

msh2• 4 years ago

The whole quote makes your clip sound different: Some of the most advanced programming, as Clive Thompson notes in his 2019 book Coders, was pushed forward by long-haired weirdos who hung around university labs after hours, hackers who considered themselves as much artisans as logicians.

And it was not said in relation to agile.

luciusdomitius2• 4 years ago

What does Clive Thompson have to do with software engineering excellence? Could you send me his github.

Judging people by their appearance is extremely superficial and guaranteed to bring you tons of problems in any more complex context.

jillesvangurp• 4 years ago

The best characterization of agile is that it's an adjective, not a noun. It's why it has become so completely meaningless. Because everybody wants to do things with at least some pretense of agility. And often it's the people proclaiming their agility the most that you need to scrutinize the hardest when it comes to their proclaimed agility. As soon as you start talking about agile as if it is a noun, you are effectively bullshitting. It's not a thing (literally). It never was.

The software crisis, is the notion that software is hard to produce on a schedule. People indeed noticed that early on before there even were compilers. Hidden in that notion is a thing that is problematic with software. In order to even have a timeline for "it" to be delivered, you'd have to specify what "it" is. That's a notion that is quite hard. Often such specifications are a combination of wrong, incomplete, and not very useful. And it leads you to an engineering mentality where specifying the thing up front in excruciating detail becomes a mandatory thing. If there's one thing we can learn from the last seven decades of software engineering it is that that is indeed extremely hard and probably futile with the exception of a few domains where anything less is just not acceptable.

The notion of waterfall, which was popular with people that did not read the original paper by Royce (he does not mention the word waterfall but is often blamed for inventing the methodology) in full. Waterfall is the notion of specifying requirements upfront, designing the software, and then implementing it. Royce actually called out in his paper that going through this process multiple times yields better results. So, the key paper on waterfall actually suggests we might want to "iterate". Still worth a read more than 50 years later. Among all the agile nonsense, this is actually a good paper. But you have to read it in context.

You might think of writing software as the actual process of specifying what the software does. The best specification is the source code you produce. It's executable and complete. Whatever it specifies, it does. You might be unhappy with the level of detail but it is a specification.

The contribution of the agile manifesto was acknowledging that reality and the notion that it is more productive to talk about modifying this specification than to come up with some meta specification. You have to build in some moments of reflection and adjustment. So most of the time you are actually specifying changes relative to the existing system rather than the entire system. The only moment that is not true is when you have not written any software yet.

Specifying software changes is what an issue tracker is used for. Bugzilla was one of the first widely used ones and it launched a few years before the agile manifesto got penned down. The insight at the time was a that a good issue tracker (and maybe a wiki) were really all the tools you needed to orchestrate software development. All the rest (scrum, kanban, sprints, planning/estimation meetings, velocity, etc.) was just retrofitted around that notion.

That's it. If you can wrap your head around iterating, you are agile, just like everybody else. The smallest possible iteration is: write an issue, write some code, create a pull request, wait for someone to merge it. Pull requests are great. Most modern software development features multiple pull requests being worked on independent from each other at the same time. Of course the notion of concurrent iterations that aren't coordinated centrally makes the heads of most scrum proponents explode.

There's a lot of great software that gets produced by large organizations, or even distributed organizations (like open source projects), seemingly without a lot of ceremony (scrum or otherwise). Open source is basically software development stripped to its bare essentials. There are no estimation sessions. No sprint plannings. No sprints even. Work happens concurrently with very little central coordination. All there is is an issue tracker, a code repository and exclusive access for those allowed to merge pull requests. Some great software gets produced that way. And I don't think it's accidental either. Some projects seem very consistent with maintaining a predictable frequency of releasing, high quality standards, etc. In fact they put many corporate "agile" teams to shame. Mediocrity is the norm, not the exception in our industry. The software crisis still exists; but mostly in supposedly agile teams still insisting on disguising waterfall as agile. The louder they talk about being agile, the less they generally are.

dgb233• 4 years ago

If a company sets a goal of boosting user engagement, Agile is designed to get developers working single-mindedly toward that goal—not arguing with managers about whether, for example, it’s a good idea to show people content that inflames their prejudices.

Pretty sure the author mixes things up in an attempt to create a narrative.

azangru6• 4 years ago

Good point. In the pre-agile world, developers would work just as single-mindedly toward a goal, prodded by a manager; only after lots of time has been spent on coming up with a plan for the product.

langsoul-com• 4 years ago

Agile, these days, seems more like waterfall every 2 weeks.

dazzawazza• 4 years ago

The tears laughter, pain and despair in the retro are the waterfall.

baud147258• 4 years ago

seems like our situation. Though the head honchos have also imposed SAFE (scaled "agile" framework), so we get an even bigger waterfall with all teams together every 4 months, which takes each time two weeks (one week preparation + one week of meetings)

Jtsummers• 4 years ago

Waterfall (as a bad management practice, which sadly is still practiced) is bad when you're trying to run a large scale project and don't have feedback (because Waterfall does not include feedback). Waterfall-the-real-thing has no feedback loops built-in or severely delayed feedback (months of dev followed by months of testing, I'm watching my colleagues across the hall repeat this mistake right now, they "finished" dev last week but it won't be installed for the testers until the fall and they've started developing their next version of the system...morons). If you actually have biweekly feedback, you don't have Waterfall.

Planning and then executing on that plan in short increments is reasonable. Planning and executing on that plan over months or years, stupidity and insanity.

alkonaut3• 4 years ago

Agile (Scrum rather, but this article appears to be about implementing agile through scrum) trades one set of problems of software development, for another set of problems. That needs to be understood I think. Some times when agile is treated like dogma and religion this somehow gets lost. It's a tradeoff. It's popular because this new set of problems is deemed to be a better set of problems to have than almost all other sets of problems you'll end up with.

Switch to waterfall, and you'll have problems with that. Try whatever process, and there will be a set of problems that comes from it. Scrum is a flawed process, but there are many flawed processes. A lot of scrum-haters (understandably) argue against scrum, but what I'm missing is the presentation of the alternatives. We are going to be miserable and stuck doing scrum for the coming century unless we can formulate and demonstrate good alternatives. And if we don't want the processes to be upper-management centric, we better formulate the alternative as developers.

The places where I have seen scrum function well (teams delivering features without burnout, without losing grip of the big picture because of constant micro-development etc.) are in teams that are high performing and highly functioning. But of course those teams would have been well functioning and productive with any process.

The allure of scrum (in my opinion) is that it makes teams that would likely have delivered nothing, deliver something, or at least not waste time building the wrong thing. It's a way for management to increase the transparency and lower the risk of software development with poor software developers and/or bad teams. And that, to be honest, isn't something to be scoffed at. Throwing a poor process onto a bad team means you'll get miserable developers and risk having nothing produced. Agile means miserable developers and hopefully some transparency too. Not great, not terrible.

Juliate4• 4 years ago

It's a way for management to increase the transparency and lower the risk of software development with poor software developers and/or bad teams.

So, it's kind of an "observability tool for teams' production process management"? (to reframe it in a more contemporary fashion)

Edit: I guess we're headed to having pulse, blood pressure, VOmax, weight, daily exercise and so... metrics also displayed along mood and burndown charts. :p

alkonaut3• 4 years ago

So, it's kind of an "observability tool for teams' production process management"?

Absolutely, I think thats literally on the label and not in the fine print of the Scrum tin.

azangru6• 4 years ago

Agile means miserable developers and hopefully some transparency too.

Why are developers who work in an agile environment miserable? If you take scrum, doesn't it include retrospectives, with the express purpose of allowing developers to reflect on their misery and to gradually improve their condition?

BlargMcLarg3• 4 years ago

Retrospectives open the ability to provide feedback. That doesn't mean anything will happen with that feedback. That's before considering social theory and tact, where it becomes obvious being rebellious is generally a bad move socially unless you're already willing to leave, are in a position of power over others or the crowd is willing to take your side. Meanwhile these "agile" environments tend to push maximum social participation from developers at the benefit of management.

"No true Scrum/agile" aside, this is the reality of many. The fundamental problem still lays in the power difference between ICs and upper management. Even if it is "inefficient", so are many things done by modern corporates. Showing something can be done more efficiently isn't a guarantee the company will become leaner.

Juliate4• 4 years ago

You may only improve what you can influence, within your agility bubble: does it stop at your team, or does it embrace the whole company structure.

alkonaut3• 4 years ago

Why are developers who work in an agile environment miserable?

Yes. And yet that often doesn't happen. And that's where one might argue "well then you aren't doing it right", but that's precisely the No True Scotsman argument.

I see teams be miserable, have all the power to do something about it, including buy-in from management, yet they remain miserable. A key feature of any alternative methodology and organization should contain this: It's the role of management to ensure teams aren't miserable. Merely giving teams the tools to do so doesn't make it happen.

bryanrasmussen• 4 years ago

Why are developers who work in an agile environment miserable? If you take scrum, doesn't it include retrospectives, with the express purpose of allowing developers to reflect on their misery and to gradually improve their condition?

first thing to improve, get rid of retrospectives

second thing to improve, if having retrospectives have them in such a way that nobody feels like they have to come up with some things you can improve, keep doing etc. etc. every two-three weeks.

p0nce2• 4 years ago

Because your average software company has power captured by management (instead of designers or producers), workers can't speak to users as they wish, making it a blind sprint into nothing, and the added value of making successful software is going to be captured by the top of the hierarchy, who is convinced to have the best ideas.

spaetzleesser4• 4 years ago

From my experience feedback from retrospectives rarely gets acted on. You can make changes that are isolated to your team but you quickly reach a point where other functions need to change their process. To make these changes you need support from senior management but I have never seen anything happen besides words.

zippergz• 4 years ago

It's another meeting on my calendar potentially interrupting whatever deep focus I was in. Alongside standup, sprint planning, demo, sprint review, backlog grooming, and whatever other non-agile meetings there are.

It's an hour plus of sitting there mostly listening to other people complain about stuff, which bums me out even if I was otherwise generally happy.

It is a ceremony without actual purpose because usually the deeper things that people bring up are not within that team's power to control.

It gives leadership an excuse for not fixing things or listening because retro is supposed to be the outlet for gripes about the process (despite everyone knowing that major changes never happen coming out of retro).

ilaksh• 4 years ago

The manifesto quote was accurate I believe but ascribing all of the management BS and everything else to "agile" was ridiculous.

The really successful software projects I have been involved with had very reasonable requirements. In that they was not a long list of them and the core problems had plenty of time allocated.

That's the first key in my mind. Limit the scope. Overall. Another thing is to close feedback loops in terms of making sure that there are actually users using releases.

The number one biggest thing that has helped me is to avoid having places for managers or users to add an infinite list of features they want. I usually only need one or two things to work on every 3-5 days. We discuss every thing directly in text (or sometimes voice) chat based on the users testing the working application as it progresses.

I think this is also best for small teams, although recently I have done mainly solo projects.

I know one of the ideas with feature/issue trackers is that there are too many things to keep tabs on without them. But it seems like for a lot of projects, having the tracker means people fail to communicate directly when necessary, create busywork, and do not focus on priorities enough. If there is so much stuff going on that it actually can't fit in the chat room then maybe that's too much stuff.

Obviously things are going to be different for a large team handling something really extensive like Google Chrome, which is actually a full overlay operating system.

rswail• 4 years ago

OK, here's my take, based on 30 years of being a programmer and "manager" and "architect" and other position buzzwords.

I hate the title software "engineer" because engineering is a set of practices and processes that software creation doesn't come close to complying with.

The primary problem is the feature backlog. It's always been that, whether it was called the "Requirements" or the "Functional Requirements" or whatever.

Working out what the product of the effort has to do is hard. It involves understanding the actual end users (not the managers of the process/product), the process/product itself, and the context and environment in which it will operate.

The stakeholders are multiple and on multiple levels. There are managers and auditors that need visibility of the ongoing operations, there are users and consumers that need the actual input or output.

I've never met a "Product Owner" that actually understands the product they are building or where and how it is expected to be deployed and operate. They rarely filter out unnecessary or dangerous "features", but just add them into the overall bucket without knowing where to draw the boundaries.

The "solution" is multi-faceted, but it needs to be directed at this area.

Domain Driven Design is one approach, actually engaging with the actual end users is vital.

Sometimes, ignoring a manager's desire (particularly for "dashboards" and "reports") is very important to keeping the feature backlog manageable.

Understanding the context/environment is also vital. Does this product deal with individual's personal information? How does SOC affect it? Is the process required to be auditable? If it is, what evidence is required for the auditors?

None of this is "software" per se or even "programming", or, even worse, "engineering". It's process and product and understanding what is being built.

spaetzleesser4• 4 years ago

I agree about product owners. I have never seen one that could really guide a project from start to finish. They either don’t understand the tech sufficiently, or they don’t understand the business case or they aren’t engaged enough.

sophrocyne• 4 years ago

Understanding the tech sufficiently and effectively rallying the team around the actual business/user justification for work are literally the job of a product manager who owns tech product development.

While they may not be common, actual product managers do exist. There are a lot of incompetent ones, to be sure, but there are also a number of incompetent engineers.

imbnwa• 4 years ago

My limited experience still has me questioning what most PO/PM types are doing all day.

brianmcc• 4 years ago

This is an interesting short series of articles from 2021, interviewing 17 "real" engineers (the proper build-a-bridge type) who transitioned to software engineering to explore whether they consider software work really engineering.

And 15 of the 17 agreed that yes software engineering really is engineering.

https://www.hillelwayne.com/post/are-we-really-engineers/

vannevar2• 4 years ago

I've never met a "Product Owner" that actually understands the product they are building...

This is sad. It's like a professional football player saying they never played for a coach who understood the objective of football. If that's truly the case, no amount of software development best practices or skill will save you from failure.

I have seen product owners who understand the product, and clearly those people exist, as there are successful products in the market. But if you've spent your career playing for the software equivalent of the Detroit Lions, I can see how the industry could appear pretty bleak.

smitty1e• 4 years ago

The manifesto’s authors may have looked like textbook engineers, in button-downs with cell-phone holsters, but “a bigger group of organizational anarchists would be hard to find,” according to Jim Highsmith.

The word "anarchists" fits in its purest sense of tossing the org chart. Yet its common meaning of "bomb thrower" hardly applies.

I typically describe Agile as "A small, stable cadre of ninjas getting it done."

Projects wind up with too much scope, a revolving-door staff, and a dearth of wizards to cut through the technical thicket.

The discussion of "tech debt vs. 'green field' projects" minimizes the threat of legacy code/data to any agile effort.

We need to talk about a "software archaeologist" specialty where one tries to plow through a mound of garbage left by the predecessors and seeks to figure out what prescriptions their therapists had written for them.

throwaway65326• 4 years ago

We need to talk about a "software archaeologist" specialty where one tries to plow through a mound of garbage left by the predecessors and seeks to figure out what prescriptions their therapists had written for them

Reminds me of one of my favorite quotes about software/computer systems - "we build our computer systems like we build our cities - over time, without a plan, on top of ruins."

taylodl2• 4 years ago

I've been developing software for 40 years and here's my unsolicited observations:

- Projects always take longer than anyone expects, even when accounting for projects taking longer than anyone expects (Hofstadter's Law)

- The project architecture will reflect the organizational structure of the entity creating the software (Conway's Law)

- The above two points are often presented as jokes but they are not, they're both actually true

- You cannot force two people who do not like one another to pair program

- Pair programming should never be forced on developers, let developers decide when to pair and whom to pair with - this is the autonomy developers seek

- Code reviews are a waste of time and present a great opportunity to destroy team morale

- Software development attracts a lot of assholes and prima donnas who all believe they are the fabled 10x developer

- It is just as important to understand the why of a feature as it is to understand the what - developers need to understand why a feature is relevant and what problem it's solving for users to be able to best implement it

- Invest in automated unit testing, application build, packaging, configuration and deployment. This is your best defense against Hofstadter's Law.

- Some teams have a knack for failure regardless of the software development methodology being utilized. The same is true for application frameworks and development languages.

- Some teams have a knack for success regardless of the software development methodology being utilized. They can also use the "bad" application frameworks and development languages and still be successful.

- Mutual respect and everyone pulling their weight are the best predictors of a team's success

- Just because someone isn't working out well on one team doesn't mean they won't excel on another team. Some people will excel regardless of the team they're placed on. There are also a few, very few, who can't seem to excel anywhere.

- Some people aren't cut out for development and since 1960 we've been trying to figure out who those people are and dissuade them from entering the profession - all to no avail. This is a sink or swim profession and it's tragic to watch people sink - especially when you can't put into words why they're sinking and how to start swimming.

- Outsiders believe developers are anti-social and don't realize modern development is a team sport. The best predictor of success is how well you were able to play team sports. If you always had lots of problems with your team mates then you'll probably have lots of problems with the members of your development team. Of course this isn't 100% accurate, but it's a good predictor. Those playing well in team sports, play in bands, or have been involved in dramatic productions tend to work well. Basically any activity having a group of people work together towards a common goal. Loners do not work well on development teams - they find success on skunk work projects.

- Finally, management is key. If you fundamentally disagree with your management then it's time to move on. There's no need to trash the management when you leave, all you need to say is you don't feel you're aligned with their goals and priorities. Some people may like the management, others may not. Don't take it personally.

digisign• 4 years ago

Generally agree, but disagree with this one:

    - Code reviews are a waste of time and present a great opportunity to 
      destroy team morale

CR has prevented an uncountable number of issues in our large codebase with junior devs. It doesn't have to reduce morale if used as a teaching moment (growth mindset) instead of opportunity for punishment. Imagine a sports team without a coach. "Just have experts (only) work for you," isn't a solution for most either.

p0nce2• 4 years ago

Code reviews are a waste of time and present a great opportunity to destroy team morale

Great post, but we would have to come to agree on that one somehow, because in Capers Jones books, code reviews are measured to be the #1 most effective method at improving quality. I'd agree they are very easy to do badly, and often overused.

throwaway65326• 4 years ago

This is an amazing list, though I'm curious given how accurate the rest of this list is how you drew the conclusion that code reviews are a waste of time?

taylodl2• 4 years ago

Three people commented on my calling code reviews a waste of time. To kick off my discussion, I should reveal that I stopped doing code reviews 20 years ago. That actually turns out to be significant. Here's why.

For the first 20 years of my career I worked at 5 different places and so have some experience with different development cultures. They all did code reviews and they all had the same flaw - they focused almost exclusively on all the wrong things. There would be endless arguments about indentation, braces placement, variable naming and so on. A little bit of the time was spent on what actually matters - the implementation. C++ (which was hugely popular in the 90's) made this even worse - now people could argue about whether a function should be overridable (virtual), whether members should be private, and on and on and on and on. After a few years of working on our own internal framework and working with commercial frameworks it dawned on me that nobody got it right. Meanwhile all these inane arguments led to time being wasted, projects missing their dates, team members arguing over stupid crap - ugh! I saw this with multiple teams across multiple organizations. Different people, same problem.

So what happened 20 years ago? That's when what we now call CI/CD tools first started appearing (remember Cruise Control?). With that we saw the advent of automated unit test frameworks, tools that could check the code against coding guidelines, and tools that could measure the amount of CRAP (Change Risk Analysis and Predictions, remember Crap4j?) and assess the security risk the code imposes. All of this is automated. I saw that automating the code reviews could get rid of a lot of the negativity, be objective, and focus on the implementation and quality of the code. This is the way.

When code is pushed to the repo I have evidence of it's quality: I know it conforms to our coding conventions (which I'm not a big believer in, but some people like to die on that hill), it's been tested (we require 40% test coverage and getter/setter tests don't count), it's CRAP score has been assessed, and it's had a security scan looking for vulnerabilities. All automated.

When working with junior developers they get immediate feedback on the quality of their code and can fix it. If they don't understand how to fix it then they reach out to one of us and we can help mentor them as to what is being reported and what the actual issue is. It's all very positive.

What I should have said then was manual code reviews are a waste of time - you want to automate that activity! I don't ever want to see a code review activity scheduled on my calendar!

TameAntelope5• 4 years ago

Okay, so what's the alternative? I read most of this article and scanned the rest, but could not find any proposal.

Should we just stop developing software? Should we go back to Waterfall?

I know every person reading this could write a dozen pages full of suggestions for how to run a software team, but I really wish the author herself had spent some time suggesting what software engineers should be doing instead of "agile", because that, to me, would be infinitely more interesting.

lbriner• 4 years ago

Similar to you. There were several hints that the author didn't really understand software. The problem with waterfall wasn't that you had to do tasks in order, it was that you needed to know mostly everything in advance to get rid of issues at design time so they didn't bite you during coding.

Also, the rumblings of agile don't really point to anything to do with agile fundamentally, just the fact that people don't quite understand it or that management doesn't wholeheartedly embrace the mess.

"Agile meant that the team didn't really know what they were producing" - that is a Product Owner issue 100%.

So yeah, nothing's perfect but it's probably the least worse of the options we have!

chris11• 4 years ago

Should we go back to Waterfall?

Waterfall seems like a strawman to me. Has the tech industry ever used it? The first reference I could find for it was back in the 70s as what not to do. I don't think waterfall should be used as a defense to scrum.

As for alternatives, you can start with giving engineering teams autonomy and ownership of their work. Gergely Orosz has a good article on project management in tech.

http://www-scf.usc.edu/~csci201/lectures/Lecture11/royce1970...

https://newsletter.pragmaticengineer.com/p/project-managemen...

TameAntelope5• 4 years ago

Every time you have a "Gather Requirements/"Design"/"Develop"/"Test"/"Deploy" set of stages, you're using waterfall. Note the absence of "collect feedback from customers".

Waterfall is in no way a "strawman", it's very real, and many shops who claim to be "agile" are still using it today, not to mention the near-ubiquity of its presence in software shops for years prior to agile taking over.

cuddlybacon• 4 years ago

Waterfall seems like a strawman to me.

Waterfall definitely existed. I've been at places that worked that way. I think Agile (and corruptions of) have replaced waterfall well enough that people no longer think it used to exist.

karaterobot• 4 years ago

This is a good point. The question isn't whether Agile is perfect (it is certainly not), but whether it's the best option we've got. If it is, then we should continue to use it. If it's not, then we should switch. I would be really interested in learning what the next attempt at a better methodology looks like.

trasz2• 4 years ago

One alternative would be to move the vast majority of managers where they belong - flipping burgers in McDonald’s - and instead spend this money on people with some idea of how computers work. The workflow will evolve naturally.

Ever wondered why so many Open Source projects deliver software vastly superior to commercial alternatives? It’s because in Open Source, non-competent folks don’t tell competent ones what to do.

neilk• 4 years ago

Open source has succeeded at creating common infrastructure, where technical vision matters most. Sometimes.

Compared with the vast, vast array of software we both need and use, open source has failed. Precisely because there’s a gap between what the developer can envision and what is needed. Hence product owners. And a gap between the resources of time and money available and what a single developer can do in their spare time. Hence management.

PS: I also think you should re-examine your disdain for “burger flippers”. Even if it’s a low-autonomy job with much repetitive labor, there are aspects – like dealing with customers under pressure – which might break the average software developer. And people are much more than how they choose or are able to sell their labor. The general lesson here is that there’s usually a lot more to every job and worker than you think.

dandare• 4 years ago

Should we go back to Waterfall?

In many areas we are already doing waterfall, except it is a sin to call it that way and we must pretend we are doing agile.

Let me give you two examples: - Contractual requirements. If you have a contract with a customer to build X, no matter how much agile lingo you use, you are basically doing a waterfall. - Legal requirements. Same as above. For instance, there is very little flexibility in being GDPR compliant.

Xophmeister• 4 years ago

    Moreover, daily standups, billed as lightweight, low key check-ins,
    have become, for some workers, exercises in surveillance.
    Particularly when work is decomposed into small parts, workers feel
    an obligation to enumerate every task they’ve accomplished. There’s
    also pressure for every worker to justify their worth; they are,
    after all, employees, who need to be perceived as earning their
    salaries.

This is a great summary about how I feel about "agile": it's patronising and anxiety inducing. The degree to which is a function of the technical and social abilities of whoever's calling the shots -- the PM, SCRUM master, whatever -- but, of course, their steering is necessary, regardless. Leaders with strong technical chops and high emotional intelligence exist, but are rare, I think.

Similarly, this is why I don't believe that a fully egalitarian group of engineers would work, either: All skill and no direction doth not a project make. Besides, true egalitarianism doesn't really exist. Some will be stronger than others; some will have weaknesses that are complemented by colleagues (hence the value of diversity); some will be brazen, while others shy. Inevitably, a de facto leader will emerge. In my experience, often the loudest, rather than the most effective.

Building complex software is a social problem more than a technical one.

andrei_says_• 4 years ago

In his talk "Agile is Dead", Dave Thomas (one of the creators of the Agile Manifesto) speaks about how distorted and perverted the manifesto's intentions have been by the "Agile" industry.

Striving to be agile, flexible, adaptable is a very valid and useful principle.

"Agile" as a noun, the cult-ish process-training-industrial-complex is not at all agile.

Link to the part where he recalls the creation of the Manifesto https://youtu.be/a-BOSpxYJ9M?t=384

It is immediately followed by the difference between agile (adjective) and Agile (noun).

Agile IMO has been perverted by business and middle management to facilitate bean-counting.

throwaway_1928• 4 years ago

Standups are the new timesheets: a humiliating exercise in micromanagement.

pydry2• 4 years ago

This quite accurately characterizes the marxist concept of alienation.

I think it's essentially just the default state of working for a business. Any business.

camgunz• 4 years ago

My brother and I have developed a "the enemy was capitalism all along" refrain, and it's pretty wild how what started as facetious irony has slowly felt more and more correct.

vannevar2• 4 years ago

...exercises in surveillance...

Surveillance by whom? A standup consists of the team and a scrum master, who is usually on the team because most companies can't afford a dedicated one.

fnordpiglet• 4 years ago

I grew up in the industry during the 90’s in the Bay Area. What we considered agile is nothing like agile of today. The process people wormed their way in and made it super dorky and prescriptive. As a leader now I only encourage agile in my most junior or dysfunctional team. In my high functioning teams I guide them towards the less prescriptive more “agile” method of my youth, which was very heavy in the emphasis of you almost certainly know what’s next so do it and call together a standup when you’re blocked. I’d note that more powerful than agile is the trend towards asynchronous work and planning. Keeping a chat as an asynchronous standup where you daily post your thoughts on work and respond to others is a huge boon and completely replaces “stand ups,” which only really work in fully collocated teams. Every company I’ve worked at had folks all over the world working together, and trying to do a traditional agile standup at a fixed time is brutal on at least someone.

lamontcg• 4 years ago

I think the problem isn't really what is being done to manage software devs, but that managers are trying to squeeze more productivity out of software teams which are insufficiently staffed and where higher level strategy is bad and the teams are setup to fail. And largely the "poorly performing" software dev teams are just a symptom of the problem and you can't rub any level of management onto the team in order to fix it.

What I'd like to see is a methodology which produced an objective measure of how overloaded a software dev team was and which could be used to justify an increased headcount. I've never actually witnessed that come out of one of these processes.

nradov3• 4 years ago

What does overloaded even mean? An agile program team has a certain velocity, and will work on backlog items in order of priority. The business always has the option to expand the team in order to increase velocity (perhaps with some diminishing returns). But if the business is short of capital, or the whole program isn't very valuable, then it is often preferable to just let the backlog grow.

dasil003• 4 years ago

Your first paragraph rings true, but I have to disagree that the second is the solution. I’ve seen engineering management play this game to justify headcount without acknowledging the strategic or product definition problems and it just leads to more chaos and pressure on the team. If engineering doesn’t have both a voice at the leadership level (for whatever scope is in question) and sufficient biz/product chops to find the right 80/20 path and team/ownership structure then headcount is just more grist for the failed project mill.

cestith2• 4 years ago

One of the problems people are having is that so much is said against Waterfall instead of just in favor of Agile or its spinoffs. People emphasize that everything to do with Waterfall is bad, and generalize from large, detailed, inflexible architecture with no feedback loops being bad. They often end up, repeated by people trained to be Scrum Masters or some other management or administrative role who are not developers themselves, hearing about all preplanning being bad and evoking waterfall.

I've been told that gating one story behind another is waterfall. I've been told that having a design story is waterfall. I've been told that architecture documents that are eligible to be revised as the system is built but created before all the code is in place are waterfall. This sort of backlash taken to heart means that anything larger than a developer can do in two weeks is infeasible. This often happens when the team building the software is not considered among the stakeholders of the team's output.

There's a sharp focus on shippable improvements. Well, to me a plugin system is a shippable improvement even if there's no plugins for it yet. A data flow diagram from this week people can use to determine which parts to build next week is an improvement. This isn't a multi-year pre-planned development schedule. This is just having some idea how to integrate the software.

xkcd-sucks2• 4 years ago

I get the criticisms of "waterfall" style development, but lack of any careful thought or specification spanning a whole project leads to really terrible contracts and projects as some people (especially Enterprise) use "agile" == "nimble" == "license to rug-pull project scope" when there is significant inertia, and occasionally sunk cost, in these scope changes.

Case in point, I'm working in an engineering capacity with a "Big Mean Secret" pharma company on a contract that is essentially "Client will work with Vendor in an agile manner to <do something vague which could encompass everything or nothing>". The company I work for is nominally a "product" company, not a "services" company, and the contract specifies fixed payment terms and timeframe.

This morning, one of my teams finished implementing some stuff that was a good amount of effort, several moving parts, etc. It was explicitly called out as top priority, and its progress has been tracked by both parties over the past weeks in a shared Jira board.

A few hours later in a biweekly update meeting with said client, we mentioned having finished the feature. The client's project manager responded that actually the person who cared about that feature left the company, so it is no longer in scope. Then they erased it from the project tracking record, and in the same breath complained that so little progress had been made against the other objectives.

And a million other little papercuts like that. It seems that "agile" style process is best confined to small / internal teams re: the "how", but the high level "what" requires comprehensive elucidation from the beginning.

michael19992• 4 years ago

How would an upfront contract have changed that outcome? Wouldn't that now-absent person have demanded the same work in the up-front plan?

Or is your concern being able to bill for that work?

xkcd-sucks2• 4 years ago

Yes, broadly speaking the concern is about being able to bill for that work. I realize that this is more an issue of terribly worded contracts, but "agile" expectations seem to encourage these.

oxfordmale• 4 years ago

Waterfall as criticised by Agile followers is mostly a myth. It is a mythical monster designed to instill fear and guide developers along the righteous path of Agile. Of course enlightenment can only be reached with expensive Agile training consultants as you are doomed to fail if you walk alone in the valley of the shadow of project failure.

https://loufranco.com/blog/waterfall-is-a-myth

yibers• 4 years ago

A few hours later in a biweekly update meeting with said client, we mentioned having finished the feature. The client's project manager responded that actually the person who cared about that feature left the company, so it is no longer in scope. Then they erased it from the project tracking record, and in the same breath complained that so little progress had been made against the other objectives.

Actually, I don't see here an issue with doing "agile" or "waterfall". It's a communications issue. Either someone at "Big Mean Secret" company did not manage communications with the client properly, or the client didn't bother to communicate properly when given the opportunity.

Either way, using a "waterfall" methodology would not have helped here. It can be said that maybe "agile" "done right" would have solved this issue, but when is "agile" ever "done right"?

(The last sentence was an over generalization of course)

RayFrankenstein• 4 years ago

“Agile In Their Own Words”. A collection of negative social media comments regarding Agile.

https://github.com/rayfrankenstein/AITOW/blob/master/README....

magic_hamster• 4 years ago

In the profession built on designing the right solution for a business problem, it surprises me sometimes that businesses or colleagues consider one methodology (like Agile) to be a one-size-fits-all for software development projects.

There is a rule in the world: no single solution can be applied to all problems alike. One-size-fits-all is always, always a lie, no exceptions.

A software project is a problem that needs to be solved according to its needs. The skills required, the communication, infrastructure and metrics always differ.

A good manager will never fully subscribe to any methodology, but instead will be truly "agile", or actually flexible, drawing inspiration from different methodologies where it's needed, knowing their team very well, and recognizing what's needed for their team to get the job done in the best way possible.

gadders• 4 years ago

> I began to explore the history of Agile. What I discovered was a long-running wrestling match between what managers want software development to be and what it really is

For "managers", substitute "customers". It's the people paying that want to know the cost up front as much as possible and when things will be ready. The managers are just trying to hold everyone to that (as much as is possible).

specialist• 4 years ago

The peanut gallery reacting to these agile rants always contrast it with "waterfall", as if those two forms of dysfunction are the only two possibilities.

I must live in a sci-fi horror movie/simulation. Where I'm the only person who remembers actual mature proven repeatable project management methodologies. Like critical path, joint application development, and a dozen others. Whenever I mention this alternate reality, I'm dismissed as a lunatic.

Hello? Does anyone see me? Does anyone else remember Peter Drucker, PMI, and shipping software?

Agile and the long crisis of software

Related Stories