The Death of Agile [video]

http://www.thoughtworks.com/talks/the-death-of-agile

161 106 12 years ago

I really enjoyed this.

One detail that stuck out at me is that after Thomas makes the claim that the single metric that matters in software is how easy it is to change, he moves on to show a balancing robot.

Now he doesn't directly connect those concepts, instead he talks about the PID algorithm. But what struck me is that Moshe Feldenkrais defined good posture almost exactly in the same way that Thomas defines good software, although he is talking about humans rather than robots. Specifically, Feldenkrais says that good posture in a given context is that which makes it easiest to move into the next desired position (or a spectrum of potentially desirable next positions).

michaelfeathers• 12 years ago

In the early XP days, we spoke about code's equipoise.

cmdkeen• 12 years ago

"the claim that the single metric that matters in software is how easy it is to change"

Not if you have actual business people that you work for, then you're delivery ratio (things achieved per person/unit of time) is key. Because that delivers predictability and enabled planning. It lets you have discussions about what level of quality to deliver, staffing requirements, manage delivery expectations and plan releases.

The key thing is that it demonstrates the cost of change to them, so after being burned they come to appreciate the benefits of building in flexibility, but it also affects how they interact with the business. Delivery, and the burnup/down it allows, is the way we as developers can train product owners to predict the risk in what is coming and use their expertise to reduce it.

I've seen far more time saved, and lost, due to the business learning to ask the right questions, and raise the right stories, than due to building flexible software to handle changes. No matter how flexible your software is the really huge changes always bring massive pain - and as developers we don't have the domain knowledge to necessary to identify those whereas business people can.

slowmovintarget3• 12 years ago

Now if we can just get management to sit down and watch this...

I keep watching Cargo Cult "Agile" turn every project into a galley, developers at the oars with the scrum master merely manning the drum and shouting "row!" Two lashes for failing to update your task in the system, no matter that the work got done...

We really need to get back to the real measure of working software instead of "are we going to have a demo or not?"

pcurve• 12 years ago

I think we all need to watch this video, particularly business and project managers. I'm shocked at the complete lack of understanding of what agile is by virtually everybody working on any project these days.

wallflower• 12 years ago

For management, it's not really about the process du jour per se. It is about standardization, frameworks, and control. Read 'Seeing like a state'

From Ruby Rogues 184 RR

"JESSICA: Alright. So, I am going to echo one of Greg’s picks because it was on my list but for a different reason. ‘Seeing like a State’ is an amazing book. And I think it’s drastically changed the way I look at software, not for the same reason as Greg talked about but because it shows why what we do is hard. ‘Seeing like a State’ talks about all the subtleties of human systems and human interactions at the local context level. It talks about all the improvisation that everyone does on a day-to-day basis and how in real human communities, we’re constantly changing the system to adjust to a slightly different reality, to corner cases we hadn’t seen before but now we have. It’s shifting and it’s not well-defined. And suddenly it makes complete sense that the hardest part of software is figuring out what we want to do. That’s it. It’s a great book."

http://www.amazon.com/Seeing-like-State-Certain-Condition/dp...

myth_drannon• 12 years ago

Funny how one comment on HN can bring a relatively old and unknown book from obscurity to top spot on Amazon sales chart (it's a sub-section but still). Of course it's just an assumption that there is a correlation between.

UK-AL4• 12 years ago

"standardization, frameworks, and control" If management wants this than good.

In my experience its the constant change(I don't mind controlled change, but hidden change), constant scope creep, constant increasing of tasks, without increasing estimates. The constant being asked to "do things faster" without caring about quality, then being blamed about quality at a later date that makes things fail. Trying to implement every single feature, rather than focusing on a small set of useful features.

In a word its management that makes projects fail. You need a system to set constraints on managament so they have to make proper trade offs and don't expect everything in a small amount of time.

It's easy to manage if you expect everything can be done. What makes management hard is making the trade offs because you have limited resources.

If you implement a standard way of doing things, you can stop management wrecking projects.

Boss comes in and asks to change something? Well it has to go through the normal estimation process and the estimate increased etc

They can't ask you to cut corners, because it must go through the same quality procedures

etc

A lot of developers are complicit in this, because they have a super hero complex. They want to show off, and have unrealistic expectations of themselves. Management think they getting a great deal, but in the end the developer can't deliver. One of the parts of becoming a senior developer is to start to really know yourself. Then adjusting your systems to compensate.

One thing i like about SCRUM is the velocity technique, because uses past history to estimate rather than ego driven estimates.

Frameworks and standard techniques are there to protect developers much more than to protect managers.

Spooky23• 12 years ago

You made me laugh, but it is sad that agile devolved into a micro-management tool that empowers the project management kool-aid drinkers.

nradov2• 12 years ago

Try to see things from the manager's standpoint. Without good demos it's tough to get sales. Without sales the developers don't get paid. If you're working on a large piece of software with dozens of developers split into several teams and you don't update your tasks in the project management system how do you expect anyone outside your team to know whether your tasks are complete?

slowmovintarget3• 12 years ago

Scrum daily stand-ups, which we have, and Scrums-of-Scrums which we also have.

__Individuals and Interactions__ over Processes and Tools

vpeters25• 12 years ago

After we switched to TFS, our supervisor was increasingly pushing us to plan our sprints and update our burndown. This went on for a couple iterations until he realized there was no actual benefit to all that... then we switched to kanban.

I'm not saying everybody should switch to Kanban, but in our case it is just enough process to keep track of our progress while not getting in the way of getting stuff done.

trhway2• 12 years ago

Two lashes for failing to update your task in the system

my back is still hurting ( our BigCo had methodically tried to implement Scrum/Agile/Lean during 2012/13 which happened to be such a great failure. It seems that management is so shell-shocked by it that they haven't tried to push any new methodology during the whole last year. Well, and the management lost all credibility here too - any attempt at pushing any new methodology today would make everybody laughing in their face :)

"are we going to have a demo or not?"

almost two first weeks of a sprint almost whole organization is working on a demo for the previous sprint - that was beyond insane.

EdwardDiego3• 12 years ago

almost two first weeks of a sprint almost whole organization is working on a demo for the previous sprint - that was beyond insane.

Wow. You're right, that's beyond crazy.

What frustrates me is that demos can be very useful, if done right, but if you're doing them because "We do Scrum, here look at this certificate I got from Scrum Alliance saying that I'm a 'Certified ScrumMaster®', see?!" like the artefacts of Scrum are magical ceremonies that will cleanse you of your software sin...

For example, we create a small video demo of a feature story as part of the story development, and we make it available for our business team (who are on the opposite side of the world) immediately. This gives them a chance to flag the occasional oversight on our part sooner, and we can fix it sooner. It works really well for us in terms of ensuring quality, but also for keeping the business team aware of what we're doing.

But if it didn't, then we should iterate on our process (like bloody Scrum says to do) and stop doing them.

pmoriarty8• 12 years ago

Anyone have a direct link to the video?

I'd like to watch it, but it requires me to run some Flash plugin, which I'd like to avoid.

devmach• 12 years ago

It looks like they have html5 player too, which plays the following file

http://embed.wistia.com/deliveries/05b7d508543a1e8f7de01e27b... (900Mb, HD)

edit: Although, I don't know if they allow you to "Download" the video in sense of the licencing...

pmoriarty8• 12 years ago

I should have said that I didn't want to run any embedded players, Flash, HTML 5, Javascript, or whatever.

Thank you for the link!

tunesmith2• 12 years ago

For those that are allergic to snark like me, skip the first fifteen minutes - it doesn't start getting constructive until then.

tunesmith2• 12 years ago

For those downvoting, here's a quick summary:

4:00 - Middle-aged white guys at a conference to figure out how to sell fear by turning agile into a "noun"

5:00 - Make agile more difficult-sounding with lots of terms so you can manipulate companies out of time and money

6:00 - Sell unnecessary roles like "scrum-master" to further manipulate companies out of time and money

8:00 - Agile plot to mock developers who don’t “fall” for the previous manipulations.

But again, the tone changes after the first fifteen minutes, so I intended that comment for those who would watch the first few minutes and think that it's just someone invested in making fun of "Agile" the whole time.

pmoriarty8• 12 years ago

I don't know what you're talking about. The first part was the best part.

imanaccount247• 12 years ago

The part where he makes fun of the agile snake oil salesmen despite the fact that he was one of the biggest offenders?

rebelshrug5• 12 years ago

Not everything about agile software development is bad. Planning poker is a great team exercise if you replace the planning part with a deck of playing cards, poker chips and beer.

pmoriarty8• 12 years ago

I'd like to show this to my company, but I'm afraid I'll offend coworkers who make their money at the company "doing Agile" and I'll get lynched, not to mention fired.

rebelshrug5• 12 years ago

Fired for voicing an opinion? Sounds like a company worth leaving.

pmoriarty8• 12 years ago

Not just an opinion, but a sharp-tongued poke at a sacred cow. I would be singled out as negative, not a team player, a troublemaker. I'm welcome to have an opinion, as long as I don't make waves. Keep your head down and do your job is the motto. The "leaders" set the tune and you march. If they tell you to jump to Agile, all you're welcome to ask is "how high?"

calibraxis• 12 years ago

Most corporations will fire you for voicing certain (non-oppressive) opinions, since ideology is important. You see it in schools too, not to mention cults. You're allowed to get to the next rung by not seeing it, already having the right attitudes, or faking it. (Consider some meanings of "cultural fit".)

prostoalex• 12 years ago

In a lot of companies it's similar to voicing an opinion about an alternative religion in an area dominated by one "true" religion.

Not to mention the fact that various job titles (scrum master?) are tied to the existence of such process within the company and are not useful outside of the context.

capkutay2• 12 years ago

Not to sound lazy, but I'd like to make a humble request for a TL;DR? I can skim long papers, but you can't do that for long videos!

rebelshrug5• 12 years ago

here's Dave Thomas' blog post on the subject: http://pragdave.me/blog/2014/03/04/time-to-kill-agile/

capkutay2• 12 years ago

Thanks!

disputin• 12 years ago

The theatrical form of agile as it has been peddled with all of its prescribed practices is nonsense piled onto the idea of agility so that it appeals to managers, for sale by consultants, and that agility is important in its toned back form, without the annoying rules that have no regard for context.

I hope this means the death of scrum.

Toine• 12 years ago

I'm really surprised by this video.

I'm a junior developer who just got out of college. I knew the software industry isn't as mature as some other industries, but I kinda thought as a whole we were moving towards a more professionnal, more robust, less amateur industry.

This guy, if I heard correctly, participated in writing the Agile manifesto, which is basically software's 10 commandements written by the Agile gods. In this talk, he basically says : "Yea actually we were kinda wrong, you shouldn't listen to anyone telling you to do anything, just run free in the jungle and you'll be fine."

So after all those years of trying to find a solution to how to do software correctly, this is his answer? Why would you say something like that?

How do you teach people then? Where can i learn how to do my job correctly? The 4 steps he give cannot be taken seriously...can you imagine medical students being taught "yea just try something and see what happens, hopefully your patient doesn't die and you'll learn something for the next patient."

walshemj2• 12 years ago

Welcome to the real world where software is never as simple and clear cut as university problems.

UK-AL4• 12 years ago

Many engineering and operational jobs have fairly standard techniques. However in software no standard practice seems to form.

tdicola• 12 years ago

Agile came about somewhat as a reaction to huge processes like waterfall, RUP, etc. that were espoused by many books, conferences, etc. at the time as the best practice and 'right' way to build software. The point being made here is that the same thing is happening to agile now, a cargo cult mentality has set in that software needs to be 'agile' and agile is just buying the right certifications, reading the right books, going to the right conferences, etc. and high quality software will pop out on time at the end of your release. The reality is it's just not that easy to build software, there is no magic book, agile or otherwise, that you can blindly follow and expect good results.

Where can i learn how to do my job correctly?

That is a really heavy question to answer. I could try give an opinion but I've only been developing software for 10 years. Maybe in another 20 or 30 years I'll have something good to tell you. The fact is you just need to build things for many years and see what succeeds, what fails, and what you can learn from each success and failure.

pjmlp2• 12 years ago

Pretty much this.

I have been developing software for almost 30 years now.

Not everything was bad about waterfall, RUP, UML, Booch and related methods.

As not everything is good about every agile variant out there.

The secret, if any, is trying to find a good mix.

Which is very difficult given the constraints in team member skills, what the sales team promises to the customer, what the customer is willing to pay for, and what the managers what to ear and sell to the customers.

Specially when most companies treat developers as replaceable cogs.

InclinedPlane8• 12 years ago

Software is still an incredibly immature field. It's been getting better, much better, but it still has an incredibly long way to go. It's worth pointing out that software is also an incredibly difficult field.

thibauts2• 12 years ago

It looks difficult because we refuse to draw the obvious conclusions and take at face value what our day-to-day experiences teach us.

Let the master programmers (with substantial backing proof) lead the way and everything will fall into place.

throwaway57522• 12 years ago

You have to understand that some people make a good amount of money (5 figures or more) speaking at conferences like this. As a result, there's a cottage industry of people trying to declare the "death of {sacred cow}" or "the end of {foundation of the field} as we know it". These are controversial and can result in speaking invitations. Obviously Dave Thomas is a little different, but he's still human. Note the silly pandering to the crowd - he's done this many times and knows how to work an audience.

Now, that said, Agile is fairly new as software engineering goes. It isn't that young of a field.. waterfall put man on the moon, after all. Agile particularly speaks to how one can remain somewhat stable while optimizing for velocity, and is particularly well suited for the type of businesses that have dominated the last 10-15 years: self hosted consumer applications or SaaS. The key to choosing a process is to understand what you are trying to optimize for and what sort of release process your project requires. Agile, for example, might not be the best solution for gamma knife firmware or embedded systems in cars. To give him the benefit of the doubt, perhaps he's reached the age of reflection and trying to emphasize that everyone should step back and consider their goals, and reevaluate their strategies in light of those goals. [edit: and understand a lot of people are lazy and/or risk-averse, and would prefer to think that there is a magic best practice, that if applied hard enough, will work for everything]

sitkack• 12 years ago

waterfall put man on the moon

As we learned with the Apollo Guidance Computer [0] and Shuttle computer systems development [1], the methodologies were not strictly waterfall. I'd like to do some more research in historical engineering techniques. I am not convinced that waterfall has had a much sway as the industry makes out.

[0] http://en.wikipedia.org/wiki/Margaret_Hamilton_(scientist)

[1] http://en.m.wikipedia.org/wiki/Iterative_and_incremental_dev...

InclinedPlane8• 12 years ago

I think you might be laboring under some misapprehensions here. Are you aware who Dave Thomas truly is and his relation to "agile"? He's one of the co-signers of the original "agile manifesto". "Agile"is his baby as much as anyone's. And he's here to say that it's being misused and abused. He's not against agility, he's saying that what passes under the "agile" moniker today has very little to do with the original conception.

MetaCosm• 12 years ago

Welcome to software development, where no one really knows what they are doing, the methodologies don't matter and everyone is wrong! Software is still a mix of art and science, there are a lot of ways to get to the right answer, and it is often very arbitrary and unbounded by physical processes. It might never settle, it might be like all the different methods authors use to write books -- strange rituals and personal patterns... there is no "right" way to author a novel -- might end up being the same with software. The end product is what you will be judged by...

How do you teach people then?

You can't really, teach to the best of your ability and try to send students out with an ability to learn. It is MORE harmful to PRETEND you know what is going on than to openly admit you don't and are attempting to learn. False knowledge is worse than real confusion.

Where can i learn how to do my job correctly?

In this field, it is a young and crazy field, maybe you will be the one who pioneers a way to actually do things more consistently... maybe you will just serve up another nonsense silver bullet.

can you imagine medical students being taught "yea just try something and see what happens, hopefully your patient doesn't die and you'll learn something for the next patient."

Doctors kill patients all the time via mistakes, it is actually one of the leading causes of death in the western world... the half-life of medical knowledge tends to be less than a decade, so a decade out of school, a lot of what they know is wrong, or worse... actively harmful. We turn doctors into gods because it COMFORTS US, not because it has any connection to reality.

pinaceae4• 12 years ago

it is not a mix of arts and science, it is akin to middle ages styles manufacture. re-inventing the wheel, left and right, hundreds of years away from real engineering. even questionable if something like the freemasons exists yet.

tjradcliffe• 12 years ago

Software is still a mix of art and science

Which is to say: just like everything else, including both art and science.

slowmovintarget3• 12 years ago

The four steps Dave gives in the video are the observations of someone practiced at distilling heuristics. Experts often speak very few words with very few specifics, for a simple reason. The words they use are weighted with all the meaning of all the things that led to them. For someone with much less experience to discern meaning from them, that person must constantly be examining their own work as they do it, thinking about how it might connect with or diverge from those words.

That's generalization for you, though. It says more, less clearly.

Regarding the Agile Manifesto, he's not said they were wrong, Dave's saying that the main principles of the manifesto are being ignored.

From his blog post:

[quote]

Let’s look again at the four values:

- Individuals and Interactions over Processes and Tools

- Working Software over Comprehensive Documentation

- Customer Collaboration over Contract Negotiation, and

- Responding to Change over Following a Plan

The phrases on the left represent an ideal—given the choice between left and right, those who develop software with agility will favor the left.

Now look at the consultants and vendors who say they’ll get you started with “Agile.” Ask yourself where they are positioned on the left-right axis. My guess is that you’ll find them process and tool heavy, with many suggested work products (consultant-speak for documents to keep managers happy) and considerably more planning than the contents of a whiteboard and some sticky notes.

[/quote]

Dave then goes on to give his heuristic as a replacement for all the consultant BS that emphasizes the very things that are supposed to be less important.

He's really saying pay attention to the requirements, check your progress toward them frequently, and every-so-often, check that they're really still the requirements. As for code structure, technology selection and so on, he as much as came out and said to work with an architect or a master developer to help make those choices correctly.

He was again correct when he said that all of software development practice boiled down to making things that were easier to change over the long run. So when other factors don't rule it out, choose in favor of flexibility.

pinaceae4• 12 years ago

don't worry, this is just an opinion.

there are a lot of shops out there that develop and deliver great, working software on a rapid schedule. succesful saas shops come to mind - and even Microsoft has picked up massive steam in that regard.

the really awesome, productive dev teams are too busy to publish talks and whitepapers about their methods - they're 100% tasked with shipping. think about that anytime you see talks and conferences, etc. the best engineers in any industry work in silence.

it's the blow hards and wannabees that end up on stage or writing manifestos :)

percept• 12 years ago

A flippant or cynical answer might be that you'll still want to know and follow "agile" anyway, if only because 1) it matters from an employment perspective, and 2) so many organizations are just now adopting it.

But you're asking about learning, and from my perspective anyone who is willing to ask incisive questions has the wherewithal to continuously improve and ultimately succeed in his/her field.

Fads and fashion...

curiously3• 12 years ago

I'm a junior developer who just got out of college

Welcome to life. You will find that the world doesn't work the way it's described in books or from the mouths of someone with grey hair.

pmoriarty8• 12 years ago

I would add: "Don't be a sucker."

gyardley• 12 years ago

I've worked with a bunch of development teams as a product manager and founder over the last decade or so. Whenever the team has come together with the rest of the company and collaborated to come up with a process that works for both them and the business, things have gone pretty smoothly, even though the processes themselves have varied widely from place to place. Whenever any party - developers, executives, product management - has tried to force a methodology without full discussion and buy-in from all parties, it's been a complete disaster.

Because of this, I believe 'doing your job correctly' varies considerably from place to place, depending on the different people you happen to be doing your job with. Learning a specific development methodology is much less important than communicating and working with others to come up with a development methodology well-tailored to the context.

rjberry• 12 years ago

I think it depends on what you're working on. If the software is completely designed and the desired outcomes known from the beginning then up front planning and processes are probably helpful. But this is such a small percentage of actual software. In 99% of cases the business itself doesn't know exactly what it wants (even though it might initially think it does) and a lot of the seemingly chaotic ways of approaching software engineering are really just reflecting that.

a-saleh• 12 years ago

"Where can i learn how to do my job correctly?"

I believe the best place is to work your job, and have the right amount of introspection :)

But what helped me the most with this "how to do software corectly" question, was to realize, that there is many different kinds of software.

If I would attempt an analogy, our industry reminds me of carpentry.

And as a carpenter you might approach differently if working alone, or with coleagues, if renovating or building a house on a green field, if building a garden bike-shed, or working on a river-dam :-)

Meaning there will never be specific 10 commandments applicable for everything.

Sometimes agile will fit your team (heck, my most productive week was spent PAIR PROGRAMMING :P )

Sometimes it won't.

infinite8s• 12 years ago

That's pretty much exactly how medicine worked prior to the twentieth century and its systemization by science.

bphogan• 12 years ago

Learn to do your job correctly by experimenting. Read other people's code. Write other people's code. Learn by doing. Learn by failing.

One of the things that I believe is incredibly important is a feedback cycle. Do a thing. See if it worked. Reflect on what you learned doing that thing.

And most importantly, ignore any feedback that starts with "You're doing it wrong." But listen carefully to anyone who says "Here's how I'd do it."

This is how I learned software development. I wrote crappy things and placed them before people I respected. And these people gave me exactly what I just explained..

You're a junior developer - there are so many people willing to help you out and learn, if you ask.

Enjoy software development!

mathieuh• 12 years ago

I'm in university at the moment and we're learning about Agile. Next year I've managed to get a year long placement with a big local software company and their interview and technical assessment mentioned Agile loads. The year after that, I have half a module (worth 8% of that year's available marks) based solely on Agile. I've also attended talks by big names like Citigroup and also local startups which were almost entirely about using Agile.

So is it in fact the case that in The Real World Agile is starting to go out of favour?

rebelshrug5• 12 years ago

No - companies invest a lot of resources into their software development processes, and changing processes often involves a significant investment of it's own, not to mention a slow down in productivity.

radicalbyte• 12 years ago

So is it in fact the case that in The Real World Agile is starting to go out of favour?

Nope, it has finally crossed the line into being "main stream". On the technology adoption lifecycle* it has hit the late majority.

It has been taken up by the "software development as career"-types who're only in it for the money; it's now a "requirement", just another box to be ticked.

Sad, but on the plus side it's better than the "waterfall" processes which preceded it.

* http://en.wikipedia.org/wiki/Technology_adoption_lifecycle

guiomie2• 12 years ago

I keep hearing the word "agile" from vendors, consultants and non-software related stakeholders at my work (a telco). For me it is a buzzword, thus I have been quite reluctant toward it.

Seeing also one of the original authors thinking so, means I'll stay even more away, and perhaps take people using the word with skepticism.

DanielBMarkham2• 12 years ago

"Agile" is a marketing term: it brands various best practices in iterative and incremental development. You go to a bookstore and want to read a mystery? They have a whole section labeled "Mystery" where you get all kinds of good stuff. You want to learn how groups of people work well together in tech? Go to the "Agile" section.

I like this, but it's tough for many people to grok, because it's not a detailed instruction book, like some other methodologies.

If you want to stay current in how people are doing cool stuff, consume Agile material on a regular basis. But take it all with a grain of salt: you learn stuff and then try it. What works in one situation won't work in another. One of the things practitioners learned early on is that tech is 95%+ about people, not bits and bytes. People tend to be a few orders of magnitude more difficult than, say, C++

pjmlp2• 12 years ago

Many developers would learn a lot if they improved their soft skills instead of yet another programming language/framework.

pmoriarty8• 12 years ago

A grain of salt??? More like a truckload of salt.

Have you not watched the video? Agile is a scam. Even one of the signers of The Agile Manifesto himself just admitted it on camera.

That some people will continue to give credence to an outright scam after it's been exposed and admitted to by one of its foremost practitioners is just a sad statement on humanity.

fpp• 12 years ago

Also read his interview on the topic in Dr Dobb's earlier this year http://www.drdobbs.com/architecture-and-design/dave-thomas-i...

The diagram he shows in the presentation around 5min is from Scaled Agile that is trying to formalise the use of agile development approaches within the enterprise context (online at http://scaledagileframework.com/ )

I believe some the key messages from Dave are:

* Use common sense and focus on the goals and what you're planning to deliver - the approaches provide you with the agility to more likely arrive there.

* People and working solutions before processes and approach

* Collaborate, explore and adjust to change vs follow a predefined fixed plan

Some negative examples I've seen:

"Funny and ridiculously expensive certifications" - Job adverts making it a key requirement to have a "scrum master" certification instead of looking for people who successfully delivered solutions (within change / complex environments) Note: It takes 1-2 days from absolute beginner to being a "certified scrum master" and it costs a stunning £1,200 in London.

"Process over common sense" - Within a company of some of my friends when they asked me to help to turnaround a critical situation with their key development team - besides 2 developers all others had left because of conflicts with their line managers, one of them now standing in as "scrum master" and prescribing to continue and do daily stand-ups like before with two 6 people teams - sometimes only one developer together with the manager.

placebo2• 12 years ago

Watching this, my first thought was how an aerospace engineer or a structural engineer or any engineer/manager of any complex established engineering field outside the software development field would react to this video. Probably with horror, shock and disbelief :)

Though I haven't given this too much thought, I think it boils down to the fact that the software engineering process is not like (and who knows if it will ever be like) older well established engineering fields, and this could be due to a combination of:

a) The unbelievably vast number of degrees of freedom and therefore infinite ways an engineer has to achieve a working model. Aside from the requirements and the hardware limits, the only limiting factor is the engineer's imagination and creativity.

b) The mind blowing complexity of a large software project. The space shuttle, definitely one of the more complex machines built by mankind, is built from about 2.5 million separate parts. The Linux kernel (from 2012, according to wikipedia) has 15.9 million lines of code, and it is by no means the largest software project.

c) The relatively young age of the field vs. the exponential rate at which new technologies are devised and introduced in it

All this I feel currently puts software development somewhere between an engineering and a form of art. However, I'm sure many people would find it unsettling to think they have to trust a form of art when flying in a commercial jet at 30,000 feet, launching a $500M space mission, or endless other modern operations that depend on software to succeed. The fact is though, that it does succeed. I doubt the same methodologies are used in every large software project around the globe, but the bottom line is that whatever methodologies are used (not taking into account different levels of effectivity or cost) most apparently appear to succeed getting the job done without a established "standard".

tonyedgecombe2• 12 years ago

Most software projects fail.

placebo2• 12 years ago

What do you mean by 'fail'? fail financially or fail technically? If you mean the former then that's a no brainer, as I assume most projects fail financially whether software or not. If you mean fail technically then I'd be interested to see numbers. It's certainly not my experience.

rcxdude• 12 years ago

Most projects fail.

bane• 12 years ago

If there's a truism, it's that Agile is and always will be a reactionary process to older Waterfall style methodologies adopted from other engineering principles. That means it suffers from some of the kinds of problems reactionary "we'll do the opposite of that old broken stuff" ideas have.

For example, I'm afraid that in many environments, Agile ends up meaning "directionless or micro-directioned iterative development"

e.g. just start building stuff, nobody has any real idea what the ends state is supposed to look like or work, but we'll iterate into something at some point. Probably when we get close to running out of money.

Old fashioned honest engineering is tossed out while the framework/technology of the month gets absorbed into this or that sprint cycle and when problems preventing what feels like forward motion are discovered, crap just gets thrown against the wall until something moves forward a step.

Things ship late, nobody can schedule anything, budgeting and hiring plans are a mess.

One of the defining aspects of being human is that we're capable of thinking a few steps ahead and formulating a plan. The way Agile works in most shops returns us to being mere beasts of coding.

Of the places I've seen, the difference between success and endless-iteration-failure-followed-by-multiple-rewrites-from-scratch-before-delivery-because-look-at-all-the-lessons-we-learned-along-the-way is that somebody sat down before any work started and defined the goal, timing, budget and expectations.

There's some planning, some mockups, some interaction prototypes, etc. at the start that's hard work to do, but defines the goals and vision the work effort should be going towards...the "shape" of the work.

Goal driven development is something that seems to be missing in too many shops. It's hard though, because you don't want to end up in the waterfall trap again, but you don't want to end up in an directionless iteration morass either.

It takes experience and lots of road miles to understand that balance and succeed.

UK-AL4• 12 years ago

Agile is planned, just on a shorter time scale and gets more vague over time. Sprints have end goals, and so do epics.

Waterfall had big upfront design, which meant every time a manager asked for a change it screwed up the design and had to modified which took a lot of time. So people didn't ask for changes and the resulting software didn't do what they really wanted.

Iterative development tries to embrace change, it's an attempt at controlled change.

enraged_camel3• 12 years ago

>For example, I'm afraid that in many environments, Agile ends up meaning "directionless or micro-directioned iterative development"

Directionless. Absolutely.

Before he retired, my uncle was the VP of Technology for a very well-known financial firm. He managed a large department of developers and IT folks across multiple countries.

We were talking about Agile vs. Waterfall once, and he said something that stuck with me: Agile works if and only if your developers have significant amounts of industry expertise. Because only then can they look at a set of vague requirements, make sense of them and translate them into solid product features using their experience and knowledge in that specific problem domain.

Whereas if you look at the software industry today, most developers specialize along languages and frameworks, rather than industry lines and problem domains. Resumes almost never say, "X years of experience in [industry]" because most companies say they want to hire developers with "X years of [insert language/framework here] experience."

So then they end up hiring developers who are really, really good at Rails/Angular/Java/C# or whatever, but have never solved problems in the company's industry because their last job was something totally unrelated. They know how to use the tools and they may be able to iterate quickly, but if they implemented the feature correctly in the first place they wouldn't need to iterate in the first place and the product would ship on time and be on budget.

pinaceae4• 12 years ago

that's exactly why product managers exist. i run precisely such a team, because there are no devs out there that know enough about our vertical - and the time learning and keeping up to date would be prohibitive.

danieltillett• 12 years ago

This is so true. I am mediocre developer at best, but because of my domain knowledge I can solve problems that developers 100 times better than me can't. Sure my code might be rough around the edges and some of my designs lacking in elegance, but they work. Working code is a million times better than elegant code that doesn't.

Edit. I have often thought that a good way to pair program would be to pair up a strong domain knowledge / weak dev with a weak domain / strong dev. I have had the opportunity to do this on occasion and the pace at which you churn out great working code is phenomenal.

lifeisstillgood• 12 years ago

For me, and I think i just worked this one out, Agile can only work in a collaborative, open, high trust environment.

If you have that, pretty much anything you do will be agile. If you don't there is bugger all you can do to get to agile without fixing it.

Personality conflicts, leaders who shut down discussions, developers who just want to do what they are told and collect a pay check, this and more will screw you up.

So go for slow development, constant learning, some humility, and a company that won't fire you for looking like you are just "thinking about it"

tonyedgecombe2• 12 years ago

This is true, I suspect that companies that fail with agile will fail with any and every methodology.

InclinedPlane8• 12 years ago

For those who aren't aware or didn't pick it up, Dave Thomas is one of the handful of originators of the "Agile Manifesto".

Other than the obvious (profit motive in selling snake oil) there are some common problems that make evaluating and applying process difficult. One of the biggest problems is actually the effect of highly talented teams. If you have a team of skilled folks then they will often succeed independent of process. If they are burdened with unworkable processes they will quite often work around them, bend them, or ignore them and get shit done regardless. Which makes it difficult to tell when you've got a good process in place as well.

Another major problem, alluded to in the video, is the desire for brain death in management. People want silver bullets, they want universal advice, they want policies that don't require effort or thought to achieve results. You see something similar in tech security, everyone wants to hire a unicorn/jesus who swoops in and fixes everything, like a landscaper who comes in on the weekends to do yard work or something. The reality is that development, and security, requires effort at every level. It requires conscientious application of skill and judgment to get good work done, and that's true at every level. You can't simply apply one or another process and then switch your brain off and hope for the best.

mattmurdog• 12 years ago

Very well said.

ItsMattyG• 12 years ago

Agile is good in uncertain environments with frequent project changes, but not in other situations... Simon Wardley has a good model that seems to work reasonably well:

http://blog.gardeviance.org/2014/11/how-we-used-to-organise-...

chriscareycode• 12 years ago

I love this talk by Erik Meijer on the topic http://vimeo.com/110554082

oldpond• 12 years ago

I think I know who Dave is making fun of, and yes, there are plenty of stakeholders who are hopelessly lost and willing to "buy agile" to make it look like they are doing something. Just get all the useless project managers to call themselves scrum masters and hold daily stand-ups (which in the old days we called micro managing) and hope and prey that something works...

I've been in this business since the 80's and a consultant since '98, and I can say that although I love the agile approach, busted organizations and "enterprise" consultants are killing the ideas. Agile is a software development approach, not an organization development methodology. The last gig I was on the customer tried to "go agile". All the departments started holding daily stand-ups which quickly became silly repetitions and micro management kill-zones. Picture this, the daily help desk standup: "Yesterday I answered phones, and today I plan to answer phones again." Everyone was doing 'agile' except the software development teams. Maybe Dave is right, all you can do is laugh about it. And then put it all on the cloud.

InclinedPlane8• 12 years ago

At one of my past jobs we did "agile" or "scrums" or whatever they called their bastardized version of an agile process. At one point this involved standups which included over a dozen people and typically lasted between 45 minutes and an hour and a half. Everyday.

EdwardDiego3• 12 years ago

Gah. So let me guess, the "stand-up" ended up with a bunch of people sitting down, because after all, you can't stand up for that long...?

...the number of people who miss the reason for standing up (to force brevity) is astounding.

enraged_camel3• 12 years ago

Unbelievable. You just described my previous employer. I was there for over six years and witnessed the transition to Agile. Literally every department, including the sales team, started doing daily stand-ups. I was in a technical consulting role, and our stand-ups were exactly like you describe: "Yesterday I answered seven emails from customers, which took me five hours, and today I'm going to spend the rest of the hours for that task on..."

The incredible thing was how much upper management treated Agile as a panacea. It made me realize that they had no idea how to actually run a company, which is why I ultimately quit. I can't work for people I don't respect.

wpietri• 12 years ago

I was involved in the Agile movement early on, and it killed me to see this happen to friend after friend after friend. What began as a bottom-up, respectful approach to finding better ways to work was co-opted by management (and the consultants they hired) and turned into just another stick to beat people with.

InclinedPlane8• 12 years ago

(As much as I hate to post twice on a thread, I've had some thoughts that I haven't seen expressed yet.)

I think a lot of folks are missing some important perspective on the origin and rise of "agile" and the current state of process in the software industry. So let's take a quick trip back to the bad old days before agile to get a sense of what software development was often like back then.

First off, a lot of software was developed "out of house" so to speak, as works for hire by consultancy type companies. There would be extensive negotiations between the customer and representatives of the development company, the end goal of this process was typically a specification and a timetable that was then mutually agreed upon, through a legally binding contract, as to what would be built.

Next, the development company would take that specification and come up with a design that would be able to match the specs. Then that design would be sliced up and handed out to various teams and eventually down to individual devs. After the individual coding was done it'd be integrated together and compiled into a working product. The product would then be passed off to QA to test in order to make sure it didn't have any bugs and that it matched the spec. After that the product would then be passed on to the customer to use.

And everyone lived happily ever after.

Of course, there are a few problematic aspects to this process. Namely: everything. Usually what would happen is that the spec and/or the design would be unrealistic or impossible to implement, and wouldn't be what the customer actually wanted regardless. Often this was handled by acrimonious bouts of negotiation, often involving lawyers. Meanwhile, attempting to build anything that worked at all using this sort of process was a nightmare. When the software was integrated only at the last minute there were always a million new problems discovered. With such "big-bang" integrations a lot of effort is spent spinning away just getting the software to build and work at all. Meanwhile, when QA is the last step before handing the product off to the customer that means that defects, especially design defects, have had the greatest amount of opportunity to fester and take the most effort to remove. And, of course, the chance that the project would chew up many staff-years of effort without actually producing anything of use whatsoever was quite high with this model, since actually building software was a fairly late step.

In the face of all these very fundamental problems with the venerable "waterfall" process model a lot of new process ideas started to gain traction, more or less culminating in the "agile manifesto". The core idea of agility is to use iterative development, continuous integration, and open lines of communication to keep on track. The "customer" (or "stake holders") can see the direction of the product mid-stream and have many chances to correct communication errors or even errors in their original conception of what they wanted. The software is always being built and always being tested so integration overhead costs are much less severe and defects are spotted much closer to where they are introduced, making them easier to fix. The software is routinely in a "shipable" state with a continually evolving subset of the "final" featureset, this lets the customer see how close the developers are to the schedule and also dramatically reduces the risk of not shipping anything at all. The developers can always time box the release and ship something of value, even if it wasn't what was originally intended.

And so on.

The thing is, today we live in a fundamentally agile world of development. Waterfall is so far from the norm it's essentially extinct. The very idea of futzing around with nothing but requirements and specs for months or years before bothering to write a line of code is so anathema to the current standards of software development it seems ridiculous. Everyone knows you start with a skeleton and you flesh it out iteratively. The idea that you'd have your code base in an unbuildable state, let alone an unusable product state, for more than a few hours or days at most is similarly seemingly preposterous.

The fact that the basic principles of agility are now so ubiquitous that they are like a mold infecting every nook and cranny of the software industry is still unsatisfactory for a lot of folks. Management wants a process they can sink their teeth into. They want something that requires no effort on their part but seems like a silver bullet that can solve any problem. They want gadgets and tools. They want a process that they can leverage to justify all of their bad behaviors while removing accountability from themselves. And that's what agile-the-noun has become. Not agility, but rather an excuse for micro-management. A way to plan without planning. A justification for short-sightedness and disengagement. A convenient rolodex of excuses for why everyone but management is at fault for being late or building something bad or broken or that no one wants.

You either die a hero or you live long enough to see yourself become the villain.

serve_yay• 12 years ago

He is quite a speaker, but really the first 15 minutes is a standup comedy routine for project managers. You can skip that part.

dmux• 12 years ago

I'll just leave this here: http://seir.sei.cmu.edu/sheard/Life%20Cycle%20Silver%20Bulle...

The Death of Agile [video]

Related Stories