What we actually know about software development, and why we believe it’s true

http://www.slideshare.net/gvwilson/bits-of-evidence-2338367

by ZeroGravitas • 17 years ago

133 54 17 years ago

slideshare.net

adamc• 17 years ago

Must have been a great talk, because even the slides are good (and normally the slides are almost useless alone).

colomon2• 17 years ago

It was a brilliant talk, easily worth the cost of driving out from Michigan for DevDays.

Maciek4162• 17 years ago

He definitely got the most thunderous applause of the day at DevDays simply because so much of the day's talks thus far had been horrible. I was following the Twitter feed of the conference and after his talk many people remarked that he had basically saved DevDays Toronto from being a failure.

9oliYQjP• 17 years ago

I was hoping Joel would send around a URL for feedback. I didn't stick around to see if that ended up happening. The problem wasn't that the talks were horrible in and of themselves. But it appears that he totally underestimated some of the audience's abilities. I couldn't have been the only person there that's used ASP.NET MVC, Python, jQuery, and Ruby. For us, the talks weren't insightful because they were all introductory.

That said, I chatted with some attendees who thought the talks were great because they only program in ASP.NET (old style) and have never been exposed to these sort of languages. I guess they're the developers that ask the questions on StackOverflow and people like us are the ones answering the questions? :-) Or maybe I was just unlucky enough to know about the languages they decided to focus on. Had there been an introductory talk on Haskell or Clojure, I'd have learned something new.

swombat• 17 years ago

It looks like it was a nice talk, but unfortunately it failed to address the reason why there is so little reliable science when it comes to software development.

The reason is, of course, that we have yet to come up with a reliable, meaningful metric for programmer productivity, one that would apply to the lone ruby on rails coder as well as to the enterprise java cog-in-the-great-machine or the ThoughtWorks elite agile programmer in a 6-people iterative team.

Until there is a meaningful way to compare the productivity of different software methodologies, it is pointless to berate the lack of scientific studies comparing the productivity of these methodologies.

As a final nail in that coffin, I don't think anyone can claim that Google is lacking in the "let's measure everything" department, with their reputation for intensive A/B testing of every single pixel change. And yet Google has yet to resolve this problem... which would tend to indicate that this is a real, non-trivial problem, not just a case of people not trying hard enough.

nopassrecover3• 17 years ago

I'm of the opinion that the reason there is no quantitative metric for programmers is the same reason there is no quantitative metric for writers, artists or musicians.

arohner• 17 years ago

I agree this is a key point. Software bug rates are not comparable to scurvy incidence rates, and developer productivity is not similar to the number of bricks a laborer can move in an hour.

gaius4• 17 years ago

Yet there are quantitative metrics for engineers. Is your Ruby on Rails application really more complex than the software flying an airliner?

teeja• 17 years ago

For sure. Before 'hit parades' music could sit around in a desk drawer for decades (Schubert) before being heard. Because innovation drove away audiences wanting to hear the familiar. The people who made sure that Schubert's stuff lived was other composers ... not audiences.

Software's the same ... the people who cheer great code (art, science) are people who understand it well enough to recognize great code. The opinions of peers is an important part of achievement metrics. And that artful kind of code pretty much counts for nothing by any standard 'industry' measures. But it's influential.

angelbob• 17 years ago

One solution to this is that you can measure similar projects. "The Mythical Man-Month" is based on this idea. If you compare large projects to large projects (or a large project to itself with different methods, different number of people, etc), you get a decently reliable, if imperfect, metric which can give good results.

Similarly, compare elite six-man teams to other elite six man teams (not very reliable, but interesting if results are extreme) or to themselves on other projects, possibly with slightly different methodologies.

Yes, it's difficult. It would be nice if we had some simple means of measuring productivity, but much of programming is very artlike -- we're not solving identical problems, and we may solve them in very different ways, and a lot of the resulting quality depends on the taste of the viewer. We're not going to get a really excellent, predictable way to measure quality.

Waiting until we do means no research, which is awfully fatalistic.

ejames• 17 years ago

Still, there's a lot of room between "no evidence" and "double-blind clinical trials", and reason to believe that statements about software development can profitably be moved towards the latter.

I would take Code Complete as a good example. There are sections on methodology that run along the lines of "Company A tried this method and got a B% reduction in defects compared to the first version of the product. Company X also used it and only got a Y% improvement compared to the previous year, but that's still worth using." Studies don't have to be totally conclusive in order to provide meaningful back-up to anecdotal reports and subjective experience.

In other words, the perfect is the enemy of the good, and good evidentiary metrics may be achievable even if perfect ones aren't.

Maciek4162• 17 years ago

Good point.

I attended this talk, and it was easily one of the best talks at Stackoverflow (Devdays) Toronto, and got a very rousing response from the crowd.

To answer your concern, the question of "why" was actually one of the overall points of his discussion. Wilson would enthusiastically agree with you that there aren't very good metrics out there not only for programmer productivity, but a number of other things that you'd want to measure in software. At the beginning of the talk he contrasted the software sciences against other, older disciplines just to show how incredibly nascent it is.

His most damning point of the current state of affairs was that there is a lot of metrics/methodology pseudoscience being spouted by people and businesses in our profession, and that we often eat it up unchallenged. His optimistic counter-point was that things are turning around and measurement, experimentation and skepticism are spreading, and as you point out, for some companies like Google, it's yielding a lot of gains, even if they haven't cracked any universal code of productivity (does anyone actually believe we ever will? This will always be a moving target).

The most frequent phrase used by Wilson in his talk was "wouldn't that be a useful thing to know?" (with respect to the result of some experiment that would show a verifiable result). Essentially, his talk was a rallying cry to attempt to stir the audience into becoming more critical about their own methods and to boost their skepticism of the things they're told about methodology.

3pt14159• 17 years ago

I also attended this conference in Toronto. I didn't like the last two thirds of the talk one bit. There is plenty of evidence that explains why less females are in software development (and in general, science) there is also plenty of science that disputes anthropogenic global warming (and also plenty of science that supports it). The speaker made highly emotionally charged statements that were in the opposite spirit of his whole thesis: DATA and ANALYSIS should drive opinions and decisions, not emotions. All of this is besides the point. What does global warming or Rush Limbaugh have to do with the science behind software development organization?

While he is absolutely correct that software development lacks rigorous testing of effective methods (agile vs waterfall), he allows, openly, volunteers (!) to be part of his test group ("I'll post my email") after already telling them what he is testing. This makes his inquisition of the most effective method even worse than nothing at all. By taking in biased volunteers he makes his numbers look like they achieve statistically validly, while failing to take a true cross-section of software developers.

My favorite talk of the night was the Ruby guy. Clean, unemotional, understandable, interesting, informative. I almost expected him to end with "live long and prosper."

mustpax• 17 years ago

Not only is there no metric, but when we force artificial metrics onto programming teams, we are forcing them to pursue a goal other than the quality of the software itself. Any benefits derived from measurability will be eclipsed by the bad practices brought about by the metrics.

Take the two of the most common pseudo-metrics: lines-of-code, bugs-closed-per-month.

Lines-of-code encourages verbosity and complexity while discouraging code reuse. It also ignores essential code maintenance tasks that usually remove lines-of-code rather than adding them.

Bugs-closed-per-month encourages the creation of bugs in the first place. Not that anyone intentionally makes mistakes, but it lets the programmer get away with getting it less-right the first time. Even worse, it assures continued dividends from an error-prone implementation.

It would be useful to seek macroscopic measures of programmer productivity in our discipline as a whole. Let's just keep in mind the potential adverse effects of applying these metrics on the ground. Realistically, any effective macro metric will be abused my PHBs all over the world in managing their 10 man teams.

pohl2• 17 years ago

Somehow even your use of the word "force" here still manages to understate the severity of the problem.

The "Performance Management" craze is sweeping corporate management, along with the pernicious use of the "Balanced Scorecard" it advocates. Unable to identify meaningful metrics to build these scorecards with, management delegates to programmers the task of inventing metrics. Programmers, who know very well how any metric can be gamed, know that they're obligated to hand management a loaded gun, and hope they can fill the scorecards with a gun of the least-damaging calibre.

This will continue unabated until a better science appears, and there's a very real possibility that one never will (if programming is more akin to an art and/or craft).

I would love to have seen this presentation.

srn• 17 years ago

I would consider "bugs opened per month" to be a good metric, assuming someone uses the software.

stonemetal• 17 years ago

What I find odd is that there used to be quite a bit of research in this area. Read older software development books they reference papers research more often. i.e. Code Complete vs. agile programming books.

raganwald• 17 years ago

I freely admit that I have no idea how to measure productivity. And until I do, I really can't tell you that what I choose to do is anything other than consistent with my superstitions about software development.

In other words, I choose some superstitions and I can build up a consistent kind of world based on those superstitions and I can test various ideas to see if the ideas are consistent with my superstitions or the logical consequences of those superstitions.

But really, I have no idea whether I'm directing development traffic or whether I have a pair of coconuts clamped to my ears and I'm waving palm fronds under an empty sky.

http://github.com/raganwald/homoiconic/blob/master/2009-02-1...

InclinedPlane• 17 years ago

Worse than that. We can't even measure success.

In the extremes of ultimate financial success or failure in the long term we can determine success, but that gives us something like 1 bit of data per 3-5 years per major software project. Is twitter successful? MySpace? Facebook? Was Netscape successful? Geocities? (The stock used to buy Geocities is currently worth about $2 billion dollars.) For each of those there are even more examples that are more difficult to determine. Modern software comes in a series of releases, how much can you attribute the success of a particular release to the existing code base, and how much can you attribute it to just the particular diffs for that release? It's a tricky problem with no easy answers.

nradov• 17 years ago

In principle it should be practical to compare the productivity of different software methodologies by counting Function Points (or something similar) as well as defects. http://www.ifpug.org/ But in practice that doesn't work very well. It's awkward to count Function Points for maintenance and refactoring work in a meaningful way. And for most organizations, the cost of just gathering the data and doing the counting is so high as to outweigh any possible benefit.

bbraasch• 17 years ago

he makes some excellent points, or should I say hypothoses?

drop 25% of the features to reduce complexity by half. I suppose this is why templates and re-usable code appeal to managers moreso than software developers. one sees the clock and the other sees something that would be much more beautiful after a redo.

my son learned in mechanical engineering school that if you want the best design and the best device, don't ask the builder to design the device. the builder will make decisions based on the complications of the build process and the desire to build. the designer would focus on the utility of the device and defer thinking to the builder on how to get the thing built.

People with different Myers Briggs score do better in pair programming. Raise your hand if that makes your brain hurt.

Mythical Man Month is the classic book on this, based on the OS360 project at IBM.

Complexity drives time to complete toward infinity. Says the moth to the flame, "you're so beautiful". Says the teacher to the class, "let's not get ahead of ourselves".

A fellow named Francis Frank used to teach project management as a position that sits between the hydrant and the dog. The scrum master takes that role now, looking for a favorable outcome from myers briggs score diversity.

I especially like the point that defects are not a factor of geographic distance, but of distance on the org chart.

tybris• 17 years ago

Programming is still a craft. If you want to know how it should be done, just look at the masters. Plenty of successful products out there that can serve as an example.

Until we develop actual computer science (rather than calling publicly funded engineering science) that's all we have.

colomon2• 17 years ago

Ummm... you do realize that the slides point out scientific studies of programming with interesting, practical results, yes? And that the author of the slides is a professor of computer science doing more such studies? One of his points in the talk was to call for the professional programmers in the audience to participate in these sorts of studies.

hxa7241• 17 years ago

If you look at research papers in software engineering, you will notice a significant thing has happened in the last several years. With the rise of, first the WWW, and then open-source repositories, there is now a large body of code available that was simply absent before. It is now usual/common for research to investigate, test, experiment against that. This must be a positive change, that we should be hopeful about yielding more substantial results.

russell3• 17 years ago

A couple of points buried in the talk: defects increase as the code size increases. The increase is greater than linear. So my question for folks is why would you program in Java rather than Python, where you are guaranteed to have more bugs, even with static type checking?

The issue of programmer productivity has been around at least since The Mythical Man Month, but the generic responses have been better estimates and better methodologies. There is even the Software Engineering Institute to make life worse for all of us. I think that except for academics and programmers themselves, no one cares. Large companies used Cobol and now Java because it is safe. Everyone else is doing the same. I once asked an executive why he picked Java over Python for a greenfield reimplementation of his site. he said it was because there were more Java programmers than any other kind.

pohl2• 17 years ago

"A couple of points buried in the talk: defects increase as the code size increases. The increase is greater than linear."

I went looking for these points in the slides, only because the slides got me in the mood to question my assumptions and try to follow citations back to source material - and it wasn't obvious that the presence of something on a slide meant that it was a known truth. (Example: the slides about Martin Fowler's claims about DSLs and productivity are in the slides as a skull on a pikestaff, not as a verified truth.)

After looking, I think you got the first point from slide 15, but I wasn't able to find a slide that mentioned non-linearity.

I like the question that you're asking, with regard to Python versus Java. I wonder, though, if that conclusion really rises to the standards of rigor that this presentation appears to advocate.

Even if you manage to dig up the right Lutz Prechelt citation, has the effect really been shown to be something that is completely unmitigated by static-provability?

russell3• 17 years ago

Slide 25: "Most Metrics' values increase with code size". The point about non-linear increase in defects is my (unwarranted) extrapolation. An earlier slide quotes a 25% increase in complexity gives a 100% increase in size.

No, it really doesnt rise to his standards of rigor. Everything that I have read, supporting that conclusion is probably anecdotal. The formal studies I have read have usually involved college students on short projects. Studies that are probably worth less than useless. Others probably suffer from the Hawthorne effect.

apotheon3• 17 years ago

> Large companies used Cobol and now Java because it is safe.

"Socially" safe, that is. Damned dangerous in other, more neglected, senses.

warwick• 17 years ago

Do you know if he meant larger as in 'wc -c' larger, or as in number of symbols larger? I'm curious because there are languages (Cobol, Obj-C) that are extremely verbose in terms of characters but not in symbols.

regularfry• 17 years ago

The classic study cited in Mythical Man Month (which I don't unfortunately have handy) said, if I'm remembering it correctly, that it was proportional to lines of code (thus 'wc -l' larger) and independent of the language used.

What I don't know is whether IDE-generated code gets you out of that problem or not.

ShardPhoenix• 17 years ago

A study (http://page.mi.fu-berlin.de/prechelt/Biblio//tcheck_tse98.pd...) by the Lutz Prechelt mentioned on the slides suggests that, all else being equal, static type checking reduces defects and increases productivity (the study was for ANSI C vs. K&R C). So a concise but statically typed language (Scala?) might be a better choice than either Java or Python.

forensic• 17 years ago

Most professions choose their best practices through a form of natural selection and evolution. Rookies try all the different ways and 30 years later they are Veterans. The Veteran then passes on the best ways he learned to the Rookies, who copy it, and then over the course of their career slowly improve on it as well.

Even the most "simple" task like scaffolding (for instance) has heaps of "craft" in it. What research team decided that scaffolders should thread scaffold clamps between their fingers when they carry them? There is a subtle technique to this that has been passed down through generations.

This is how all professions learn the tricks of their trade. People try stuff, figure out what works best for them personally, and then pass that knowledge down to the rookies. Certainly with programming this happens at university - I've had profs who give practical advice like, "Make sure your program is always capable of compiling throughout the entire dev process." This is folk knowledge yes, but it's practical folk knowledge that HAS been tested over the course of careers by many people.

It is not economical to have many dedicated research scientists trying to figure out these things. They are figured out by people as they do their job.

The most important thing is mentoring. Paul Graham widely shares the things he has learned from experience and those things HAVE been legitimately tested because he has tried ways that failed and tried ways that succeeded. Just because it is not double blind does not mean that it hasn't been tested.

groby_b• 17 years ago

It is not economical to have many dedicated research scientists trying to figure out these things. They are figured out by people as they do their job.

... it is not economical to have many scientists trying to figure out the cures for illnesses. They are figured out by people as they treat other people.

Is that what you really want to say? Basic science does not have any immediate economic pay-off, no. But it often pays in spades later on.

because he has tried ways that failed and tried ways that succeeded

These are ways that worked for PG - and we have no idea if luck is involved. See his latest article - luck plays a large role in all startups.

I'm not saying mentoring is bad. In fact, it's pretty much the only thing that produces any results right now. But creating a solid foundation for the engineering side of computer science doesn't strike me as a bad idea...

known• 17 years ago

Selling Software != Selling Consulting

Hence it depends on type of the company.

What we actually know about software development, and why we believe it’s true

Related Stories