The Effectiveness of Test Driven Development (With Case Studies)

http://grokcode.com/439/test-driven-development-and-the-meaning-of-done/

by grokcode • 16 years ago

40 31 16 years ago

JoeAltmaier4• 16 years ago

"If we control the bug rate, is TDD faster?" is the central question. I assert that TDD is the Cause of the bugs, so the question is nonsense.

Code should be developed with as large a view of the design as possible. TDD restricts development (by definition) to satisfying the tests. The effect is of 'moving the bugs around' since the existing tests rarely completely describe the whole design. You get into a loop of "oh, yeah, well we Really meant to write a test to cover that case too", repeated ad-nausuem.

prateekdayal• 16 years ago

I would argue that TDD makes you think more about the design of your system. Also, it makes it easier for you to refactor (as you don't worry too much about accidentally breaking something) and hence keep the code quality high over time

kerkeslager• 16 years ago

In addition to making you think more about your design, I'd say it makes your design inherently more flexible. Once your design is flexible enough to do two different things, it's much more likely to be able to do three things than if it could only do one thing. And when you do TDD, your design is always capable of two things: fulfilling the requirements and testing.

ericHosick• 16 years ago

I think one problems with TDD is that it doesn't lean towards testing bugs in a feature/scenario (describing the whole design). That is, TDD focuses on testing a unit and it is effective/fast at testing the fringe cases of units.

Behavior Driven Development really shines in the aspect of "completely describing the design" as it focuses on driving development against a well defined behavior (using scenarios). BDD lets you know where the bug is: in the code or in the specifications (the behavior as specified by the customer). So there is a lot less "chasing around". If you are trying to cover behavior of a system only using TDD then I agree you can end up chasing around a bug.

However, both are necessary and highly effective when used together.

newt6• 16 years ago

I'm not sure what you mean by "TDD is the Cause of the bugs" - the TDD projects had 40-90% fewer bugs. Even if their remaining bugs would have been caught by test coverage of things that occur in live use, that's still a good result, right?

JoeAltmaier4• 16 years ago

Well, sort of. TDD defines bugs as "things the unit tests catch". And the unit tests are very narrow.

So yes, you record fewer bugs, fix those narrow cases quickly and pretend you're done.

Then it gets to the field, and whatever you didn't catch becomes a 3-alarm fire.

I vastly prefer shakedown tests of the sort where you launch a 'bot army' on cloud servers to thrash the app. Especially if there is Any network/client/server component.

Ramone• 16 years ago

The article misses an important point: People who've done TDD know that it is the easiest/fastest way to unit-test. Few aspects of programming are more painful than trying to unit test code after the fact, especially if it was written without testing in mind.

Retric4• 16 years ago

I fail to see how unit tests are required for a shippable product. For a wide range of things, buggy and shipped is better than clean and ready in 3-6 months.

steveklabnik3• 16 years ago

But is buggy and shipped in 3-6 months or clean and ready in 3.25 to 7 months better? (that's what the time increase numbers are from TFA).

brown9-2• 16 years ago

This seems like a false choice to me. Why are these the only two choices?

billmcneale• 16 years ago

It's not the fastest way, by far. It tends to lead to more testable code, sure, but it's not fast.

For example, TDD encourages code churn: you write tests for a lot of throwaway code.

Personally, it usually takes me two/three attempts before I'm satisfied with the version of the code that I have. Writing tests for these throwaway attempts is a waste of time.

As you become more experienced in writing tests and testable code, you realize that you no longer need TDD to write code that can be easily tested, so I see TDD more as "training wheels" for beginners.

100k• 16 years ago

I think TDD is useful, but practitioners should not be dogmatic about it.

As I discussed in my talk "Testing is Overrated", programmer testing finds a different class of bugs than manual testing or code reviews.

One of the problems is that programmers are not very good at writing tests. They are much more likely to test "happy path" scenarios. Steve McConnell reports in Code Complete that mature testing organizations have 5 times more dirty tests than clean tests, while immature testing organizations write 5 clean tests for every dirty test.

Another big problem is that unit tests are never going to show you that your software blows. Only user testing can find that.

If you want to ship good software, I think you need to do a combination of code reviews, unit tests, system tests, user studies, and manual testing. Different organizations will find a balance that works for them.

That last is a key point. I like writing tests, but I hate the dogmatism that if you're not writing tests you're doing it wrong. Obviously, good software has been written without tests, and buggy crap has been fully TDD'ed. In the end, the team matters more than the methodology.

http://www.scribd.com/doc/8585284/Testing-is-Overrated-Hando... http://www.scribd.com/doc/8585444/Testing-is-Overrated

thesz2• 16 years ago

There is interesting video from RSC2010: http://video2010.scottishrubyconference.com/show_video/11/1

"Dependent Types: A Look At The Other Side" by Larry Diehl

It talks about using Types for Driving your Development. So it is TDD, but without tests, without runtime and programmer's overhead of them.

When using TyDD, you isolate core that should go as bugless as possible and write DSeL for it in language with strong type system. The talk above tells about Agda, but we and some number of other companies in the world (Eaton http//eaton.com/, Echo http://aboutecho.com/ - from the top of my head) successfully employ Haskell for that task.

That way you shoot two birds with one stone: you get provably correct software and you get an ideal language to express your core problems.

Best of all, you get free "requirement tracking" - whenever your requirements change, you express them in types and then apply changes where they should, guided by friendly compiler. Still getting provably correct software (modulo types) at the end.

mdaniel• 16 years ago

That was "interesting" in that I like expanding my horizons, but it was a terrible presentation. Plus, it looks like Haskell and I got the impression from his lead-in that he was going to introduce a new language.

However, I actually came here to say something else about his material: if it is harder to decompose the business requirement into Haskell than it is to write monkey tests for one's Blub language, I am not certain anyone wins when going with a provably correct implementation.

thesz2• 16 years ago

He talked about Agda2.

My experience using advanced type system tells me that you can get away with functional tests. Types, also, takes much less code space-programmer's time than tests.

So I do not see how they could be harder to use. Certainly types will be unusual, but not much harder.

gasull• 16 years ago

It's down for me. Here's the link to Google cache:

https://webcache.googleusercontent.com/search?q=cache:grokco...

joevandyk• 16 years ago

Anecdote: I'm not smart enough to design software without having an idea of how it can be automatically tested. My design has to prove that it can be easily tested (both via unit tests and integration tests), and I've found that writing some of the tests up front helps with that.

The Effectiveness of Test Driven Development (With Case Studies)

Related Stories