Experiment: Unit testing isn't enough; You need static types, too

http://evanfarrer.blogspot.ca/2012/06/unit-testing-isnt-enough-you-need.html

by fpgeek • 14 years ago

189 267 14 years ago

evanfarrer.blogspot.ca

timruffles• 14 years ago

I wish more of these open questions in coding (and everything) had people making such a good effort to research them! There never is a simple answer, or they'd not be an open question.

Really inspiring, painstaking piece of work.

efarrer4• 14 years ago

Thanks for the compliment, one of my goals is to put the science back in computer science.

timclark• 14 years ago

Interesting, but worth remembering, as Rich Hickey says, every bug has got past both your unit tests and your type checking.

barrkel2• 14 years ago

That statement, though, is almost content-free. It's a truism. If you don't have type checking, then every bug has gotten past your unit tests; but some proportion may have been prevented with type checking. And vice versa. The statement doesn't say anything about the value or non-value of unit tests or type checking.

mike_ivanov• 14 years ago

No, it's not content-free, at least when put back into context.

It's like coffee. If your coffee is shitty (complect), then no amount of sugar (unit tests) or cream (type checking) will make it any better.

Flow2• 14 years ago

Speaking of Clojure, I've recently started to learn it since I like Richs reasoning about state, time, value and so on. His speeches are nothing short of fantastic in my opinion.

Anyway, it occurred to me that with Clojure it could easily happen that I trade bugs caused by state problems to bugs cause by type problems.

wtetzner2• 14 years ago

> Anyway, it occurred to me that with Clojure it could easily happen that I trade bugs caused by state problems to bugs cause by type problems.

With Haskell, on the other hand, you may not need to trade :)

drharris• 14 years ago

It is definitely refreshing to see some actual evidence in something where arguments tend to be based on speculation, experience, or opinion. Now, we just need someone to research Emacs v. Vim, Tabs v. Spaces, etc.

crazygringo• 14 years ago

Don't forget about braces versus indents, and above all, semicolons or not! :)

On second thought, let's forget about them after all...

Deestan3• 14 years ago

Any development environment where there exists a rule on Braces versus Indents is a place I'd stay well away from.

Put braces where they make the code readable. Use indents instead where they make the code more readable.

And if you do need a rule, make sure it's based on actual need. I.e. start the discussion with "the past two months we've had 4 non-trivial bugs which could have been avoided if we enforced braces.", not have a bunch of people bicker about their Personal Preferences (usually goes by the name "Best Practices").

SoftwareMaven• 14 years ago

Braces really are evil, just because of how much time is spent bikeshedding about them. In what universe will a program function better if braces are on the end of the line or the next?

That's one thing I really like about go. It's nearly as opinionated about its braces as Python is about its not-braces (almost...).

praptak• 14 years ago

> Tabs v. Spaces

Not exactly this, but there exists "Program indentation and comprehensibility" http://www.cs.umd.edu/~ben/papers/Miara1983Program.pdf which tries to tackle the "2 spaces vs 4 spaces vs 8 spaces indent". Not very good tho, since they seem to add superfluous indent levels like a separately indented "then" after "if" (Pascal), which effectively doubles the actual indent of the semantic block.

glenjamin• 14 years ago

I applaud the effort to try and dissect the problem scientifically.

If a whole program has 1 bug due to being implemented with dynamic types over static types, and that bug has gone unnoticed, then it can't be particularly important.

The other common proposition is that dynamically typed languages are faster to write in than statically typed languages. If this is true then we need to compare the saving in development time with the cost of the bugs which go undetected.

My gut feeling says that this line of analysis is never going to prove that static typing in inherently "better".

cageface12• 14 years ago

What? He specifically says that many of the bugs he discovered were exploitable. Just because you missed them doesn't mean some hacker won't.

Deestan3• 14 years ago

> The other common proposition is that dynamically typed languages are faster to write in than statically typed languages.

This part is also very tricky, as most of people's hard-earned experience with this is (almost by definition) old. In recent years, type inference has reduced the type-caused slowdown immensely. E.g. in Haskell you can (+) write large statically typed programs without specifying any types at all - they are inferred by the compiler.

(+) But please don't.

Dn_Ab3• 14 years ago

Heh I had to parse your sentence twice as (+) is a function in Haskell.

IsTom4• 14 years ago

Recent years? 70s more like.

ML way is to not write type annotations unless they're needed, not that I agree with it, but it certainly works in some way.

johnkchow2• 14 years ago

From my personal experience, the saving in development time is very much real.

I used to work on .NET, and you'd have to program against crazy, non-intuitive patterns in order to have your code "clean" (I hated IoC containers as well as writing all that boilerplate code for Attributes. And for Java, remember that hilarious post about Factory-Factory-Factory patterns http://discuss.joelonsoftware.com/default.asp?joel.3.219431?)

I'm not saying statically typed languages are bad. Type errors always bite me the ass in Ruby, but it's a small price to pay (IMO) for better maintainability.

EDIT: By the way, it sounds like I'm an ignorant dynamically-typed lover. I'm not, I still yearn for those type safety net, but I'm just speaking from a pragmatic perspective.

cageface12• 14 years ago

The kinds of static type systems you find in C# & Java are too primitive. Something like Haskell with perhaps a little less religion about mutability is a whole different story.

Dn_Ab3• 14 years ago

Using Java and C# as an example of typed languages is like using Youtube comments as a representation of humanity.

Once you get a hang of them, writing in any modern statically typed declarative language enters you into a state such that code flows out effortlessly. It is a wonderful feeling.

pragmatic• 14 years ago

I can't speak for/to java but C# has come along way.

I used to feel just like you did (at the asp.net 1.x) days. I went to Ruby on Rails for awhile. But once NET 2.0 (generics/nullable types) and then 3.5 (LINQ), I was hooked on C# again. We also have the dynamic type now.

I want the features a statically typed language gives me with minimal boilerplate.

It's hard to give up IDE features like "find everyplace this method is called" and know that you found all of them (disregarding reflection, etc and assuming private methods).

Visual Studio will do this type of thing for Python now. We aren't terribly far away from having the benefits of both.

I really like python as a scripting language. I just don't feel as good about scripting languages in big projects as I do a modern language like C#. (Just my humble opinion, no science).

papsosouid20• 14 years ago

>From my personal experience, the saving in development time is very much real.

But your personal experience is comparing expressive languages to unexpressive languages, and then making the mistake of thinking you are comparing dynamically typed languages to statically typed languages. What does .net's ugly (in your opinion) API have to do with static typing? Factory-Factory-Factory patterns is a joke even in java, but it also has nothing to do with static typing at all.

>By the way, it sounds like I'm an ignorant dynamically-typed lover. I'm not

Well, you are repeating the most commonly used and easily debunked strawman, so it really does sound like ignorance. Have you used a modern statically typed language? If not, by definition you are in fact ignorant.

bad_user3• 14 years ago

     I hated IoC containers as well as writing all
     that boilerplate code for Attributes

C# and Java are only representative for static languages because they are mainstream.

My static language of choice is currently Scala. It is less verbose than both C# and Java (for example you never have to generate setters/getters). Also, whenever I work with Java I always work with projects such as Lombok / Guava.

And I have no need for "IoC containers". This is partly because of Scala, but also because of the libraries I'm using. Currently I'm using DropWizard for the development of web services, which is just a thin wrapper around Jetty + JAX-RS. Because the resources (commonly known as controllers in MVC frameworks) are configured as instances (not classes specified in web.xml), then IoC is easy because you are free to do the bindings yourself.

IMHO, the necessity for IoC containers in C# / Java has risen from poorly designed frameworks. So you may not have a choice in the matter, however using something like Google Guice in combination with Scala is much nicer, because of Scala's mixins in combination with structural typing.

specialist• 14 years ago

Java, XML/XSL/XSD, and design patterns (gone wild) all hit about the same time.

Most (but not all) of my pain slinging Java come from misuse of XML and design patterns. Now we're on to services (SOA). Yippee. It's just a socket, marshaling, retry logic. Does it really need its own ecosystem of consultants, conferences, and books?

Java the language could be more terse with type inference, string literals, default visibility modifiers, and some other syntactic sugar. (I'm pretty optimistic about Ceylon as a successor.) But 18 years on, for the most part, I'm still very pleased with Java. Except autoboxing and annotations; garbage.

Java the platform (JDK, J2EE, java.util., collections, javax., misc APIs) should be burned to the ground, plowed under, and complete do over. And any one who says "factory method" gets keelhauled.

tikhonj5• 14 years ago

An interesting thought: a good type system can actually make a language more expressive.

A perfect example is QuickCheck. QuickCheck allows you to write complicated tests very simply by relying on the type system. You just write out the invariant and the type system automatically figures out which random generators to use to run the tests.

QuickCheck has been ported to a bunch of other languages, but it's more complex and seems harder to use in dynamically typed languages as compared to Haskell.

A simpler example in the same vein is Haskell's read function. Essentially, read is the opposite of toString--it goes from a string to some value. The beauty is that you never need to specify what type you're parsing; it can figure out what type it needs to be thanks to the type system. So instead of having a bunch of functions like parseDouble and parseInt, you have a single read function. This also makes the library prettier by maintaining the symmetry between show and read (toString and fromString).

pimentel2• 14 years ago

How can you translate from python to a static language, when the code is written for the interfaces, and not types? How willl you translate a function receiving a (possibly custom) iterable, when the function doesn't care about the type, but just whether it implements a next() method?

sbmassey2• 14 years ago

Haskell type classes are sufficiently flexible to represent things like 'does this type have a next method' without having to instrument the actual type.

pimentel2• 14 years ago

But could that kind of type system be considered similar to the usual Java-like little-flexible static typing? I think I would prefer a static vs dynamic implemented as Java vs Python (since those are usually the subjects on every one of these discussions)

dons3• 14 years ago

> the function doesn't care about the type, but just whether it implements a next() method?

You're describing an "existential" type.

dllthomas10• 14 years ago

... or a typeclass.

pja2• 14 years ago

Hindley-Milner type systems to the rescue! (With a sprinkling of Haskell style type classes.)

  f :: Iterable a => a -> b
  f x = (do stuff with x...)

f is defined to be constrained to accept only on types which implement the Iterable interface (however that's defined) as it's first argument and the compiler (or interpreter) will enforce that constraint.

emillon• 14 years ago

> Hindley-Milner type systems to the rescue!

You're implying that all type-systems derived from HM have a form of bounded polymorphism. They're not, for example OCaml does not have type classes (you could probably encode a lot into objects, though).

troels• 14 years ago

There's a difference between the implementation type and the protocol type. Or as Java calls them - Classes and Interfaces. Unfortunately most classical OO language collate the two, causing much confusion.

Symmetry2• 14 years ago

Haskell's typeclasses are very close to this, but even closer is Go. You just define and interface and the functions that require it, and the compiler goes off and check's by itself whether the type you're passing in satifies the interface - no need for you to annotate it yourself. You lose Haskell's nifty "deriving" feature, but it's even closer to Python's duck typing with compile time checks.

JustinJ70s• 14 years ago

Scala's type system can handle this as it support type safe duck typing.

cageface12• 14 years ago

I'm convinced that dynamically typed languages are a transitional technology that will be superseded once we develop type systems that are both usefully strict but also flexible.

After over ten years working in dynamic languages I'm very happy to have a compiler on my side again.

gnuvince4• 14 years ago

What dynamic language were you using and which language are you using now?

cageface12• 14 years ago

I pretty much lived in Rails from 2004-2011 but did a lot of Python and Perl before that.

These days it's mostly C++/Obj-C but I'm keeping an eye on Haskell for iOS.

ocharles2• 14 years ago

Amen, I've spent the last 3 years of my career working on an increasingly complicated Perl project and frankly, I've had enough. My pet projects now are all Haskell and I can't believe how fun it is.

I spend most of my time these days fixing bugs and regressions due to the sheer scale of the project, and I'm a running meme at work for saying "a type checker could have caught that!". I can't imagine going back to a dynamic language now.

MatthewPhillips2• 14 years ago

Are you having fun because you switched to a statically typed language or are you having fun because you switched to a functional language? Would you be having less fun had you chosen Clojure?

zemo2• 14 years ago

>type systems that are both usefully strict but also flexible.

that's basically the design criteria behind Go's type system.

Peaker13• 14 years ago

I think Go's type system is decades behind the state of the art. I can't believe they repeated the mistake of Boolean Blindness [1]. Sum types and pattern-matching are crucial for useful strictness with flexibility. This results in funny things like encoding the optional error result in Go as a type product rather than a sum, allowing reading of a result even if it does not exist due to an error.

[1] http://existentialtype.wordpress.com/2011/03/15/boolean-blin...

dochtman• 14 years ago

Seems to me that that's what static typing proponents have been saying for years (if not decades), and it still isn't true.

That said, I'm currently hacking on a compiler to see if I can come up with a design for a static language that's almost as pleasant to use as a dynamic one, so I'm not completely hopeless. But it certainly seems like the data point that until now no such superior type system (strict but flexible) has become widely popular should not be underestimated.

IsTom4• 14 years ago

What you are looking at are not two variables (strict, flexible), but also "ease of use". Arguably dependent-typed languages are most strict, but also most flexible: you can write type of a function that only accepts primes on input. They are hard to use though. Dynamically-typed languages are flexible and easy to use, but not strict. Java-likes are strict and easy to use, but they're very unflexible.

cageface12• 14 years ago

That's because it takes years or even decades to really get this right, like a lot of other sophisticated technology. Look how long it took to get the JVM to where it us today. This is fundamental research and hard stuff.

swah3• 14 years ago

So we're talking about type inference - which way are you going?

akkartik2• 14 years ago

I haven't heard of anybody suggest that static languages should allow lists to contain arbitrary types. Hmm, do tuples cover all the use cases for that feature?

IsTom4• 14 years ago

If you want a list of heterogenous "things" you either a list of "I don't care about what's on list" and then it should work without much in languages like Haskell, ML and the rest or you actually have list of things with a common property, such as "I want to be able to turn these things into strings and sum them" or "These are either numbers or strings" -- all functional languages work well with the latter through ADTs and the first one can be solved easily in Haskell with existential types (I don't know about other languages)

afrozenator6• 14 years ago

The author really needs to be complemented on rewriting swathes of code from Python to Haskell.

In Google, in my project, we've had runtime errors in Python code, due to wrongly spelled variables(although that is a different problem), and type error, something a compiler would have caught.

Strong type checking is something that I truly like about Haskell and OCaml, I'm reasonably convinced that once my program has passed the typechecker, it is logically correct. Though debugging in Haskell is truly a different ballgame altogether (I'm a Haskell noob).

I'll stop here lest this turns into a flame war.

16s• 14 years ago

pylint can help with misspelled variables and type errors. I started using it recently and love it. I still love my C++ compiler though and would not trade it for anything else.

riffraff2• 14 years ago

pyflakes also detects typos easily, and it's quite useful. In languages that don't catch anything you can usually still use tools for static verification.

As another example, Java may let you get NullPointerException, but FindBugs detects a lot of those.

kingkilr7• 14 years ago

    I'm reasonably convinced that once my program has passed the typechecker, it is logically correct

Yup, this is one of my favorite things about Haskell, that's how I know that http://bpaste.net/show/32033/ is a totally correct program.

dons3• 14 years ago

If you're hoping to catch a specification error, don't use a type like `Integer -> Integer`, which doesn't capture the specification except in a most general sense.

Just as you should write good tests, that actually test for useful properties, so you should write good types -- and get useful proofs back from the compiler as a result.

drunkpotato• 14 years ago

He qualified with "reasonably convinced", and you countered with an unreasonable example. That is the definition of attacking a strawman.

afrozenator6• 14 years ago

I forget whose quote this is - The computer is a wonderful machine, it does what I tell it to do, not what I want it to do.

Your point taken, but that is a problem in programming language X, which has Y type system, for all permissible values of X and Y.

pyre3• 14 years ago

  > we've had runtime errors in Python code, due to [...]
  > type error, something a compiler would have caught

Are wrongly spelled variable names and type errors the only runtime errors that you get? If not what percentage are they?

Personally, I think that people tend to obsess about the specific type of errors because the 'solution' (static-typing) is something that already exists, whereas there is not easy solution to other types of flaws.

Peaker13• 14 years ago

In Python, there are a lot of errors like "NoneType has no attribute '...'", and those disappear too.

Missing imports and redundant imports also go away.

Lots of lots of invariants in the program can be encoded as types, too, so any bugs relating to them go away too.

When you want parallelism, you get useful guarantees about not changing the deterministic result you had before you added parallelism.

I used to use Python, but after Haskell, there's no way I'd go back...

afrozenator6• 14 years ago

Yes, probably that is my confirmation-bias at talk. But more than anything it is the frustration that the whole make into a par file and deploy it on a production system, only to find just moments later that the binary isn't up because of a typo is frustrating in any language.

kingkilr7• 14 years ago

If you can convert a program from one language to another (which is non-trivially different), in the time it takes to complete your masters, I'm pretty sure it wasn't a very interesting program. Further, the quality of the developers is going to play a large role in how effective any tool (and make no mistake, static typing is a tool) is. This is not intended as a disparaging remark to the authors, but in the 30 seconds I spent reviewing each of these code bases, I was totally unimpressed: none of them seemed to follow PEP8, and several of their test files weren't even unit tests, they were just a random scrip that appeared to exercise a tiny part of the codebase. I therefore conclude that the methodology used in operating this experiment was flawed and, consequently the conclusion cannot be taken as scientifically valid.

scott_s6• 14 years ago

I have not looked at the programs, but programs don't have to be long to be interesting. There are many "interesting" programs under 100 lines of code - and they can be important if they form the kernel of a larger program.

Also, the author responds to a similar point in his comments: http://evanfarrer.blogspot.com/2012/06/unit-testing-isnt-eno...

kingkilr7• 14 years ago

The author doesn't really respond to the point, he simply says they were non-trivial in complexity, which may be true, but that doesn't mean they're non-trivial in their dynamacism (is that a word?). Moreover it in no way responds to the claim that these codebases aren't very good.

efarrer4• 14 years ago

It seems like you're two criticisms are as follows: 1. The sample size is too small. 2. The quality of the Python code isn't that great.

I (the author) would like to address both of these. First of all I completely agree that more research needs to be done. I mention this in the paper. I have provided a data point not a proof. It took me a couple months of several hours a day to the do the translation, I hope more people translate more programs to see if the results hold in the face of a larger data set. Second I agree the quality of the Python code isn't that great. I wanted to see whether unit testing obviated static typing in practice. In order to avoid selection bias I choose the projects at random. I picked the first four projects that were < 2000 lines of code and that had some sort of unit testing.

I believe that my methodology, nor my conclusions are flawed but that all should remember that a single experiment does not make a scientific proof. I hope that others will try to replicate this experiment on many more code bases. If they do it will be interesting to see the results.

dmethvin• 14 years ago

The lack of static types really comes into play not inside a library, but in the interface between the external code and a library. Note that at least a few of the bugs that were found involved invalid API inputs. From the library writer's view it's not a bug because those values are not in the domain of defined behavior. From the caller's view it's a PITA that the library doesn't yell at them when they pass it garbage. Of course, they could find the problem with unit tests but the further up the food chain you get the more scarce unit tests tend to be.

jknupp• 14 years ago

The author's interpretation of the argument in favor of dynamic languages seems purposefully naive. I don't think that any proponent of dynamic languages or unit testing claimed that the mere presence of unit tests guaranteed bug free code or that it was impossible to have type related errors at run time if you have unit tests. It's a more nuanced argument asserting that the benefit to programmer productivity when using dynamic languages outweighs the cost of potential type related errors not possible in a statically typed language. Whether it has merit or not is the question I hoped this paper would answer.

soc883• 14 years ago

I wonder how long it takes until the first one picks Java to argue against typed languages.

AngryParsley2• 14 years ago

I think most people would agree that these two circles overlap on a Venn diagram:

    (    Bugs found by unit tests (     ) Bugs found by type-checking    )

The disagreement is how much. Also, type-checking is free[1], while unit tests have to be manually written.

I'm glad someone spent a lot of time trying to answer this question, but I don't think it will affect my choice of language in any new project. I like to write code in the languages I like, and bugs be damned.

1. A common argument is that static-typed languages slow development. I'm not touching that land-mine.

swannodette2• 14 years ago

Via Philip Wadler, http://wadler.blogspot.com/2011/09/experiment-about-static-a...

akkartik2• 14 years ago

On HN: http://news.ycombinator.com/item?id=4082775

Goladus3• 14 years ago

> 1. A common argument is that static-typed languages slow development. I'm not touching that land-mine.

If you claim type-checking is free, the counter-argument is not that it "slows development." The counter is that type-checking is not free because it incurs measurable costs. You may sacrifice dynamic features, you may have to add declarations and type casts-- these are all costs whether they "slow development" or not.

(Simple solution: don't waste time trying to claim type-checking is free and just focus on the benefits.)

AngryParsley2• 14 years ago

I would feel bad editing my comment now that you've replied, but my original intent was to say that type-checking is "free." At some point the quotes were lost.

To get as close to the land-mine as I am willing: I like C. I like Python. I like Ruby. But most of all, I like using the right tool for the job.

Peaker13• 14 years ago

I am not claiming static typing is free.

But it is also untrue that you have to "add declarations and type casts". With type inference, you don't actually have to.

kingkilr7• 14 years ago

a) If you aren't touching that land mine you're missing the point. You can't say that one is free (except maybe it has a cost), and the other has a cost. That's just nonsensical.

b) Second, I'm going to argue that tests are actually free. And I think this because I don't care how good static type proponents think they are, I know they don't wait until it compiles and SHIP SHIP SHIP, they actually run their damned program. The alternative to automated testing is manual testing, not no testing.

wtetzner2• 14 years ago

> I'm going to argue that tests are actually free.

> The alternative to automated testing is manual testing, not no testing.

Actually, unit tests have different costs than manual testing. That doesn't make them free. For example, with unit tests, you now have more (possibly buggy) code to maintain.

cyrus_• 14 years ago

Many of the unit tests would have to be written in a simply-typed language like Haskell too -- the author notes that only a handful of tests could be entirely eliminated due by the rewrite. Probably worth further study.

jerf2• 14 years ago

In my admittedly-limited personal experience doing something similar, it's really hard to characterize this in a sane way. What you end up with is a pile of unit tests which are "really testing something", yet, some non-trivial percentage of the tests are still redundant to the type system. You feel like you can't throw it away because of the percentage that is a real test of functionality, but if you'd been starting from scratch with the stronger type system you'd have written fewer tests with very different focus.

It's hard to even come up with an example, but consider testing that an HTML generation library doesn't unexpectedly emit text unescaped. The pile of tests you write if you're in Python or Perl mostly translate to Haskell unscathed, in that each individual test still is testing something ("is the href attribute on <a> encoded properly? is the name attribute on <a> encoded properly? ..."), yet considered as a whole the tests have significant redundancy with the type system, because if you set the types up correctly there's a great deal fewer possible ways to screw up than there used to be.

x17• 14 years ago

Are we really talking about type checking or the larger circle of validation (of which type checking is just a small part)?

( Bugs found by unit tests ( ) Bugs found by input validation )

Or in other words...

String s = "lastname'; drop table user--";

...is still a perfectly acceptable string.

It seems to me that type checking is the simplest form of validation (are you an int, are you a String) and nothing more. It wont tell you if that int is positive or negative or if that string is an email.

When dealing with either static/dynamic languages I think more unit tests should be spent validating.

papsosouid20• 14 years ago

No, this is just common ignorance of static typing. That string is a perfectly acceptable String. But it isn't a perfectly acceptable Query, and you can't pass a String to the database, only a Query. In order to turn a String into a Query, it has to be passed to a function that escapes problem characters safely. You need to use such a function regardless of dynamic vs static typing, but static typing enforces that you always use that function, and can't forget and accidently submit an unescaped string to the database.

zmoazeni• 14 years ago

Context: My day job has been coding Ruby for years, and I'm a big fan of Haskell.

Someone was asking me about my thoughts on static vs dynamic. And I realized that I have two reasons for my tests a) I didn't do something stupid like mistype a method or variable name and b) validate my logic.

Sometimes I think it swings 20/80 and others 80/20. But either way, I know that's why I gravitate towards integration tests in Ruby, and that some of them would go away if it were statically typed.

Even after coding in Ruby all these years, I'm still just as afraid of (and susceptible to) problems in a).

mikeryan• 14 years ago

Also, type-checking is free[1], while unit tests have to be manually written.

The use of type-checking doesn't negate the need for unit tests. It just adds another layer of validation. The Unit Test "cost" is still there.

Peaker13• 14 years ago

You can write far fewer tests if you have good static validation.

For example, taken to the extreme, you can write 0 tests in Agda, and still have more assurances about correctness than if you had 100% coverage in a dynamically typed program.

davesims14• 14 years ago

The sample size of this study is statistically too insignificant to draw the conclusions given. The counterfactuals to the claims of dynamic type advocates were already true and provable, so even if the sample size of the codebases studied were statistically significant (not to mention vetted for quality of unit test as well as coverage) the conclusions are nevertheless trivial.

In addition, the hidden assumption is that all static and dynamic typing are created equal, i.e., since Haskell is statically typed and Haskell appears to have caught Python bugs that unit tests did not, therefore Java will catch bugs in a Ruby codebase, C++ will catch bugs in a JavaScript codebase, etc. Of course this assumption is gratuitous. Haskell in particular has a specific sort of type checking that is far different from Java's or C++'s, for instance.

Further, not all dynamic systems are created equal. Ruby, for instance, I think can be shown to require fewer lines of code to achieve similar functionality to, for instance, Java. Fewer lines of code, should in principle mean fewer opportunities for defects. Dynamic languages with metaprogramming features like Ruby's or Smalltalk's should in principle be able to eliminate more code duplication than an environment like C++. This aspect of dynamic languages should be taken into account, again with a statistically significant sample size, and weighed against bugs caught by static typing.

The study is interesting as a preliminary investigation, but the conclusions should have been much more modest, proportionate to both the sample size, in terms of % of production codebases and the extremely important idiosyncratic nature of Haskell vs. other static typed environments. Something like: "The study has shown Haskell's type system will catch some bugs not caught in an otherwise well-covered Python codebase. These bugs could in theory have been caught by unit tests, therefore it is recommended that when using a dynamic language, more care must be taken to cover these types of bugs."

That would have been a more appropriate and modest conclusion, consistent with the data, than the sweeping generalization "You need Static Typing."

papsosouid20• 14 years ago

>In addition, the hidden assumption is that all static and dynamic typing are created equal, i.e., since Haskell is statically typed and Haskell appears to have caught Python bugs that unit tests did not, therefore Java will catch bugs in a Ruby codebase, C++ will catch bugs in a JavaScript codebase, etc.

That assumption isn't hidden, it is made up. By you. The question was "can static typing catch bugs that made it past a decent (and common) test suite". The answer to that can drive interest in static typing, and thus more language with useful static type systems. Just because java has a crappy type system, doesn't mean we should be content with that.

davesims14• 14 years ago

> That assumption isn't hidden, it is made up. By you.

Not at all. The assumption is clearly implied by the conclusion of the study, which makes an unwarranted equivalence of all languages that have 'static type checking':

"The translation of these four software projects from Python to Haskell proved to be an effective way of measuring the effects of applying static type checking to unit tested software."

> Just because java has a crappy type system, doesn't mean we should be content with that.

I don't know what this means. If the study was meant to comprehend such a broad category as 'static type systems,' and from the explicit language of the study, it clearly was, then absolutely Java must necessarily be included. Otherwise, the study, as I noted, should have restricted its conclusions to a scope of Haskell vs. Python, with at most modest and well-qualified statements regarding the broader implications of static vs. dynamic in general.

davesims14• 14 years ago

I think I should clarify what I meant by "the conclusions are nevertheless trivial." Let's look at the key statement in the conclusion of the study:

"Based on these results, the conclusion can be reached that while unit testing can detect some type errors, in practice it is an inadequate replacement for static type checking."

As I've already pointed out, this seems to me an ambitious and over-reaching conclusion, given the scope of the study.

But, equally important, it is simply an example of something that was already provable. It should be axiomatic that in principle automatically-generated validation like that provided by static typing should in theory be able to catch type errors not caught manually in a dynamic context, either for reasons of human oversight or human error.

In other words, it seems to me that all that has been done here, is to provide a few concrete examples of what was already true and uncontroversial: auto-generated coverage of specific types of validations can be more comprehensive than some human beings will be in some environments and contexts. It has not shown that the perceived benefits of dynamic typing with good unit tests are outweighed by this fact, nor that, statistically-speaking, errors of this type are common enough to warrant a preference of static typing over dynamic typing with unit tests in all contexts.

wissler• 14 years ago

You make a good point -- I don't think any statistical study will ever be able to show that static typing is better.

I do think that a rational argument can show it, but my argument is too long to fit into the margin.

scott_s6• 14 years ago

I think that a study over a broad set of applications of considerable complexity could provide enough statistical evidence that most people would be comfortable coming to a conclusion. That study, though, would take a very large effort. Large enough that it may never be done.

ScottBurson• 14 years ago

I see both sides of this argument. The OP -- at least in this blog post; I haven't read the paper -- spends most of his time talking about how he's demonstrated the insufficiency of unit testing. For the purpose of that argument, it really doesn't matter that he used Haskell as opposed to some other type checker.

It's only in the last two sentences of his "Conclusion" section that he turns the argument around, and here is where he oversteps:

While unit testing does catch many errors it is difficult to construct unit tests that will detect the kinds of defects that would be programatically detected by static typing. The application of static type checking to many programs written in dynamically typed programming languages would catch many defects that were not detected with unit testing[...]

Clearly, this is overbroad. For starters, he should have used "could" in place of "would". And it wouldn't have been a bad time to remind the reader that Haskell's type system differs from those of other statically typed languages with which the reader may be more familiar.

I don't quite agree, though, that the conclusion is "trivial". Maybe I'm just out of touch, but I wasn't aware of a good test of how true the dynamic argument was in practice, as opposed to theory -- particularly claim #2.

efarrer4• 14 years ago

I (the author) appreciate the feedback. I believe that many of your criticisms are addressed in the actual paper. First of all I completely agree that my sample size is too small for a conclusive proof. I mention in the paper that I hope that others will try and replicate this experiment on other pieces of software. I do think it's appropriate when conducting an experiment to publish a conclusion, not that the experiment will constitute proof (or an established scientific theory), but as a conclusion to the study that others can try to confirm or refute.

I also mention in the paper that it would be beneficial to conduct this experiment using different type systems for the reasons that you stated above.

The argument against static typing that I was testing didn't mention any particular type system nor any particular dynamically typed language, it was a general argument that stated that unit testing obviated static typing. Because the argument was so general and absolute I felt that any static type system that could be shown to expose bugs that were not caught by unit testing would be enough to refute the argument. I was not trying to prove that any type system would catch bugs not found by any unit tested software. The paper also points out that I'm trying to see whether unit testing obviates static typing in practice, in theory you could implement a poor mans type checker as unit tests, but my experiment was focused on whether in practice unit testing obviates static typing.

Finally I believe that my conclusion in the paper was at least a bit more modest than that of the blog post. The lack of apparent modesty in the blog post was caused more by a lack of ability on my part to accurately summarize than an inflated sense of accomplishment and self importance.

davesims14• 14 years ago

Thanks for the response! I appreciate the effort you went to here, this was no small task you set yourself to.

I appreciate the clarification. I think now I see better where your emphasis was: the purpose of the paper was to refute an argument, and of course the level of burden of proof is different and far less in that case. I think this misunderstanding on my part is what caused me to call the conclusions 'trivial' -- too strong and dismissive language on my part anyway.

The irony is, you were attempting to do to the unit-testing-is-sufficient argument what I was attempting to do to what I assumed yours was: provide one counter-example to falsify a broad and generalized thesis.

That said, I think I would have liked to have seen your original unit-testing-is-sufficient argument punched up and qualified into something a little more reasonable and real-world. As you stated the argument, it seems like a straw man to me. It seems one could reduce your version of the argument to something like: "Dynamic languages with unit test coverage will always catch errors that statically-typed environments will." And of course this is far too broad and unqualified a statement, and that is precisely why all you needed was one counter-factual to refute it. You didn't even need a handful of Python programs, or 9 or 20 or 100 errors to prove your point. You only needed one, as you stated above. This is why the burden of proof for your thesis was so small, but also why, in my opinion, even with that reduced scope and more modest conclusion, we haven't really learned much.

As someone who has spent most of my career in statically-typed environments and the last 6 years or so mostly in dynamic environments, and also as someone who has made something like the argument you were attempting to refute, I have to say I would definitely never have made such a brittle and unqualified statement as the one you refuted in your paper. To put it more directly, I think I'm probably a poster-child for the kind of developer you were aiming your thesis at, and I don't feel that my perspective was adequately or reasonably represented. More importantly, having looked at the examples given in your paper, I may have learned a bit about the kinds of errors that Haskell can catch automatically that some coders might miss in a dynamic environment, but not much useful to me in my everyday work context.

I think a more reasonable version of the argument, but more qualified and therefore requiring a far larger sampling of code to prove or refute, would be something like: "Programs written in a dynamic language with adequate or near-100 percent unit test coverage, are no more prone to defects than programs written in a statically-typed language with a comparable level of unit test coverage."

I agree this is a very important conversation to have, and again kudos to the work you put in here. Obviously people have strong opinions both directions, and the discussion, however heated at various moments, is an important one, so thanks for this!

swannodette2• 14 years ago

Static types or static analysis?

  * KLEE: Unassisted and Automatic Generation of 
    High-Coverage Tests for Complex Systems Programs  
    http://llvm.org/pubs/2008-12-OSDI-KLEE.html
  * Erlang Dialyzer 
    http://www.erlang.org/doc/man/dialyzer.html
  * Datalog based systems 
    http://www.cse.msu.edu/~cse914/Overheads/mmcgill-java-static-race-detector.pdf

If you want static analysis hard coded into your language - what feature set do you want to support? The following support different styles of programming thus have very different static type systems.

  * Standard ML / OCaml
  * Haskell
  * Scala
  * Typed Racket
  * Qi

And there's still the question that some kinds of extremely useful programs are very difficult to write in popular languages with strong static typing. miniKanren, a flexible embedding of Prolog and Constraint Logic Programming into Lisp, comes to mind here. I've seen versions of miniKanren written in Haskell and it abandons the most powerful feature of miniKanren - it can be trivially applied back on the language it is written in!

IsTom4• 14 years ago

Dialyzer is not very good. We've been using it, but it rarely catches anything non-trivial. On the other hand Haskell has a better support for unit tests than many of dynamic languages.

gtani2• 14 years ago

http://www.artima.com/forums/flat.jsp?forum=106&thread=3...

http://tratt.net/laurie/tech_articles/articles/a_modest_atte...

http://www.tbray.org/ongoing/When/201x/2011/12/27/Type-Syste...

http://blog.jcoglan.com/2011/12/29/black-box-criteria/

http://www.stanford.edu/class/cs242/slides/Types.ppt

eta_carinae3• 14 years ago

The advantage of statically typed languages has a lot less to do with tests and everything to do with tools. IDE's can do very little when no type information is available, and most automatic refactorings require human supervision when performed on dynamically typed languages (read this for details: http://beust.com/weblog/2006/10/01/dynamic-language-refactor... ).

thebluesky2• 14 years ago

Tooling and performance are two largest advantages. Disadvantage is more verbosity, but it's a trade-off.

With dynamically typed languages I find that you still need to worry about types, but you have to trace through the code to figure out what type a particular variable is (esp. if you're not the only one working on a project). In a statically typed language that information is readily available.

Peaker13• 14 years ago

The claim static typing adds verbosity is a common fallacy.

Haskell code is more concise than most dynamically typed languages.

papsosouid20• 14 years ago

>Disadvantage is more verbosity

How so? Java is verbose, but that isn't due to being statically typed, it is just a verbose language. Statically typed languages with type inference are no more verbose than dynamically typed languages.

gnuvince4• 14 years ago

> Disadvantage is more verbosity, but it's a trade-off.

When I was learning Haskell, I wrote a program to count the value of a cribbage hand. The code was half the length of an equivalent JavaScript program.

The expressiveness of a language has more to do with the length of a resulting program than static or dynamic typing.

eta_carinae3• 14 years ago

> Disadvantage is more verbosity

Type inference cuts down the verbosity tremendously.

debacle4• 14 years ago

The better programmer you are, the less you need static types.

But a group of ten programmers create a codebase that is only as good as the worst programmer's code.

simias3• 14 years ago

No true scotsman much?

debacle4• 14 years ago

I don't know how you could contrive what I said to be cognitive dissonance.

If you have a team of N good programmers, you can probably write without static typing. If you have a team of N-1 good programmers and 1 bad programmer, you should really use a language static typing.

hackinthebochs3• 14 years ago

Let me guess, you love dynamic languages, right?

I really wish we would all stop ourselves when we're about to make a statement of the form "Great programmers do X (which I happen to do)". Without any rationale to back it up, its just ego-stroking.

njharman• 14 years ago

> "frequently cited claim by proponents of dynamically typed programming languages that static typing was not needed for detecting bugs in programs"

Who says that? There's trade offs, multitudes, in choosing paradigm / language. It's never so black and white (except for academics, and twits who like to argue more than code)

I did not read paper, don't have time for 60 pages of pointlessness.

zopa7• 14 years ago

> "Who says that?"

http://news.ycombinator.com/item?id=4137283 comes pretty close to making that claim, and that's just on this page.

> "There's trade offs, multitudes, in choosing paradigm / language. It's never so black and white"

Sometimes, it is. Most people don't use COBOL anymore, with reason; better languages came along. Programming is still a new field in the scheme of things. It would be strange if our languages were perfectly optimized, with no room for improvement without offsetting costs.

> "twits who like to argue more than code"

Some of us like to do both :) But go program; we won't stop you...

efarrer4• 14 years ago

I agree there are generally trade offs, that's essentially the results of my "60 pages of pointlessness" paper. If you did have time to read the paper you would notice a reference to this book http://my.safaribooksonline.com/book/software-engineering-an... that is an agument for unit testing instead of static typing. I think we need to use the scientific method in computer science and not just base our ideas of intuition, belief or absolutes like "It's never so black and white".

jhrobert• 14 years ago

There is a case for "optional typing". It's speed. I guess everyone agree on this one.

Better than "strict" typing, sometimes "hints" about the type would be enough.

ie: "ii often integer" versus "ii" or "ii integer"

From then on, those who favor strict typing would be happy, those who favor dynamic typing would be happy and those who favor "efficiency" (whatever this means) would be happy.

Let's close this silly debate.

6ren• 14 years ago

Regarding bugs, I think it's clear that static typing does detect more errors - the issue is whether it's worth the trouble.

I think a more significant difference is the inertia of the codebase, how difficult it is to change. Static types make it harder to evolve interfaces; but unit tests make it even harder.

But both the above are just fiddling: the real advantage of static typing is runtime speed; the advantage of dynamic types is development productivity. In the history of programming, the latter always wins.

Peaker13• 14 years ago

> But both the above are just fiddling: the real advantage of static typing is runtime speed; the advantage of dynamic types is development productivity. In the history of programming, the latter always wins

Have you used Haskell? Speed is just a (nice) bonus. Type safety is not about speed at all.

And I am much more productive in Haskell than I could be in a dynamically typed language. I can turn the code base inside out, and trust the type system to guide me to everywhere that needs to be fixed. When I compile it, it almost always works. I barely have to test anything.

dspeyer• 14 years ago

For the cases I was easily able to count, this is one bug per thousand lines of code. I realize that's a handwavy metric, but it's enough to say we're not talking about a ton of bugs.

On the other hand, successful conversion of these codebases is a very interesting result. Apparently static, at least static as sophisticated as Haskell, actually can express most of the idioms in standard python. This surprises me, as I've never seen a C++ program of significant complexity that didn't resort to void*s somewhere.

gouranga• 14 years ago

Slightly different view here.

I think the nirvana of typing is hybrid static/dynamic and it hit us with visual basic 6. I don't think anyone really noticed it though.

It supports traditional type checking by the compiler, runtime type inference and dynamic typing without boxing. Each case can be chosen at will. It supports all theoretical programs, supports unit testing and runtime assertions.

The least buggy software I've seen over the years was written in vb6 (by professionals, not the crap that haunts the web).

Now I'm not saying we should all switch to vb6 but some of the ideas may be worth investigating.

petercooper• 14 years ago

On a related topic, another interesting paper: http://www.cs.washington.edu/education/courses/cse590n/10au/...

anamax6• 14 years ago

It's almost always the case that applying a new set of tests finds more bugs so this experiment doesn't actually prove its conclusion.

Experiment: Unit testing isn't enough; You need static types, too

Related Stories