efarrer

Hi, I'm the author of the blog post and the paper. I certainly do not argue that static typing obviates unit testing. I agree that both are valuable together. I did however find examples where individual unit tests could be deleted because they didn't test anything that wasn't covered by the static type system. In other words static typing didn't eliminate the need for unit tests (for catching bugs), but it did reduce the number of needed tests (in real world code).

Hi, I'm the author of the blog post and the paper. Every bug that was caught by the Haskel transition was verified to exist in the original Python. If I couldn't reproduce it in the Python, I fixed my translation. More information on my methodology can be found in the paper. https://drive.google.com/file/d/0B5C1aVVb3qRONVhiNDBiNUw0am8...

Hi I'm the author of the blog post and the paper. In the paper I provide a reference to such a claim by proponents of dynamic typing. From my paper: "Because some error detection can be done by both unit testing and static type checking, some proponents of dynamic type checking claim that static type checking is not needed [3]."

and the reference:

[3] J. Spolsky and B. Eckel, “Strong Typing vs. Strong Testing,” in The Best Software Writing I. Apress, pp. 67–77, 2005.

I do agree that development time is an important factor and one that I didn't address simply because that would require a different type experiment. I hope that researchers look into that. I do also think that in addition to development time that overall maintenance time is considered. For example its plausible that dynamic languages are faster to develop, but take longer to make changes due to it being harder to understand, and changes aren't guided by the type system. It is also possible that dynamic languages are both faster and have lower TCO. To be clear I have no idea which is faster, and which has a lower TCO. I'd love to see some scientific evidence on development time and TCO.

I agree there are generally trade offs, that's essentially the results of my "60 pages of pointlessness" paper. If you did have time to read the paper you would notice a reference to this book http://my.safaribooksonline.com/book/software-engineering-an... that is an agument for unit testing instead of static typing. I think we need to use the scientific method in computer science and not just base our ideas of intuition, belief or absolutes like "It's never so black and white".

Thanks for the compliment, one of my goals is to put the science back in computer science.

It seems like you're two criticisms are as follows: 1. The sample size is too small. 2. The quality of the Python code isn't that great.

I (the author) would like to address both of these. First of all I completely agree that more research needs to be done. I mention this in the paper. I have provided a data point not a proof. It took me a couple months of several hours a day to the do the translation, I hope more people translate more programs to see if the results hold in the face of a larger data set. Second I agree the quality of the Python code isn't that great. I wanted to see whether unit testing obviated static typing in practice. In order to avoid selection bias I choose the projects at random. I picked the first four projects that were < 2000 lines of code and that had some sort of unit testing.

I believe that my methodology, nor my conclusions are flawed but that all should remember that a single experiment does not make a scientific proof. I hope that others will try to replicate this experiment on many more code bases. If they do it will be interesting to see the results.

I (the author) appreciate the feedback. I believe that many of your criticisms are addressed in the actual paper. First of all I completely agree that my sample size is too small for a conclusive proof. I mention in the paper that I hope that others will try and replicate this experiment on other pieces of software. I do think it's appropriate when conducting an experiment to publish a conclusion, not that the experiment will constitute proof (or an established scientific theory), but as a conclusion to the study that others can try to confirm or refute.

I also mention in the paper that it would be beneficial to conduct this experiment using different type systems for the reasons that you stated above.

The argument against static typing that I was testing didn't mention any particular type system nor any particular dynamically typed language, it was a general argument that stated that unit testing obviated static typing. Because the argument was so general and absolute I felt that any static type system that could be shown to expose bugs that were not caught by unit testing would be enough to refute the argument. I was not trying to prove that any type system would catch bugs not found by any unit tested software. The paper also points out that I'm trying to see whether unit testing obviates static typing in practice, in theory you could implement a poor mans type checker as unit tests, but my experiment was focused on whether in practice unit testing obviates static typing.

Finally I believe that my conclusion in the paper was at least a bit more modest than that of the blog post. The lack of apparent modesty in the blog post was caused more by a lack of ability on my part to accurately summarize than an inflated sense of accomplishment and self importance.