Evolutionary couplings between files reveal poor software design choices

http://ergoso.me/computer/science/github/software/evolutionary/couplings/2014/12/10/evsrc-evolutionary-couplings-reveal-poor-software-design.html

by armish • 12 years ago

242 82 12 years ago

ergoso.me

ColinDabritz• 12 years ago

This is an amazing approach to making some hard-to-understand aspects of software more visible. This sort of improved information is how software process and tools evolve over time. I look forward to seeing where this project goes.

barrkel• 12 years ago

A "correctly layered" app with UI view separate from UI logic separate from server-side logic etc. will show up as coupling, if commits are feature oriented.

There's certainly a hint as to where to look for bad coupling, but expected "coupling", like tests, need to be discounted.

masklinn4• 12 years ago

A "correctly layered" app with UI view separate from UI logic separate from server-side logic etc. will show up as coupling, if commits are feature oriented.

It would show some coupling commits, but e.g. bug fixes should/would be segregated to the relevant files, not spread across the system.

nostrademons3• 12 years ago

Basically bugfixes would be localized to relevant files, but features would spread across files.

The grandparent's making an important point in that you can't design a system such that all possible changes you might want to make are localized to one area of the code. Engineering is about trade-offs: if you rigorously separate view from logic from database, you make it harder to add features that must touch all three. Conversely, if you make each feature its own file and add in hooks to the view/logic/database layer so they call out to plugins, you make it easy to add new features but very difficult to understand what each layer as a whole is doing.

The best you can do is choose the ideal architecture for your particular project, in the particular point in time that you're working on it. That's why basically every software system needs to be rewritten as it grows up: the ratio of complete rewrites to new features to bugfixes to maintenance refactorings changes as the system matures and the requirements become more precisely known. It's also why we have a software industry; if there was one ideal way to design a system for all domains and all points in time, someone would go design it and be done with it, and none of us would have jobs.

richmarr• 12 years ago

Tests should have some coupling, but good tests need to change less frequently than the target code. Bad tests need to change every time. It seems like there's still value to explore there.

mempko• 12 years ago

I think it is a mistake to think of coupling caused by TDD to be a false positive. What this outlines really is that TDD will force you to edit two files instead of one for many changes. This is a clear indication of how TDD will slow you down.

pionar• 12 years ago

So does non-TDD. If you're fixing a bug, your tests should test for conditions that trigger that bug.

If you're modifying behavior, you will need to update the tests that break because of that, whether you write tests before or after.

masklinn4• 12 years ago

0. It has nothing to do with TDD

1. It's a tooling artefact, testing systems certainly don't have to mandate split code and tests. Rust's test framework allows tests in the same file as the tested code, the testing guide recommends that unit test live alongside the code they test[0] and the standard library follows this practice[1]. I'm reasonably sure you can also do so in e.g. py.test[2]

2. I'm not convinced editing two files slows you down, most editors and window managers will let you put both files side-by-side and trivially jump between them. Are java developers slowed down by having to jump between files?

[0] http://doc.rust-lang.org/guide-testing.html#the-test-module

[1] https://github.com/rust-lang/rust/blob/4deb27e/src/libcollec...

[2] by marking all python files as "test modules"

desdiv• 12 years ago

0. It has nothing to do with TDD

OP is referring to this line:

It also turns out, in both softwares, a majority of the couplings are attributable to Test Driven Design, where a source code is coupled to its test. So these are apparently false-positives I should take care of in the next version of the pipeline.

jxf• 12 years ago

This is a clear indication of how TDD will slow you down.

Not shown: the part where the critical production bug you introduced was caught by your test suite, thus saving you countless hours of agony, angry customers, and lost revenue.

nostrademons3• 12 years ago

Also not shown: the part where the feature that was exhaustively tested was removed a couple months after launch because requirements changed or the market shifted.

It works both ways, and probably one of the biggest skills to being an effective developer is understanding whether a piece of code you write is likely to be thrown away shortly or whether it's going to live forever and cause umpteen headaches for the maintainer. (This itself is surprisingly counterintuitive: I have seen high-priority code backed directly by an executive thrown away a week after being written because of shifting perspectives within the organization, and I've also seen a one-character typo in a "throwaway" migration script result in restoring a million+ users from tape backup.)

conistonwater• 12 years ago

"Not doing TDD" absolutely does not mean "not doing testing". You can definitely still test your software, even if you your development process is not test-driven like that. I can't quite understand why this is an issue.

chongli7• 12 years ago

Instead of writing tests in a separate file, why not express the same logic in the form of types in the lines directly above the code which implements that logic? This has the added benefit of making your code self-documenting and giving rise to powerful tools such as type-guided implementation inference and search.

tunesmith• 12 years ago

Sure, it will slow you down compared to the idea of writing only the source file... assuming the source file is as well-structured and bug-free as it would have been by writing the test. In which case, why would anyone ever write any tests anywhere ever?

briantakita2• 12 years ago

This is a clear indication of how TDD will slow you down.

This is a classic problem of externalities. You can only quantifiably judge what you are quantifiably measuring.

In this statement, you are only measuring the file couplings. I hope it's obvious that there are many other factors at play.

For example, let's look at a "typical" development iteration for a feature.

---

With TDD:

1. you write the automated test

2. run the test

3. if the test passes goto 6

4. edit the software

5. goto 2

6. finish

---

With manual testing:

1. edit the software

2. manually test the software

3. if the software does not do what you need it to do, goto 1

4. finish

TDD gives you a quick feedback loop, since the verification step is automated, at the cost of up front time spent on writing the test.

There's also automated regression testing, which is useful to prevent regressions when you change the software system.

---

When the software becomes complex:

* regression tests offer a quick feedback loop that scales almost linearly

* manual tests have a slower feedback loop and are often given to the QA staff, which involves communication overhead, scheduling, meetings, etc.

rverghes• 12 years ago

You'd probably see something similar when looking at a C/C++ project. I would expect significant coupling between header and source files.

The solution there was for future languages to combine the two. Maybe we'll see a future language combine unit tests and source into the same file.

Heh, maybe such a language would refuse to compile if public functions did not have an associated test.

gear54rus• 12 years ago

refuse to compile if public functions did not have an associated test

Sounds like my worst nightmare.

pmontra• 12 years ago

No need to wait. Rust has basic unit tests inside the source files doc.rust-lang.org/0.12.0/guide-testing.html

sliverstorm• 12 years ago

Editing more files is not strictly a bad thing. Files are an organizational tool. This means they have cost (overhead), and they have payoff (structure). There is always cost, there is always payoff. The trick is to find the "best" point in the curve, and you cannot do that if you focus only on the payoff or only on the cost, as you have done here.

__david__• 12 years ago

That looks really interesting, and I'd love to run it on my own projects. I'm curious though, why do you think you need to make it a web service? And why tie it to github? I would like to run it on a local git repository and output the results into a local file. That seems like a good small program.

jlarocco• 12 years ago

I'd also like to run this on our software, but being a web service tied to GitHub kills it for me.

armish8• 12 years ago

right now, it is not really tied to GitHub; but only expects you to give a git repository URL. Coupling it GitHub will allow more information on the system, for example knowledge on commits that resolve issues on the tracker. That is a long-term goal of this project.

bacongobbler• 12 years ago

It's not really tied to github. If you look at the source[1] it's just a `git clone` wrapped in a shell script.

[1] https://github.com/armish/evsrc/blob/master/scripts/evSource...

brysgo1• 12 years ago

I played around with this a bit in June. This is what I managed to come up with in the time I was interested in it:

https://bitbucket.org/brysgo/git-coupled

armish8• 12 years ago

cool! Thanks for sharing the link to your project. I am not really a Ruby expert, so wasn't able to figure out how you calculate the couplings. I will be more than happy to compare the results across these two tools.

krschultz• 12 years ago

There was another recent HN post that showed an analysis of IntelliJ's architecture using a source code analyzer [1]. Does anyone have more information on these types of tools? There seems to be a genre of tools that are used to inspect the architecture of a program, and I have no idea where to start learning about them.

[1] http://t.co/Ja6uOLRGkQ

armish8• 12 years ago

Thanks for the pointer; didn't know about that plug-in. As a long term Intellij Idea user, implementing this as a plug-in to Intellij is one of the things I would like to do in the short term. Would be really great to make the tool let you know about possible coupling as you work on the project.

Let me know if you can get your hands on a list of related plug-ins, I am really curios to see them in action.

RangerScience• 12 years ago

Huh. Could you also use this to flag something for a potential bug, if a historically coupled pair is not coupled in some commit?

danieltillett• 12 years ago

I actually think this might be the best use case of this proto-tool.

patmcguire• 12 years ago

Are there undocumented flags that you're running this with? I wind up with a series of graphs too densely packed to make sense of when I run it on angular.

armish8• 12 years ago

The PDFs that come out of igraph layout are not that pretty. The screenshots I used on the blog post are generated with Cytoscape 3.1.1, which gives better layout and better styling options. For Angular.js, I went with "partialCorrelations_0.3.sif".

nutmeg• 12 years ago

The first two times I read this headline I thought it was about fruit fly evolution.

armish8• 12 years ago

I know, right? I was really happy when I learned that this branch of the CS is also referred to as evolutionary, because I was inspired by the evFold approach, which is related to evolution in multiple organisms. This is a bit confusing for people coming from the biological science domain, but also nice that we share some terminology between two fields ;)

hcarvalhoalves• 12 years ago

Very interesting. Would like to see based on function/class/module instead of file though.

dkarapetyan• 12 years ago

Hold on. I have an interface file "interface.d.ts" and a whole bunch of other files reference it. Whenever I make changes to any files that depend-on/reference that file I of course also make changes to that file. This means that every file in my project is coupled to that file. How is that indicative of good or bad design?

tracker1• 12 years ago

Tight coupling is generally considered bad practice, it leads to more accidental variance and complexity. In general adding either polymorphism, or additional methods to a class are considered safer. I'm not saying in your case it was the wrong choice, or that cleaning up design is bad.

Generally if you have to change a whole bunch of related files when you change one, it's an issue with the design.

emmelaich• 12 years ago

Not sure you can know that without knowing the problem domain.

There's a tension there with one-and-one-place-only otherwise known as Don't-Repeat-Yourself.

adekok• 12 years ago

I'm running it now on a repository with 15 years of history, and ~15K commits. Let's see how it goes. :)

One thing I noticed is that the scripts are written to run once, and always do everything. It would be better to have a Makefile and dependencies, so that the you can run it multiple times, and only the changes are updated.

I'll see if I can push some fixes to github.

armish8• 12 years ago

let me know if you find points that can be improved; I would be more than glad to pull your changes in to the main repository.

Thanks for trying this out.

grandalf• 12 years ago

I think Rails should rename has_many to couples_many

Logical coupling crops up in lots of unexpected places as well:

<%= partial :foo %>

Partials are functions but with no clear argument signature, so they may be used sloppily with no obvious way of determining what (interface, state expectations) they are coupled to.

chacham15• 12 years ago

Am I the only one who disagrees with the premise that two files that tend to change a lot together indicate poor software design choices?

If you change an API, you will have to change consumers of the API. Does that mean that your code is bad?

This exists even in a low level examples: if you change a c++ class, you will need to also change the corresponding header file.

Or perhaps, am I misunderstanding the concept?

lifeisstillgood• 12 years ago

The author noted there were "legitimate" linkages such as files and test files for those files, header files would be another such case. Eliminating those (which should be moderate to trivial) will leave files that really have no business knowing about each other's internals

lnanek22• 12 years ago

Ideally, with good encapsulation and individual files keeping to single responsibilities, you would more often get away with just changing implementation and not changing API. Although you could take the size of the change into account to penalize API changes in other files less than implementation changes in both files and ignore header files if you didn't want to clump consumer changes into the same code smell. If the API has to change, you may have exposed too much implementation to the consumers.

armish8• 12 years ago

agreed; those kind of linkages are not necessarily due to bad design but are, on the contrary, intrinsic to the design of the language you code in. Ideally I should consider these kind of things and exclude such pairs (X.cpp <-> X.h) from the final results, but this is still pretty much work in progress.

Good catch, though. Thanks.

briantakita2• 12 years ago

The OP does not really define "bad" software. I'm also not convinced of the concept of "good" vs "bad" software.

When reviewing code, we tend to judge how the final output looks based on the reviews' aesthetics. There is almost no emphasis on the process of building the software nor is there much emphasis on how long it takes & how reliable the software is.

While aesthetics & clarity are important the notion of "good" or "bad" software depends on the context of the judgement. Is it good/bad for the programmer? Is it good/bad due to the costs of development? Is it good/bad based on it's flexibility toward changing requirements? Is it good/bad based on the flaws in deployed system? Is it good/bad based on the feature velocity?

Why is software productivity so difficult to measure? Software is complex & software is created in complex situations. It is tough to get an "apples to apples" comparison when comparing complex contexts. It's like comparing two people. Is one person better than another? Usually it depends on the context...

hyperpape• 12 years ago

I think what's clearer is that bad coding is one reason why we often have to change too many files at the same time. We see software where implementation details surface in so many places that you have to change five files to change anything.

Now, it's affirming the consequent to say "you change two files, therefore you must have repeated yourself/coupled two things too much", but if you think those are common problems, then it will still be sound to say "you repeatedly changed these files in sync, you should look there to see if you've coupled them too tightly."

tootie• 12 years ago

That's brilliant. Reminds how did DNA sequencing back in the day. Genetic linkage analysis.

lnanek22• 12 years ago

That's pretty cool. There are a lot of tools, both public and private at companies kept as trade secrets to do static analysis of code to pin point potential errors, memory leaks, etc.. This is the first analysis I've seen that actually looks at more than one revision of a project in source control, though. Every other tool basically just analyzes one revision at a time.

jwmerrill• 12 years ago

There's some really interesting thinking here, but interpreting this naively would suggest that the perfect software project has only 1 file.

armish8• 12 years ago

that is partially true: single file apps will never have couplings in this manner. One extension to this project would be to find couplings between particular regions of the file and then even single file apps will start falling down. I think it is not true that if you cannot find any coupling using this tool, your software is well-designed; it would just mean this tool is not smart enough to capture those bad designs.

Couplings between different regions of the files, however, are relatively harder to find and requires some more thinking in terms of implementation.

e28eta• 12 years ago

That's where the Cohesion metric comes into play for OO (and maybe other programming paradigms?). Aka: is each class a single cohesive unit, or do you effectively have two (or more) classes melded together with disjoint data and methods.

Something I think is particularly interesting about this approach is that it is language agnostic. It's probably even independent of "programming". It could also be useful for general documents: if I edit wiki page A and always edit B too, should they be the same page instead?

michaelfeathers• 12 years ago

You can do some quick and dirty analysis to find classes that change together in the same day. Often you discover that there are some faulty abstractions.

http://michaelfeathers.typepad.com/michael_feathers_blog/201...

I think that this sort of repository analysis is going to be standard practice within the next couple of years.

pnathan• 12 years ago

This is similar in aspects to http://google-engtools.blogspot.com/2011/12/bug-prediction-a..., which might be a fun set of ideas to link into future iterations.

a3voices• 12 years ago

Thanks, just thinking about coupling made me consider some new design improvements in my code.

swatow• 12 years ago

perhaps off topic, but evfolds algorithm looks very close to estimating a Markov network. Can anyone comment on how their model differs from a Markov network, and how these differences arise?

For reference, given some variables, a Markov network is a parsimonious way to express arbitrary covariance matrices in terms of individual interactions between groups of variables (in this case, pairs of variables). Their approach looks very similar to estimating the Maximum Likelihood or MAP graph.

ilovecookies• 12 years ago

Use this on the linux source code

paulsutter• 12 years ago

I was thinking about writing up a small application paper for this project, but I am really terrible at reading papers from the Computer Science field, let alone writing them.

Thank goodness for that. This blog post was so much easier to read then a formal paper.

Why are formal papers so tedious to read? Imagine how much time we'd waste as a group if he written this as a paper. Many of us would give up before finding the actual information, and those of us that /did/ identify the actual information would have invested a lot more time than it took to read this excellent blog post.

evincarofautumn2• 12 years ago

I have found that the best papers are hard to read because they’re informationally dense, so you have to slow down to really process every sentence and unpack the author’s thinking in your head—but once you do, you get a lot of knowledge from just a few pages. So it ends up being worthwhile.

Average papers are hard to read because they’re trying to emulate the style of the good papers, but without having enough actual content. The length and register of a blog post are definitely a good fit for the average essay.

Crappy papers are hard to read because they’re crappy, and recasting them into a different format would reveal that they contain no information at all. :)

rwallace2• 12 years ago

True, but it is also true that the best papers would be even better if they were written in less terse and cryptic style so that the same information could be obtained more easily.

buss• 12 years ago

Why are formal papers so tedious to read? Imagine how much time we'd waste as a group if he written this as a paper.

This is a huge problem in the academic field. Papers are written in academese, not because it's a particularly good way of distributing information, but because it's expected. I absolutely agree that less formal language is both good for the discipline, the technical non-academic audience, and for the lay audience.

Steven Pinker recently wrote a book called "The Sense of Style" which talks about academese and writing well about complex subjects (something which he, indeed, has a lot of experience in!). I haven't read it yet, but I've heard it's quite good.

wedesoft• 12 years ago

Note to myself: put all source code into one file ;)

Evolutionary couplings between files reveal poor software design choices

Related Stories