Please stop breaking the build

by luu • 11 years ago

72 42 11 years ago

    Web programmers are hyper-aware of how
    10ms of extra latency on a web page load
    has a noticeable effect on conversion rate

I wish this was true, but I'm really skeptical that the average "web programmer" even uses the phrase "conversion rate" on a monthly basis.

nullspace• 11 years ago

I know neither yours nor my comment strictly related to the topic - which is about not breaking builds, but the effect of latency on a web page only has a tiny contribution to conversion rates or other user satisfaction metrics as compared other factors - like stability and features. Hence, I think it's not _important_ for most web programmers to be hyper-aware of the latency - as long as it's reasonable.

hurin• 11 years ago

I'd very much like to have some more information about how the article author gathered this data - are these all master branches of said projects, numbered releases?

How does the data account for variation in development procedures (e.g. some projects use master as their bleeding edge branch)?

scott2449• 11 years ago

The data is not really relevant for the reasons you state + failed build does not equal published artifacts. How long were the builds in that state? etc.

mirashii• 11 years ago

Also consider that if its checked on a commit by commit basis that doesn't collapse merges in somehow, then you actually check every commit that was made during incremental work on a different branch. If you're doing TDD or any test first methodology, that's going to often mean you intentionally commit broken tests to fix them.

Also if its a commit by commit basis, the percentage of time comparison is completely invalid.

chrismcb• 11 years ago

The author is talking about broken builds, not broken tests.

lambda• 11 years ago

  How does the data account for variation in development 
  procedures (e.g. some projects use master as their 
  bleeding edge branch)?

That shouldn't matter. You should always be able to successfully check out, build, and pass all tests on master. Otherwise, how are people expected to get development done? If you aren't able to build and run tests because you need to be fixing something that someone else committed broken, it will be a lot harder to ever make real progress.

Even if it's "bleeding edge", master should always build and always pass at least the minimal "fast" test suite; some projects may have more thorough "full" test suites that take many hours to run, and requiring that those be run on every change would slow down development too much, but at the very least, the quick "sanity check" test suite should always pass before something can land on master.

hardwaresofton• 11 years ago

Am I missing something, or isn't this problem easily fixed by enforcing testing before push? As long as the appropriate tests are available, seems like this could be solved

jdlshore• 11 years ago

It's not necessarily that simple, but it is simple. You need to do two things:

1) Integrate the master branch (or whatever your guaranteed-good branch is) with the code you're about to push. This prevents integration conflicts from causing the build to fail.

2) Test it on a reference machine. This prevents environment assumptions from causing the build to fail. (Such as installing new software or setting an environment variable, but forgetting to make it part of the build.)

These are both easy to do. My preference is to push to a testing branch on the integration machine, merge in the master branch, run the tests, then merge the testing branch back into the master branch. (There's a bit more to it than that, to cover edge cases, but that's the gist.)

Sadly, most teams and CI tools aren't set up to do this--although, as the article says, it's not rocket science. In fact, I'm surprised it's not obvious that you should do it this way.

justincormack• 11 years ago

Doesn't help if eg you have multiple reference platforms - many open source projects support a range of platforms (eg OSX, Linux, Windows) - much harder than in house software that has a standard platform. You really have to stage commits then.

icefox4• 11 years ago

While both of those are nice, even doing something as dumb as having a git pre-commit hook that runs 'make' (or whatever is the equivalent) catches an amazing amount of errors.

jmckib2• 11 years ago

This was mentioned as a solution in the article.

http://graydon2.dreamwidth.org/1597.html

tensorproduct• 11 years ago

I agree. I thought that this was the whole point of having a CI system.

I don't have a huge amount of experience in open source projects, so maybe they do it differently. Anywhere I've ever worked, not breaking the build and not causing regressions were a prerequisite to getting any pull request serviced. That's how you keep the master from breaking.

On the other hand, building a testing in a clean environment costs money. A company will pay for server time if they believe it's cheaper than developer time (it is). Maybe OS projects just don't have those resources.

Animats• 11 years ago

Welcome to distributed programming. Just because it works for one developer doesn't mean it works for the others, at least not until all the conflicts are resolved. One could have build systems which didn't permit a change to commit in the master branch until everything compiled and all tests passed, but Github isn't set up that way.

The build may also break because of a change in an external dependency.

ForHackernews2• 11 years ago

The build may also break because of a change in an external dependency.

It shouldn't unless you've been lax in how you specify your dependencies, or the library maintainers have been sloppy in what they claim for backwards-compatibility.

Obviously in the real world, both of those things happen.

zeendo• 11 years ago

Package repositories go down too (Maven, RubyGems, NPM, etc...).

Unless by 'lax' you mean 'didn't check in your dependencies'...I don't think I'd call that lax, though.

golergka• 11 years ago

One could have build systems which didn't permit a change to commit in the master branch until everything compiled and all tests passed, but Github isn't set up that way.

If you use pull requests instead of commits, you can set up such a system to automatically run make test.

jmckib2• 11 years ago

This isn't mentioned until later in the article, but it seemed important.

The worst thing about regular build failures is that they’re easy to prevent. Graydon Hoare literally calls keeping a clean build the “not rocket science rule”, and wrote an open source tool (bors) anyone can use to do not-rocket-science.

http://graydon2.dreamwidth.org/1597.html

metajack• 11 years ago

Which has been doing amazing things for Servo and Rust for years now!

dbaupp• 11 years ago

A member of the Rust community, barosl, wrote Homu (bors 2.0) which Rust has been using in place of the first bors for a few months now. It's even better than the already-great first version! I'm seriously considering setting it up for my own projects, since I believe it has support for using Travis CI as a testing backend, in addition to buildbot (which Rust and Servo both use).

https://github.com/barosl/homu

Example interaction: https://github.com/rust-lang/rust/pull/23381#issuecomment-80...

Docs: http://buildbot.rust-lang.org/homu/

CyberShadow• 11 years ago

Interesting. For the D programming language we use something similar but a little different: once a reviewer approves a pull request, they toggle an "auto-merge" flag in our CI tool, and a bot merges the pull if all tests pass (after merging with master, of course).

scotty794• 11 years ago

Why builds break on our team? Because we have no idea how to add precommit hook to perforce intellij client so it tries to build and test the whole thing locally before pushing changes to repo.

detaro2• 11 years ago

and your team isn't disciplined enough to do it manually ("nothing is stopping us" is an explanation why things happen sometimes, but it is a bad excuse)

johan_larson2• 11 years ago

As I heard it, one of the late-nineties projects at Microsoft had a big problem with build breakages. They finally instituted a policy of tracking down every build breakage, and conducting a little humiliation ritual of awarding the offender a "big sucker" award at the weekly project meeting. This award had to be displayed on the developer's office door for a week or something.

You could dodge the award even if you did break the build by showing that you had run the full unit test suite before you submitted your code.

scotty794• 11 years ago

I wouldn't say undisciplined, just human. If you can not do something you from time to time won't do it.

bjwbell• 11 years ago

Personally I don't think any team is disciplined enough to ALWAYS do it manually and the con-founder of "it works on my machine".

icefox4• 11 years ago

Why not have local pre-commit hooks? Even if you are not using git-p4 clobbering some sort of frontend tool to p4 submit that first runs make etc that everyone would use would be a big win for your team.

http://benjamin-meyer.blogspot.com/2010/06/managing-project-...

scotty794• 11 years ago

Everybody uses intellij. We couldn't find ability to add local precommit hooks there and we didn't have resources to build our own custom tool for such basic operation.

icefox4• 11 years ago

The only way I have been able to get away from Ben's law is to have an automatic system that does builds before code can get in. This can be server side, user side, pre-commit hooks, pull-request bots or anything that catches build breakage before they can get into the main branch. But without something automatic in place the project is destined to fall to Ben's Law.

Ben's Law - When every developer is committing to the same branch the odds that a commit will break the build increases as more developers contribute to the project.

http://benjamin-meyer.blogspot.com/2014/01/bena-law.html

wyldfire• 11 years ago

It's not completely clear to me how the build start times were used to calculate uptime. Especially since a popular usecase for Travis/github integration is to trigger a build-upon-PR. If that triggered build fails, the PR is marked as such and normally the content wouldn't get merged. This is designed to prevent the exact problem described, but are these Travis PR-triggered builds filtered out of the uptime input dataset?

michaelochurch• 11 years ago

I'd love to see the numbers on Haskell. I believe (and would hope) that it would fare well.

heisenbit• 11 years ago

Is 99.9% or 3h/month really the "professional" norm?

I remember a larger project where we struggled for a longer time to anywhere beyond 20%. Eventually we got a lot better but nowhere near 99%.

I also wonder whether it is worth it? Are all developers really blocked when the build fails? Some yes but if I could I would design the production chain that most would not be.

Isn't there a trade-off in how much to invest into build tooling and automation vs. in functionality? Does building a MVP include building a perfect software production pipeline?

mbell• 11 years ago

I also wonder whether it is worth it? Are all developers really blocked when the build fails?

In our setup building the deployment artifacts is contingent on tests passing so if the build isn't passing, no one can deploy.

pmahoney• 11 years ago

But you could build and test the candidate master branch before it actually becomes the master branch...

MBlume• 11 years ago

Wait, what? How? Can't you just build things before you merge them into master?

johan_larson2• 11 years ago

Three nines is way beyond what I have seen. At most places I have worked, the build was broken for an hour or two each day.

There was one place where the official builds had been broken for so long that people stopped paying attention to them. The new dev director put a stop to that, to his credit.

Please stop breaking the build

Related Stories