How Rust is tested

https://brson.github.io/2017/07/10/how-rust-is-tested

by brson • 9 years ago

299 76 9 years ago

brson.github.io

JoshTriplett6• 9 years ago

Today the longest-running configuration takes over 2 hours.

I'm curious which configuration that is, and how it could be accelerated.

sanxiyn2• 9 years ago

FYI, it's i686-apple-darwin at the moment.

steveklabnik7• 9 years ago

Looking at the build that's building right now, all of them have passed but one, which has been going on for 2 hours and 23 minutes: https://travis-ci.org/rust-lang/rust/builds/252052639

"env": "RUST_CHECK_TARGET=dist RUST_CONFIGURE_ARGS=\"--target=aarch64-apple-ios,armv7-apple-ios,armv7s-apple-ios,i386-apple-ios,x86_64-apple-ios --enable-extended --enable-sanitizers --enable-profiler\" SRC=. DEPLOY=1 RUSTC_RETRY_LINKER_ON_SEGFAULT=1 SCCACHE_ERROR_LOG=/tmp/sccache.log MACOSX_DEPLOYMENT_TARGET=10.7\n",

iOS, it seems.

Looking at the other builds,

* s390x Linux

* i686-apple-darwin

* x86_64-apple-darwin

Seems apple is an overall slowpoke.

sanxiyn2• 9 years ago

It seems to vary from build to build... Compare https://travis-ci.org/rust-lang/rust/builds/251809226 where everything is under 2 hours except i686-apple-darwin.

ihsw23• 9 years ago

This has more to do with Travis-CI than Darwin et al.

https://www.traviscistatus.com/

Notice how the active OS X builds reaches the limit of 216 jobs and the backlog of OS X jobs increases without decreasing. This is a capacity issue that is likely related to high OS X build costs.

Another thing to notice is that the backlog correlates strongly with time of day in the EST timezone, where the backlog starts to take off after 9am EST.

ihsw23• 9 years ago

As mentioned elsewhere in this thread, it stands to reason that Travis-CI is the culprit here.

https://www.traviscistatus.com/

Their slowest build jobs are all on Darwin/OSX-based systems and (notably) this is a problem numerous other projects have -- slow jobs on OSX build systems. There is a large backlog of OSX build jobs that starts around 9am EST.

JoshTriplett6• 9 years ago

Is that true for the paid version of Travis as well? If not, might be worth paying for it, to support a community as large as Rust's. (If the paid version is that backlogged as well, I'm surprised they don't put more resources into accelerating it; perhaps not enough of their customers care about macOS?)

Alternatively, given that Mozilla has their own extensive CI infrastructure (for Firefox), perhaps it'd make sense to migrate to that instead of Travis?

JoshTriplett6• 9 years ago

At some point though LLVM began doing a valid optimization that valgrind was unable to recognize as valid, and that made its application useless for us, or at least too difficult to maintain.

What optimization was that? Some searching didn't turn up any relevant results (other than this article).

steveklabnik7• 9 years ago

I couldn't quite remember either, and brson told me that https://github.com/rust-lang/rust/issues/11710 was probably it. Can't seem to find where it was actually finally disabled though.

kibwen• 9 years ago

Money quotes:

"This is a known valgrind false positive and Rust already has a lot of suppressions for it. It's reported upstream on the LLVM bug tracker but is not really an LLVM issue, just a limitation of valgrind. LLVM is allowed to generate undefined reads if it can not change the program behavior."

[...]

"Just for the record, the undefined read doesn't change the behaviour because it's just a check whether free should not be called, i.e. either it jumps over the free call right away, or it performs the second check."

pornel• 9 years ago

I appreciate the thoroughness of this. I've been using Rust for 2 years now, and the compiler has been rock solid.

losvedir• 9 years ago

I love the concept of the "crater run" (or I guess now "cargo bomb"?), where the entire crate (i.e. "package") ecosystem is compiled, as a test for speculative compiler changes.

It's one of the huge benefits of being both a statically-typed compiled language and offering a modern package system. A lot of languages have the types & compiling but no standard package ecosystem (Haskell, Go, Java), or the package ecosystem but not the compiling (Ruby+Gems, JS+npm, etc). Rust is the only language I can think of that has both, although I'm sure there are others. I'm not sure if there's tooling for compiler writers in other languages to compile them all, though.

And this feature will only get more and more powerful as the ecosystem grows. I think it's a seriously important tool to keep the rust language feeling stable.

steveklabnik7• 9 years ago

"crater run" (or I guess now "cargo bomb"?)

Crater is the older tool that runs builds. CargoBomb is the newer tool that runs builds and tests.

mrec• 9 years ago

Yeah, I was really impressed by that as well when it was first announced. Makes perfect sense once you hear it, but as an old fogey I just wasn't used to thinking on that kind of scale.

I had a similar lightbulb moment years ago when someone pointed out that y'know, these days it's perfectly feasible to test a simple float32 function by running every possible float32 bitpattern through it.

amichal• 9 years ago

True... as long as you know what output you are expecting for all 2^32 patterns

(Not your point I know)

Dylan16807• 9 years ago

These days? Grab a 20 year old Pentium II and you can test a 300-cycle 32-bit function in an hour.

houli• 9 years ago

This is what Stackage(https://www.stackage.org/) LTS releases are in the Haskell world. A compiler and a large subset of the Hackage repository known to compile together and pass tests.

3JPLW• 9 years ago

You don't need a statically-typed or compiled language to have this sort of infrastructure! Julia does something similar for important changes — and it actually runs all the unit tests of all the registered packages.

gsnedders• 9 years ago

Heck, an extreme is what I did for Carakan, Opera's last in-house JS VM, where we were running the testsuite of pretty much any notable JS library in 2010 in CI; that caught loads of subtle bugs. (That said, obviously harder to automate, given there's no standard for in-browser testing!)

Far as I'm aware, every browser is running some version of the jQuery testsuite nowadays, though I doubt that's the only third-party library tested in such a way in each browser.

akavel• 9 years ago

AFAIK, Go devs actually do a similar thing (i.e. test the compiler on a body of selected notable third party packages/apps) [citation needed]. An official central index similar to Rust's https://crates.io for Go is https://godoc.org (though there are other non-official ones too)

brson3• 9 years ago

Yes, my impression is that many production-grade compiler dev teams have technique like this, though often they do not talk much about it, often they are proprietary and/or covered by NDAs, and often they are, as you say, 'selected' code bases. I'm very curious for other compiler teams to say more about what they do here.

ubernostrum• 9 years ago

In the Python world where there are multiple viable implementations of the language, they often take large/notable packages and make those packages' test suites part of the language implementation's test suite.

elihu• 9 years ago

Haskell does have a community package repository (hackage) and a command for installing libraries and their dependencies (cabal).

https://hackage.haskell.org/

eru3• 9 years ago

Actually, I'd say that these days stackage is the place to go for Haskell.

twic• 9 years ago

What stops Java doing this is not that it doesn't have a "modern" package system, it's that the package system ships compiled bytecode rather than source code.

Conversely, what stops other languages with source-shipping package managers doing this is that packages don't include tests. One notable exception to this is Perl, where packages can, and typically do, include tests, and these are run as part of the normal installation process. There are even some public-spirited souls who download and test everything that gets uploaded:

http://www.cpantesters.org/

This incorporation of tests is both a consequence of and a justification for the fact that Perl inter-package dependencies aren't versioned, so you can never really be sure that you're going to download a combination that works.

saghm3• 9 years ago

An interesting thing I learned recently about cpantesters is that they'll also test packages against the bleeding edge versions of the interpreter, so sometimes a package maintainer will get an issue filed (or a PR) saying "your package is going to break in about eight months", which I find both funny and cool.

simcop2387• 9 years ago

In fact I used this for the past Perl release 5.26 to help test everything since there was a known to be breaking change that was going to occur. I need to redo.my efforts now that the release has happened but it did help identify a couple of key packages that needed to be updated

moomin• 9 years ago

You say it's the absence of tests, but Ruby breaks RubySpec all the time, so I think the real issue is that the maintainers plain don't care.

bennofs• 9 years ago

This technique is used to at least some extent in the Haskell world as well. Library changes are sometimes judged by the amount of breakage that they cause (often, Stackage or a subset thereof is used for these tests as a set of "commonly" used packages).

Also, Haskell definitely has a standard package system: Hackage. (Stackage is just a "view"/subset of Hackage)

samth• 9 years ago

This doesn't need a statically typed language at all. Here's the same thing, with test runs, nightly, for Racket: http://pkg-build.racket-lang.org/ (that's building all packages on the current release, there's another one against the nightly snapshot).

ajdlinux• 9 years ago

At my employer, we did some internal work on "compiling" - i.e. installing (including compiling any native extensions) and running test suites - on the entire Python, Ruby and NPM ecosystems.

Conference talk: https://www.youtube.com/watch?v=v7wSqOQeGhA

chii• 9 years ago

no standard package ecosystem ... Java

that's not true! The standard "package" ecosystem in java is maven, and the maven repository!

Scaevolus4• 9 years ago

There’s a big downside though in that landing patches to Rust is serialized on running the test suite on every patch, and it takes a particularly long time to Run the Rust test suite in all the configurations we care about.

One solution is speculative batched testing. Kubernetes uses this to keep up with its very high rate of PRs (>40 merges/day) and ~1hr maximum test times.

Given a queue of patches A, B, C, D, start the normal testing against A, but also start testing the merge of A+B+C+D against the master. If the batch passes first, merge them all. If A is merged first, it's still "compatible" with A+B+C+D, so they can go ahead. Doing a batch in parallel with a single PR like this slightly increases testing load, but ideally doesn't cause any slowdowns versus the fully serialized case.

It looks like bors already supports something like this with "rollups", where simple fixes can be marked for testing together: https://internals.rust-lang.org/t/batched-merge-rollup-featu...

brson3• 9 years ago

Oh, neat. I've never heard it proposed quite that way before, where both branches being tested are compatible. That's real smart.

So far we haven't been able to stomach doing parallel integration builds because of the doubling of expenses.

As you see, our current solution is rollups, where a human batches low-risk PRs into one big PR. It works just ok, and is super high-maintenance.

Scaevolus4• 9 years ago

You already trigger Travis tests for PRs before bors does an auto run, right?

In that case, it wouldn't double expenses. Assuming batches are slightly effective, it should reduce expenses-- with >50% success rate with batches of >=3 PRs, you don't have to do individual testing for the later PRs in the batch.

steveklabnik7• 9 years ago

Similar, but different, yeah: it gives you an easy button to make a new PR out of A + B + C + D together. We tend to batch up documentation changes this way.

eru3• 9 years ago

That's close to what Google does internally. They have quite a few more than 40 commits a day to one giant repository.

Scaevolus4• 9 years ago

They aren't serialized on each other, though. That's the main problem here.

Manishearth• 9 years ago

I'd at some point drawn up a proposal for this, back when Rust started having problems with this. At the time I think we didn't really have the budget for such testing either.

bors' rollup support only lets one mark things for rollup, which show up differently in the queue. It's effectively a priority=-1. This is for things which probably won't break the build, and contributors make a batch rollup every now and then (bors has a button for this). Generally when I do rollups I go through these and ensure they're green on travis.

I've considered giving homu (the engine behind bors) better support for automatically doing this (at least automatically showing travis statuses) but I don't have much time.

loonyphoenix• 9 years ago

Doesn't this process break when, e.g., master + A passes tests, master + A + B fails, but master + A + B + C + D passes again (having been fixed by C + D accidentally)?

I mean, it's pretty unlikely to happen, but it's possible. Plus it doesn't matter if all you cared about was that the HEAD of master passes tests, but not the whole history.

However, the current approach, if rebasing is used to squash every merge request into a single commit on top of master, allows you to have a linear history of the master branch where every commit is known to pass the whole test suite that was in existence at that moment, which, I think, helps a lot when bisecting.

I'm not sure if Rust compresses merge requests into a single commit on top of master, though.

Scaevolus4• 9 years ago

Yes, it's theoretically possible. When it happens in Kubernetes, it's far more common that m+A+B failed because of a flaky test than because B was actually broken, and C+D fixed it.

What we often see is m+A fails because of a flake, and m+A+B+C+D+E passes and merges as a batch. We have a lot of flaky tests. :(

arielb1• 9 years ago

In Rust the "verified product" is the chain of bors merges - we even save build artifacts for each node in the chain for easy bisection of issues. A rollup creates a single bors merge for all the commits in them, so it's only 1 link in the chain.

hsivonen• 9 years ago

It would be really nice if Travis had Linux-on-ARMv7 and Linux-on-aarch64 options on real ARM hardware. I wonder what the economics of Travis building such a thing on top of an existing ARM cloud offering like Scaleway would be.

jorgec• 9 years ago

Unit test in a nutshell, for a business system:

Practically every business system consists of a view, the logic and the database, plus some webservice and whatnot, but mainly its those 3 parts (the so called 3 layers).

Can you unit test, test the view layer? Not really. Can you unit test, test the database by doing real test? Again, not really and its not always possible. So, Unit test is mainly focused in the layer in between of the view and database and both, by their nature, can't be automatically tested.

I like the idea of Unit Test but, for business system, its moot.

nercury• 9 years ago

I don't get it, because you have just explained how to unit test everything else. Which is way better than nothing. Not sure how it is moot, also not sure how it is related to the article :)

wfunction4• 9 years ago

The thing we do differently from most is that we run the full test suite against every patch, as if it were merged to master, before committing it into the master branch, whereas most CI setups test after committing, or if they do run tests against every PR, they do so before merging, leaving open the possibility of regressions introduced during the merge.

Hm, Travis CI on GitHub runs tests on a pull request before the merge. Is what they're doing really that unusual?

saghm3• 9 years ago

As it says in the quote you provided, the difference is that it tests what the result would be of the merge, which differs in that a merge might introduce regressions; if you've never had a bug introduced by a merge not doing what you expected, consider yourself one of the lucky ones.

wfunction4• 9 years ago

I'm not sure I'm understanding things correctly. What I meant was that the test chronologically occurs before the merge takes place. But the test includes the merge itself; it's testing what the result would be after the merge. Have you used the system I'm talking about/do you know what I mean? Am I still misunderstanding something?

Kaivo• 9 years ago

But that only works well if the PR is up to date with the targeted branch.

AFAIK, you have to manually merge the targeted branch into the PRs to be in an equivalent situation. Also, other commits might still be added into the targeted branch between the time the tests have been run and the PR is merged.

wfunction4• 9 years ago

Oh I see, so the problem is that the merge commit is not the same as the one that the test runs for if there are any intervening commits between them. Interesting, I didn't quite realize that! (Though it seems obvious in hindsight, since a new push doesn't make all past PRs' tests re-run.) Thanks for explaining that! So it seems it's something between what they do and what they were trying to avoid. GitHub should really put a button to re-run tests on merges that are out of date...

y2kenny• 9 years ago

Not unusual at all. Any organization using Gerrit with a rebase-only workflow with any popular CI systems (like Jenkins) have been doing this for ages.

sudeepj• 9 years ago

I have been taking inspiration from the Rust project and the way they go about things in general. In my workplace, we have a bot similar to bors, which does the merge request testing and is the gatekeeper.

Tools like bors act as a "force-multiplier".

crncosta• 9 years ago

I dream with the day we will have a GCC Rust compiler.

pas• 9 years ago

What would the benefits of that be?

TotallyGod• 9 years ago

Heads up that your Google fonts (in site.css) trigger a warning about insecure scripts, and in Chrome at least they therefore don't load by default.

Outrageous• 9 years ago

Anyone care to enlighten me why fuzz testing and not mutation testing?

How Rust is tested

Related Stories