Finding Compiler Bugs by Removing Dead Code

http://blog.regehr.org/archives/1161

by mehrdada • 12 years ago

141 19 12 years ago

blog.regehr.org

pedrocr• 12 years ago

It's an interesting method. Does anyone know if these kinds of torture tests get collected into a common library to help any future compilers or if they just result in a paper and bug reports to current ones?

TheLoneWolfling• 12 years ago

Good question.

Another idea: taking bug reports for one compiler for a language and using that as fuzzing input for another compiler for the same language.

pascal_cuoq5• 12 years ago

This gives very poor coverage. The testcases in one compiler's regressions suites are usually reduced to the shortest program that causes a previous bug, and another compiler that does not share any source code with the first one will have no particular reason to contain bugs that are revealed by the same programs. Sure, you may find some this way, but it is not likely to be the best use of your time. Invent something like Csmith/EMI instead, that will be more efficient!

The C language needs a good test suite with known expected results and good coverage of the language's definition, but one way to build this suite is not to catenate all the regression tests of existing compilers. Actually, part of the reasons why we do not have such a reference testsuite is that it is not as easy as putting together regression tests from various origins.

pascal_cuoq5• 12 years ago

The generated tests themselves are hardly worth the bytes they occupy. The value is in the compact program that can generate as many of them as desired very quickly, with varying options to focus on varying aspects of compilation (with or without bit-fields, with or without deep pointer nesting, …)

We forced Xuejun Yang (who turned Randprog, the prototype that came before Csmith into Csmith) to fix more bugs than were necessary for his PhD or than you can expect a research prototype to have bugs fixed (I am one of the developers of Frama-C, pitting Frama-C “against” Csmith was my hobby for a summer, and we found and reported as many bugs in Csmith as we found bugs in Frama-C). The sentence “over the last couple of years I’ve slacked off on reporting compiler bugs” near the post's conclusion is telling. You can expect the same story to unroll for EMI. Researchers are not rewarded for maintaining software ad vitam æternam, even if the software is useful, but hopefully they have or will soon release the generator as open-source, and then if you find it too useful not to use further, you can fix or work around bugs as you find them.

As an example, I think that there is still one bug in Csmith that we work around by ignoring programs that have the symptoms that usually indicate that bug (we still use Csmith to test Frama-C after we have finished a major feature that could introduce the sort of bug it can detect).

See also http://blog.regehr.org/archives/1058 on the same blog for additional thoughts on the fate of academic software.

regehr5• 12 years ago

Pascal, have you reported the bug? Xuejun is doing some Csmith work right now and we can probably get him to fix it.

Jabbles2• 12 years ago

I disagree that the tests are not obviously worth keeping. Having a large suite of regression tests is vital to stop bugs reappearing.

This type of research spawns other research and projects outside of academia by acting as a proof of concept, even if the original researchers stop reporting bugs.

For instance Csmith (and others) inspired Gosmith, which has found a number of bugs in the Go compiler. I hope that someone will use the obviously successful strategy of EMI to improve it further.

https://code.google.com/p/gosmith/ https://code.google.com/p/go/issues/list?q=label:GoSmith

michaelfeathers2• 12 years ago

Has metamorphic testing been exploited for property-based testing?

davorak• 12 years ago

Looks like there is significant overlap between metamorphic testing an property based testing. Can anyone come up with examples that are clearly one but not the other?

http://en.wikipedia.org/wiki/Metamorphic_testing

regehr5• 12 years ago

Metamorphic testing means taking an existing test case and mutating it into a new test case that produces the same answer (or at least an answer that can be easily predicted).

Property-based testing is, as far as I can tell, a meaningless term since all testing is property-based.

petercooper• 12 years ago

I knew this reminded me of something, and it turns out it was one of his older posts which is also worth a read: http://blog.regehr.org/archives/970 (Finding Undefined Behavior Bugs by Finding Dead Code)

pasbesoin• 12 years ago

Just mentioning since I looked for myself.

The PDF it links is 404, but the Wayback Machine caught it in a few different revisions (the date of Regehr's post would correspond more closely with the earlier revision):

http://web.archive.org/web/20130810232543/http://pdos.csail....

http://web.archive.org/web/20131214080638/http://pdos.csail....

Finding Compiler Bugs by Removing Dead Code

Related Stories