Greatest: C unit testing library in one file

https://github.com/silentbicycle/greatest

by luu • 13 years ago

98 45 13 years ago

thirdsight• 13 years ago

I've never used a unit test framework for C ever.

I usually write a test exerciser for the library/command and a "known good output" file then diff the output and the known good.

will take a look at this for certain.

awda• 13 years ago

More sophisticated C unit testing libraries (libcheck, for example) use nice compiler features beyond ANSI C, like linker sets, in order to avoid error-prone duplicating like you see in these examples.

While that may not work in some environments that don't support these extensions, I'm having trouble imagining ones that support C stdio but do not support linker sets.

silentbicycle14• 13 years ago

I (the author) have been doing a lot of embedded development lately, and in those cases you really can't assume very much beyond core ANSI C features. That's also why greatest doesn't do any dynamic allocation, at all. There's a relatively small struct for the current testing progress, and everything else can easily run out of ROM.

My primary goal with greatest is to assume as little as possible, and avoid imposing any additional constraints on projects just because people want to test them. I also want the implementation to be very transparent -- it's just a 600-ish line header file, carefully documented. It's a major pet peeve of mine when I have to work around features that try to automatically be helpful, but have edge cases that break down in opaque ways. (Build systems tend to be particularly bad in that way.)

Also, I'm beyond shocked that the name wasn't already taken. ;)

One of my upcoming tasks is making stdio optional in greatest, because printf doesn't do you a whole lot of good if you're running it on an Arduino or something, but in those contexts automated testing is especially valuable.

meekrosoft• 13 years ago

Cool, I really like this! I wrote a popular C function faking framework in a single header file for exactly the same reasons: all bets are off in embedded targets. https://github.com/meekrosoft/fff

The test C test framework in the project is 4 lines long, but does require the standard library. https://github.com/meekrosoft/fff/blob/master/test/c_test_fr...

abtinf2• 13 years ago

C unit testing framework in 2 defs:

http://www.jera.com/techinfo/jtns/jtn002.html

silentbicycle14• 13 years ago

I wrote greatest because minunit doesn't provide any of the infrastructure that nearly every test runner grows eventually, in some ad hoc way. Halting after the first error is just not good enough after projects grow beyond a certain size. Also, being able to run just specific suite(s) or test(s) by name patterns from the command line really speeds up one's workflow.

I still wanted something as straightforward to use as minunit, though. Not knocking it. I'd rather use minunit than something that imposed its own architectural opinions on my projects, or consumed resources already in short supply. Also, I make a lot of small C utilities and libraries, and having to define a "project" just to have tests adds a bit too much friction.

(I wrote a bit more about that here: http://spin.atomicobject.com/2013/07/31/greatest-c-testing-e... )

abtinf2• 13 years ago

Agreed. I've skimmed the source and it looks like it would be much easier to integrate into a build system as well.

I find Minunit helpful because its easy to memorize and dump into a new project for quick testing. Its brevity also seems to help introduce c devs to unit testing.

sigvef• 13 years ago

Minunit has grown significantly since the initial 2-def version: https://github.com/siu/minunit .

to3m• 13 years ago

One disadvantage of the macro-based approach - and it looks like greatest suffers from the same problem - which isn't actually that unusual - is the difficulty of setting a breakpoint on the code that fails a test. This comes in handy when a test fails and you'd like to investigate.

And due to C's limitations, you will actually have to do this quite often, on a wide range of failures, because the failure messages don't seem to print any details about the values being compared. So you'll need to hit the debugger to find out anything out.

silentbicycle14• 13 years ago

I'm been thinking about ways to get around this. I'm using macros for __FILE__ and __LINE__, but it does interfere with breakpoints.

One of my open issues is to add an ASSERT_EQ_MEMCMP function that does print out the difference, probably as a hexdump. Similarly, greatest could have something like 'ASSERT_EQ_NUM' for the cases where printing the value is meaningful, and an ASSERT_EQ_STRUCT that takes cmp and print functions. I've been thinking about it for a while, because I want to get it right before I change the API.

mratzloff• 13 years ago

I'm using Seatest[0] for a hobby project I'm writing. I was looking for simple xUnit-style assertions. Works pretty well out of the box, but I did modify it to make it ANSI C compatible and change the output. (I want colored terminal output, darn it! Green if OK, red if not!)

[0] https://code.google.com/p/seatest/

silentbicycle14• 13 years ago

Author of greatest, here.

I wrote a blog post about it a couple months ago: http://spin.atomicobject.com/2013/07/31/greatest-c-testing-e...

exDM693• 13 years ago

Nice unit test framework you have written, exactly what I have been looking for my C projects. I, too, dabble in embedded environments and can't rely on big libraries.

One question/suggestion, though. In the end of all the samples you need GREATEST_MAIN_DEFS/main()/GREATEST_MAIN_BEGIN/RUN_SUITE(xxx)/GREATEST_MAIN_END. Could this be a simple oneliner like GREATEST_MAIN(xxx)?

I do understand that this is probably required for embedded systems where you might not have a regular main(), but I would just like to write GREATEST_MAIN(xxx). Does this make sense to you? Are there use cases where it makes sense to do something else in main()?

Anyway, good stuff!

silentbicycle14• 13 years ago

Thanks!

I use GREATEST_MAIN_BEGIN and GREATEST_MAIN_END so that people can put other things in main, around them. Also, there are often several RUN_SUITE calls, and I can't depend on vararg macros portably without also depending on C99. While there are some ways to work around that, they don't deliver enough value to justify the added complication.

dasmithii• 13 years ago

Macros are surprisingly useful in the unit testing context. Whether or not it's considered good practice, I like to embed my tests within the header files that they're associated with. With macros and gcc's -D flag in order to generate production and testing builds separately.

I'm new to the world of C, is this typical?

silentbicycle14• 13 years ago

I've seen people do that before, as well as putting the tests in the .c file, guarded by "#if TEST_MODULE_NAME", and exporting a (module_name)_run_tests(void) function.

My preference is to have separate test_module_name.c files. Failing that, I would include tests in the module's .c file rather than the header, so that the header can be relatively short and focused on documenting the module's interface, rather than its implementation details.

greatest doesn't force any specific style there, by design.

simias• 13 years ago

It's fairly typical. The standard "assert" macro behaves exactly that way, it does not generate code if the code is compiled with NDEBUG defined.

Of course you shouldn't abuse it but in this instance it's perfectly fine IMHO.

This reminds me that in addition to that the linux kernel has a pretty cool feature for debugging called "dynamic debugging" where you can enable and disable "printk" (the kernel's version of printf) messages dynamically from the shell through the debugfs:

http://lwn.net/Articles/434833/

habitue• 13 years ago

I used to use FCTX for unit testing, which was just a header file. I don't know if it used nonstandard compiler features under the hood. Unfortunately, while it's mirrored, it doesn't look like the main development website is around any more.

Github mirror: https://github.com/imb/fctx

RhysU• 13 years ago

+1 on FCTX, even in its dormant state. A sample: http://agentzlerich.blogspot.com/2011/01/c-header-only-unit-...

SvenDowideit• 13 years ago

will you be adding it to CCAN (the C code archive, in a similar spirit to Perl's CPAN)?

see http://ccodearchive.net/

aslakhellesoy• 13 years ago

Joe Walnes' tinytest is just a 113 line header file: https://github.com/joewalnes/tinytest

Greatest is 591 lines .h and 133 lines .c.

You can see tinytest in action here: https://github.com/cucumber/bool/blob/master/c/test/test.c

silentbicycle14• 13 years ago

They've got a lot in common. tinytest looks a bit simpler, but also doesn't do any allocation and looks reasonably portable. Cool.

The single biggest difference is that greatest has more control over about which test(s) it runs. tinytest just bails at the first failure, whereas greatest can run everything and report, or individual test(s) or groups of tests whose names match a substring. That adds some to the implementation, but I've found it particularly valuable once a program has several modules and hundreds of tests.

Also, greatest supports parametric testing, which means that adding fuzzing/randomized testing is pretty straightforward (though usually project-specific).

tinytest requires C99, greatest doesn't. That shouldn't matter, but unfortunately sometimes it does. Embedded compilers for proprietary architectures can be lacking, in particular.

greatest also doesn't assume ANSI escape sequences are okay. Sometimes dumping out a bunch of [1;31m stuff is annoying. That's a matter of taste, though, and it'd be easy to make it optional.

silentbicycle14• 13 years ago

Also, a minor nitpick: the 591 line header is the whole thing. The 133-line .c file is only an example.

gcb02• 13 years ago

this is a bloated minunit :)

just the short of thing a QA dev would do to keep his job.

i kid. kid. It is nice, but overkill in my humble opinion. Failing on the first error for unittest is a feature. because you are not supposed to even commit it if it is failing, so no need to fancy graphics and reports. It either passed, or not.

diab0lic• 13 years ago

Is it though? If I made changes that broke something surely I would want to know everything that broke rather than just the first test in an arbitrarily ordered set of tests!

Though I like your style, very simple, and simple is good. The state is binary, tests are passing or they are not. Unfortunately in this particular reality the added complexity of knowing all broken tests can prove very useful.

silentbicycle14• 13 years ago

Knowing that a change broke a specific set of tests tells you a lot more than knowing it broke the first one of them that happened to run.

If I'm using fuzzing to run thousands of tests, and all the ones whose seed led to the same specific bit pattern are failing, that's particularly useful information. I found a particularly subtle bug in heatshrink (https://github.com/atomicobject/heatshrink) that way, but a single failing test would have looked like noise.

kabouseng3• 13 years ago

I'd like at least the option to keep on working, making changes, and at the end fix all the broken tests. Sometimes it's easier to develop this way than to stop everytime and fix a broken test, usually when you change the architecture / big code changes.

exDM693• 13 years ago

Failing on the first error for unittest is a feature. because you are not supposed to even commit it if it is failing, so no need to fancy graphics and reports.

A test suite may run for a long time and bailing out at first failure is a very stupid thing to do. It may be nice to have it as an option but not the only way of doing it.

Even a humble test suite might take 5 minutes to run, going to grab some coffee and coming back to see the first test fail would be a bad thing. A big test suite might take hours on a CI server, having it fail on the first test would be a major waste of time.

And with a modern distributed version control, there's no reason why you shouldn't commit code (not to the master branch!) that doesn't pass tests. To share it with your team mates or push the code to be tested on a CI server, for example.

gcb02• 13 years ago

if you unit tests take that long, break it up.

or you are probably doing functional testing in your units.

yxhuvud• 13 years ago

Failing on the first error is not a feature, because the error that happen to be first may not give enough information on where it went wrong, but a different test may.

swah• 13 years ago

Also consider that in embedded software the hardware changes much more. You can't just "not commit something that fails". Something might work on hardware v1 but fail on v2 (that uses the cheaper version of the microcontroller, whatever).

kabouseng3• 13 years ago

CppUTest isn't bad for C projects, even if the test framework uses C++. http://cpputest.github.io/

edit - grammar

kabouseng3• 13 years ago

Sorry I don't want to hijack the authors effort, I am just mentioning it for anybody considering additional testing frameworks.

puzz• 13 years ago

Yet another very simple single-file in 65 lines of code here: https://github.com/tkrajina/yactest

Mo1oKo• 13 years ago

In our team we relay on Unity for our unit tests. We do embedded C and try to practice TDD as much as possible. The author claims it depends on Ruby and such to work. I wanted to point out this is not the case, and that actually we are using it 'clean' with no trouble.

silentbicycle14• 13 years ago

Unity alone doesn't, but ceedling and CMock do. I was referring to them as a group, because I usually see them used together, but you're right.

(I work with most of the maintainers of those tools, and have some minor commits myself.)

norswap• 13 years ago

In the same flavour: https://github.com/orangeduck/ptest

NigelTufnel• 13 years ago

Question to professional C programmers: Do you write unit tests?

At my current job I've never met C++, let alone C programmer who writes unit tests. Is it common?

silentbicycle14• 13 years ago

In my experience, unit testing is more common in embedded C development, because the code is held to much higher standard of accountability.

Sloppy embedded code can kill people, or at the very least lead to incredibly expensive hardware recalls. With the exception of poorly thought-out privacy policies, it's unlikely that errors in a web app can cause that level of problems.

It's also much easier to deploy an update to a web app than a Mars rover or a car's brake controllers.

aidenn0• 13 years ago

I see few unit tests, but lots and lots of integration tests, and it looks like greatest could be used for that too.

Greatest: C unit testing library in one file

Related Stories