Benchmarks of various C++ build tools

http://retropaganda.info/~bohan/devel/wonderbuild/benchmarks/time.xml

by ingve • 11 years ago

44 13 11 years ago

to3m2• 11 years ago

Which cmake generator was used? The links seem to be dead, so it isn't clear.

I mention this because I'm using cmake for a current project, and backend choice seems to be somewhat important. I found a noticeable improvement when I switched from using a make-based backend (NMake on Windows, GNU Make on OS X and Linux) to the Ninja one (see https://github.com/ninja-build/ninja).

Based on my notes from the time, full build times were basically unaffected on a per-core basisi - but no-op builds were markedly quicker. My notes don't say anything about builds where only one or two files were changed, but my recollection is that that case was noticeably improved as well.

For whatever it's worth, figures from my notepad from a few months ago. Apologies for the scrappiness; these were just brief notes I took at the time, all rather random, just to determine whether the improvement was ~0, or >0... no more than that :/

All times in seconds. All tests performed on my laptop - 3.1GHz dual-core i7.

Windows in a 3-core VM:

    nmake full = 143.72 (1 cpu)
    nmake no-op = 2.22
    ninja full = 140.41 (1 cpu)
                 79.06 (3 cpus)
    ninja no-op = 0.135

OS X native/Linux 4-core VM (unsure which - probably Linux VM):

    GNU make full = 46.81 (4 cpus)
    GNU make no-op = 0.93
    ninja full = 49.36 (4 cpus, I suspect - by default Ninja uses all available cpus)
    ninja no-op = 0.03

(Other notes: cmake works on Windows, and has a reasonably-sized ecosystem, but... OMG. The scripting language is fucking insane. Still worth a look though.)

GFK_of_xmaspast• 11 years ago

My notes don't say anything about builds where only one or two files were changed, but my recollection is that that case was noticeably improved as well

I don't have timings, but this has been my experience as well.

piokuc• 11 years ago

After maybe three years of experience with cmake I must admit its scripting language is just disgusting. Makefile provided a nice, well thought out, simple DSL, with just three concepts: targets, dependencies and actions, and while it didn't provide enough support for handling multiple compiler toolchains, etc. I think replacing it with the cmake shell like 'scripting language', sporting one data structure (string; lists are really just semicolon separated strings...), is an example of very, very bad engineering.

to3m2• 11 years ago

Yes. If they had used something like Python, it would probably be heaps better. (Though to be honest I'm not really sure I trust the cmake developers' collective judgment. I'm sure they could manage to make it awful.)

But, even though it's horrid, I do find it CMake easier to work with than make on a day to day basis. CMake's scripting language is at least straightforwardly imperative; CMake has at least some support for working around certain platform/toolchain differences (though this support is rather half-hearted in some key places); it has a couple of handy features that make working with C/C++ libraries quite easy; it has builtin support for test suites. All more convenient than doing it all by hand in GNU make, and you've got support for multiple backends too. Oh, and it supports Windows, too.

Though I admit it only through gritted teeth, it provides value.

bjohan4• 11 years ago

The cmake benchmarks used the make backend on linux and cygwin, and the vcbuild, msbuild and mscv-ide backend on windows/msvc. The ninja backend is indeed faster than the make one. An update to these benchmarks is needed to take it into account.

PS: I fixed the broken links in that benchmark page.

mianos• 11 years ago

Nice to see that WAF, being python is competitive. It goes to show that smart can beat brawn.

berkut• 11 years ago

It's only really SCons which gives Python as a build system a bad name in terms of speed in my experience.

waterhouse• 11 years ago

And judging by the "number of system calls" section, SCons's performance problems don't appear to be due to the Python runtime.

kodr• 11 years ago

Agreed, but it seems those benchmarks were made to promote wonderbuild. I'm not sure if it's still updated (it doesn't seem so), and it uses old versions for cmake and waf.

it would also be nice to compare other build system like ninja+meson.

lorenzhs• 11 years ago

This is pretty old and the title should reflect that. The versions of the software used are ancient by many people's standards: Linux 2.6.24 (released six years ago, in January 2008), ext2 file system (ext3 was introduced in the 2.4 series, seven years before that kernel release), gcc 4.2.4 (also 2008, 4.2 series is from 2007), cmake 2.4.7 (2007). Overall it seems like these undated measurements were performed in early 2008, nearly six years ago. A lot has changed since then and I don't see how this article in its current form should still be relevant today.

bjohan4• 11 years ago

I take note that I should put a date on the page! I actually ran again some of these benchmarks again over the years (last time about one years ago), on the same hardware and updated software, and results didn't change significantly. FWIW, at the time these results were first published, I had my working dir on ext2 because it had higher performance than ext3.

bjohan4• 11 years ago

Hi, I'm the author of these benchmarks, and yes, they are getting old. One of Scons' maintainers kindly asked me a few days ago to try again with current Scons version, so I've started to update and run these benchmarks again with current software and hardware (SSD). It takes some time to do it right (blame API breakage!) and collect all the results, so I won't be able to publish them before at least a month or so. I will include the meson+ninja and cmake+ninja combinations, and the new CBS tool (an experimental build system from Qt).

Benchmarks of various C++ build tools

Related Stories