Warp, a fast preprocessor for C and C++

https://code.facebook.com/posts/476987592402291/under-the-hood-warp-a-fast-c-and-c-preprocessor/

by jamesgpearce • 12 years ago

195 121 12 years ago

code.facebook.com

jevinskie8• 12 years ago

Are precompiled headers and the performance increase from switching to clang[0] not enough?

[0] http://clang.llvm.org/features.html#performance

lbrandy• 12 years ago

Is X not enough?

Nope, it's not, for all X.

Danieru• 12 years ago

Error: Type mis-match near "all". Usage of symbol X as type 'Situation' in line 2 does not match usage in line 1 as 'Solution'.

pjmlp4• 12 years ago

Surely not.

I know of a project that takes around 3 hours to build with Clang when you do a "make all".

deadfa11• 12 years ago

Sure, but is that using PCHs? And how much of that time is preprocessing? I don't think the parent question is whether clang can compile any project in less than 3 hours. I think the interesting question is if warp is more compelling than clang from a preprocessing perspective.

duaneb4• 12 years ago

I mean, I do too, but they take even longer with gcc. I can remember when compiling could take a full day. These metrics mean nothing without comparison. Clang is hands down the fastest c/++ compiler I've ever used.

jevinskie8• 12 years ago

I got a chance to try this out. This is the file I used: https://gist.github.com/jevinskie/9843152

Top-of-trunk clang from http://llvm.org/apt/ takes ~2.1 seconds on my machine to preprocess the file to /dev/null. warp (compiled with -O4 -frelease -fno-bounds-check) takes ~2.8 seconds.

So this test case is faster with clang even without precompiled headers. It is hard to make a benchmark for clang's precompiled headers because the AST is lazy-loaded from the PCH. You would have to actually have code use values from the header.

EDIT: I forgot another advantage of clang: the preprocessing and compilation and assembly are all done within the same process, eliminating process creation overhead.

andralex4• 12 years ago

Thanks for the data point! (Also don't forget -frelease.)

gillianseed• 12 years ago

The compile speed difference between a modern GCC and Clang/LLVM is nowhere near that dramatic, this comparison you linked is against GCC 4.21 and GCC 4.0 which are 7 and 9 years old respectively.

jevinskie8• 12 years ago

I really only meant to refer to the -fsyntax-only portion which is all that matters when it comes to C preprocessor performance. I know GCCs codegen has steadily improved but I'm not aware of any work to speed up the parser.

comex3• 12 years ago

Precompiled headers are a pain to use. But clang modules are supposed to be precompiled headers done right:

http://clang.llvm.org/docs/Modules.html

When they're usable for C++, there should be little reason to use any special preprocessor, since every header file (not just a static common set) need only be compiled once into a binary format rather than being included into N source files. It can't happen any sooner for me... but as of recently, they're still very broken, so I'm still waiting.

pmr_4• 12 years ago

Clang modules for C++ are going nowhere as long as the concerned working group of WG21 isn't proposing anything.

While it seems that everybody has almost the same idea how modules should look as part of the language almost no one can agree on how they should be specified or what part of a module should actually be specified at all.

aDevilInMe2• 12 years ago

Does Andrei have a sticky D button on his keyboard? He seems to mention it in every other sentence, where as I thought this post was about C and C++.

I would ask if there are any improvements between using wrap or just using clang.

m0nastic• 12 years ago

Warp is written in D, by the guy who created the language. It doesn't seem weird that Andrei would want to talk about that (especially considering the fact that I assume he's very much interested in seeing D in use in more places).

I had the same question about how it compares to clang though. I suppose now that it's open source, someone can do some testing.

aDevilInMe2• 12 years ago

" I assume he's very much interested in seeing D in use in more places"

Yes, I think you are correct and this is what I was getting at. It seems like a propaganda exercise to promote D, you only have to look at the end of the article to see this "And join the D language community for the D Conference 2014 on May 21-23 in Menlo Park, CA."

For the record I am very much aware of who Andrei is and the influences he has on certain languages.

DannyBee11• 12 years ago

I would be shocked if there are improvements over clang (and if there were, clang would fix them) in the general case.

clang has a pretty optimized preprocessor (It even has things like using SSE2 to do some neat things with character processing).

GCC has a fairly modern one, but it's still beatable (and clang beats it handily).

pjmlp4• 12 years ago

Well, as I mentioned on a parallel thread, I know of a project taking around 3 hours to compile, that doesn't look very optimized to me.

WalterBright23• 12 years ago

Feel free to try it and see for yourself! Post results here.

andralex4• 12 years ago

I'd be curious to see how clang does, too. What matters to us is that warp is easy to get into so we can easily adapt it to our build system (in particular multithreaded preprocessing that saves on opening same included files multiple times).

wnissen• 12 years ago

He understands C++ well enough to have written "Modern C++ Design" ( http://amazon.com/dp/0201704315 ) back in 2001. If he's interested in D, so am I.

AsmMAn5• 12 years ago

You can see this guy on http://forum.dlang.org/ almost everyday talking about D's designer with the community.

fpelliccioni3• 12 years ago

This makes no sense. Alex Stepanov continues to programming in C++. What language are you going to use now? (¿?)

andralex4• 12 years ago

Preprocessing with clang would be very tricky in a system that uses gcc for the compilation proper (which we do). Clang adds its own predefined #defines, which may change the resulting code.

DannyBee11• 12 years ago

Sure, if you are stuck with GCC, and have no plans to move, it makes sense to improve GCC's preprocessor for a number of reasons (warp produces cacheable artifacts, etc).

But it's actually not that tricky, since you can just change the defines it makes (after all, if you are maintaining your own toolchain, you are maintaining your own toolchain).

In fact, you are already doing it with warp to emulate GCC's defines.

Suffice to say, we've done it before to provide clang diagnostics but build with GCC.

bames53• 12 years ago

You can undefine predefined macros in clang using -U. I took a little time to write some commands that should produce something close to the right defines/undefines to let one compiler's preprocessor masquerade as another compiler's preprocessor:

http://coliru.stacked-crooked.com/a/17122f90ec070d55

EvenThisAcronym• 12 years ago

Does Andrei have a sticky D button on his keyboard?

Well, it is written in D. Walter (not Andrei) is talking about how he leveraged D's features to his advantage when writing Warp.

andralex4• 12 years ago

I appreciate this piece of feedback. I've been consciously trying to keep the article interesting technically and avoid it being construed as an advertisement for D, to the extent a couple of coworkers and at least one other colleague (http://goo.gl/QZ5ELn) were unclear about warp being written in D at all. I'll do my best to tone things down further in the future. There's plenty of exciting stuff going on to be worth avoiding alienating people.

fpelliccioni3• 12 years ago

WalterBright23• 12 years ago

Walter here (author of Warp). AMA about Warp.

haberman3• 12 years ago

Could you describe in a bit more detail how the ranges-and-algorithms style applies to Warp? What are the major algorithms that you are gluing together? What is the high-level design of Warp?

WalterBright23• 12 years ago

how the ranges-and-algorithms style applies to Warp?

First off, a text preprocessor is a classic filter program, and ranges-and-algorithms is a classic filter program solution. Hence, if that didn't work out well for Warp's design, that would have been a massive failure.

The classic preprocessor design, however, is to split the source text up into preprocessing tokens, process the tokens, and then reconstitute output text from the token stream.

Warp doesn't work like that. It deals with everything as text, and desperately tries to avoid tokenizing anything. The ranges used are all ranges of text in various stages of being preprocessed. A major attempt is done to minimize any state kept around, and to avoid doing memory allocation as much as possible.

Warp doesn't use much of any classic algorithms. They're all custom ones built by carefully examining the Standard description of how it should work.

phaedrus• 12 years ago

Do you employ any detection of boost preprocessor library use, so that e.g. BOOST_PP_FOREACH gets processed as a literal loop rather than a large expansion?

WalterBright23• 12 years ago

Nope. Boost is just another bucket of chum as far as Warp is concerned. I did use Boost extensively to validate correct operation of Warp and profile execution speed.

pmr_4• 12 years ago

Hard problems for C++ compilers are keeping track of where a piece of code originated and providing good diagnostics in code expanded from macros. Can warp help with this or was speed such a paramount goal that the rest of the tool-chain cannot provide good diagnostics as soon as macros are involved?

reubenmorais2• 12 years ago

FWIW, Clang can (and does) help you with that. See "Automatic macro expansion" in http://clang.llvm.org/diagnostics.html

WalterBright23• 12 years ago

Warp doesn't go beyond inserting linemarker records so that GCC can identify source(line) for subsequent error messages.

zwieback• 12 years ago

According to your bio you're a lapsed ME, like myself. Do you think not coming from a CS background gives you an opportunity for novel solutions in compiler development work?

[My first real project used Zortech C++ for OS/2, thanks for the fond memories...]

WalterBright23• 12 years ago

I think not coming from a CS background has resulted in me reinventing the wheel on several occasions. (I thought I'd had a brilliant insight, only to be told it was well known.)

I'm credited with a few innovations (like NRVO), but for the most part I just put together novel combinations of other peoples' ideas :-)

srean2• 12 years ago

Are file I/O asynchronous in Warp ?

On a related topic could you tell us about coroutines/fibres in D ... How are they implemented ? Can they be called from C ? are they are used in D's standard library, examples of a few notable use cases (I guess one would be Vibe.d), examples of asynchronous idioms in D.

Since that's a lot of questions, pointing to documents would be fine too.

..and thanks for answering questions here.

WalterBright23• 12 years ago

No, Warp doesn't do asynchronous I/O.

The D runtime library does have fibers:

http://dlang.org/phobos/core_thread.html#.Fiber

floil• 12 years ago

Will Warp's design allow modes to be added, where it can be be used as a preprocessor replacement for compilers other than gcc (msvc, clang)? Would you accept such patches? I know mcpp (the only other standalone c preprocessor I've seen) tried to do this.

Other than the speed processing, is there anything about how Warp operates that would make it easier to implement a distributed build system? It seems like the preprocessor can sometimes play a role in whether builds are deterministic enough to be successfully cacheable (__FILE__ macros and such).

WalterBright23• 12 years ago

Warp only predefines the macros required by the Standard. For all the other macros predefined by cpp, it is passed those definitions on the command line via a script file.

The same should work for any other compiler, provided they don't use preprocessors with custom behavior.

I don't see any reason why Warp cannot be used in a distributed build system.

0x092• 12 years ago

Since Warp is meant to be a drop-in replacement for GCC's cpp, do you plan to include a traditional (pre-standard) preprocessing mode? This has been a source of some agony in FreeBSD, preventing clang's cpp from fully replacing GCC cpp on some ports (mostly X11 related due to imake). So far the partial solution has been ucpp.

http://gcc.gnu.org/onlinedocs/cpp/Traditional-Mode.html

https://wiki.freebsd.org/PortsAndClang/CppIssues

http://code.google.com/p/ucpp

jevinskie8• 12 years ago

Is there a PR for that on http://llvm.org/bugs/ ?

WalterBright23• 12 years ago

I didn't know anyone still expected to compile pre-1989 code.

jedbrown• 12 years ago

warp is currently able to preprocess many source files, one after the other, in a single command. [...] warp is set up/torn down only once for the entire set of files, rather than once for each source file.

I'd like to learn more about this. I spend a fair amount of time building on HPC systems. Frustratingly, compiling on a $100M computer is typically 50x slower than compiling on a laptop due to the atrocious metadata performance of the shared file system. Configuration is even worse because there is typically little or no parallelism. Moving my source tree to a fast local disk barely helps so long as system and library headers continue to reside on the slow filesystem. A compiler system that transparently caches file accesses across an entire project build would save computational scientists an enormous amount of time building on these systems.

srean2• 12 years ago

This is why I asked Walter about use of asynchronous idioms in Warp. It ought to help even if there is no parallelism involved. It is perhaps a TODO. Walter would be able to elaborate.

mrich• 12 years ago

So your build speed is limited by metadata file accesses to network storage? Your network/storage must be really bad then. How about mirroring all the needed headers locally before building? You could establish that as a makefile rule.

wehadfun• 12 years ago

Could any of this be used to improve comiling speed in other languages?

pmr_4• 12 years ago

Which other languages use the C/C++ preprocessor? I've seen it used for generating data or other source code when a preprocessor comes in handy, but never as a full-fledged component of another language.

I also think that a modern language that uses includes instead of modules is just outright insane.

plorkyeran2• 12 years ago

Objective-C(++). I don't think obj-c changes anything about how the preprocessor works, so presumably it'll work fine for that.

Using the C preprocessor in a non-superset-of-C language would be a pretty odd choice.

aidenn0• 12 years ago

Many assemblers have the option to use the c preprocessor.

nemetroid• 12 years ago

The Glasgow Haskell Compiler has a language pragma for running CPP before compilation. I've seen it used a few times, for #ifdef compatibility between Windows and Linux for low-level stuff. I don't think anyone would ever want to use #include or macros, so the issue of speed is not very interesting in that context.

example: http://hackage.haskell.org/package/system-filepath-0.4.10/do...

WalterBright23• 12 years ago

It'll speed up any language that uses a separate C preprocessor.

The same overall design could be used to implement many textual macro preprocessors, such as runoff, makefiles, etc.

duaneb4• 12 years ago

Could you substantiate this claim? I don't necessarily not believe you, but it's an easy claim to make without any quantifiable argument. It's also hard to test against eg clang, where preprocessing is performed in the same process as the compiler and assembler, and the performance of a preprocessor is really less interesting than total compile time.

zenbowman2• 12 years ago

Must say facebook has quite the lineup of all-star programmers:

- Carmack

- Abrash

- Alexandrescu

- Kent Beck

Cannot say I'm not a little jelly of those who get to spend time with these fine gents.

http://www.southparkstudios.com/clips/babjj8/jelly-school

aristidb• 12 years ago

I find it a bit weird that the top comment in this story fawns over several star programmers, but ignores the guy who actually wrote the tool that the story is about.

boxy_brown• 12 years ago

The article suggests he is not a full-time employee of Facebook.

astrodust3• 12 years ago

Those are the greats today, and with a talent pool like that it's likely that many future greats will be created there. You have to do more than write good code to be legendary.

Maybe Facebook might end up being the Xerox PARC of our modern times.

dllthomas2• 12 years ago

So, to be clear, this is a reimplementation of cpp?

WalterBright23• 12 years ago

I have never looked at the source code to cpp.

But Warp is designed to be a drop-in replacement. It doesn't produce char-by-char exactly the same output as cpp, as the whitespace differs, and the decisions about when/where to produce linemarker records are different.

But the output is functionally identical (any differences are bugs in either Warp or cpp).

dllthomas2• 12 years ago

That fits well within my use of the term "reimplementation." I appreciate the specificity of your response, though - always best to answer every possible question when you're not sure which is meant.

haberman3• 12 years ago

Interesting, I did not realize the standard left room for preprocessors to make implementation-defined decisions!

mfonda• 12 years ago

Great line from the interview:

WB: I can guarantee that you are wrong about where your code is spending most of its time if you haven't run a profiler on it.

Definitely a good thing for all programmers to think about!

yetanotherphd• 12 years ago

One thing that really sped up my compiler was eating a ketogenic diet, and eliminating preprocessed foods.

Warp, a fast preprocessor for C and C++

Related Stories