Using GNU Make's 'define' and '$(eval)' to automate rule creation

http://blog.jgc.org/2012/01/using-gnu-makes-define-and-eval-to.html

by jgrahamc • 14 years ago

17 10 14 years ago

blog.jgc.org

reedhedges2• 14 years ago

Oh god, this way lies madness. :) It really does. Once you get beyond a fairly low level of complexity, there will be maintainence problems: It will take you too long to get it right, it will be impossible to figure out when something goes wrong, and you'll drive the next poor developer crazy if he doesn't have a complete knowlege of make (which is most of us). Yes, sometimes it may be neccesary to do this if you want to reuse some rules and functions but modify them under different conditions, AND you are restricted to doing it all in the Makefile (rather than preprocessing, calling out to external scripts, etc.)

ArbitraryLimits• 14 years ago

I couldn't agree more. Once upon a time I spent about a week refactoring a really ugly build system (produced a combination of .NET assemblies, C++ libraries, Java applets, and a hacked version of Apache with bash embedded - don't ask) and got everything just right with this approach, then within a week it was all broken again. I removed the rule-generation from the Makefiles and everything was fine after that. Since then my policy has always been that copy-pasta is OK in the build system, and not to worry about it.

(I'm not saying generating the rules from code in a real build system wouldn't be a good idea, jut that it isn't with make.)

mturmon2• 14 years ago

Using "define" to set up templates for rules is pretty straightforward and very powerful. It allows the set of rules to be augmented at runtime, which can enable new uses of make.

I used this mechanism to automate a pipeline for scientific data. The template (the "define...endef" block) held several rules needed to re-make results for one granule of data. When you ran make, it looked for all source granules, and a foreach() mapping the template across the source granules, just as in the OP, set up rules for each one.

Then you could dump a new source granule in a directory, run make -j 8, and get parallel "builds" of the results for free.

As long as it's documented, it can save a lot of repetition.

Not coincidentally, jgc has a nice article on make debugging ;-)

http://drdobbs.com/article/print?articleId=197003338&sit...

reedhedges2• 14 years ago

I'm not saying you can't do it. Make is really powerful and fun once you take a few minutes to think about how it is supposed to work (rules that generate file outputs), but a lot of programmers don't (who treat rules as procedures resulting in a file thats 90% phony rules). You have to be very careful with define to generate rules. Just be aware there are syntax pitfalls, compatibility pitfalls (between relatively recent versions of make), and just confusion if you aren't clear on what you're doing.

Peaker2• 14 years ago

Make should really be superceded by a build system that is both faster and more correct.

One of these options today is the tup[1] build system.

It seems to be a relatively modest/new effort, and is already far better than Make.

[1] http://gittup.org/tup/

CJefferson2• 14 years ago

The big advantage of Make of course is that you can assume it is basically everywhere.

I have just looked at tup, and it looks interesting, except I don't really understand why the auto-dependancy only kind-of works. If building X.c uses header.h, and header.h is created by script.sh, why do I have to tell tup about that ordering? Why can't it just see it has to run script.sh to get header.h, before building X.c?

evmar• 14 years ago

The build system can't know beforehand which scripts generate which files (unless you're implying it ought to run everything that looks like it might be a script and observe what outputs it writes).

However, once you've compiled a .c file the first time (or run the script the first time), the build system can remember which inputs and outputs were involved (either because gcc tells it which files it used, or by going lower-level and intercepting the file-opens of the program) and reuse that information in subsequent builds.

Peaker2• 14 years ago

The build system does record all the actual dependencies and:

1) Only executes stuff after their actual dependencies changed (over-specified dependencies are ignored here)

2) Yell if dependencies are under-specified (can't access files not specified as dependencies)

3) Correctly handle parallelism -- this is where overspecifying dependencies may hurt somewhat, because it can under-parallelize. However, if inputs change -- the dependencies can also change, so if you use previous dependency lists you will also get parallelism wrong. So parallelism makes it a harder problem.

If you don't care that much about fine-grained parallelism, you can simply over-specify your dependencies, and tup will do the right thing.

mturmon2• 14 years ago

A similar version of this idea is also in the gnu make manual, at:

http://www.gnu.org/software/make/manual/make.html#Eval-Funct...

Using GNU Make's 'define' and '$(eval)' to automate rule creation

Related Stories