An opinionated approach to GNU Make

by DarkCrusader2 • 7 years ago

414 194 7 years ago

jgrahamc3• 7 years ago

For one-line recipes you can use semicolon

    foo.o: foo.c ; $(CC) $<

If you find $(@D) hard to remember then do $(dir $@) instead.

I wrote a book about GNU Make stuff (https://nostarch.com/gnumake). If (and only if) you've read the GNU Make Manual from the FSF my book may help you. It's not for newbies. Don't want the book? All the articles are here: https://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-art...

mbrock• 7 years ago

With backslashes before newlines you can do “multiline oneliners” too.

jgrahamc3• 7 years ago

Yep. One of the reasons I recommend the FSF manual is that it's a great introduction: https://www.gnu.org/software/make/manual/make.html#Splitting...

dvdkhlng• 7 years ago

You can also do "multiline multiliners" by adding a target .ONESHELL. By overriding the variable SHELL, this allows you to put arbitrary multi-line scripts of your preferred programming language into makefile recipes.

https://www.gnu.org/software/make/manual/html_node/One-Shell...

sweettea• 7 years ago

I can attest that it's a great book. It took my Make savvy from "pretty average" to "can build Makefiles that conform pretty decently to DRY". Not all of the book is useful to me, but it's a pleasant read and also a good well-indexed reference manual and I consult it at least monthly for fun.

bilekas• 7 years ago

$(CXX) -MM $(CPPFLAGS) $< | sed 's,$$$\.o[ :],\1.o $@ : ,g' > $@

My makefiles are beautiful... To me

And they're sure as sh!t not wrong, or they wouldn't work! :)

rs23296008n1• 7 years ago

Looks like your cat pressed a few keys when it jumped up onto your desk.

iforgotpassword• 7 years ago

I ragequit as soon as he said not to use tabs. The article is invalid.

loriverkutya• 7 years ago

He is completely right about that, sadly.

bregma• 7 years ago

Ah, yes, dial-up modems on a noisy line. I remember the 1990s like they were just yesterday.

anon9001• 7 years ago

Experimental, but this exists: https://github.com/mrtazz/checkmake

calpaterson• 7 years ago

Terrible clickbaity title that he walks back throughout the entire post - "this is not dogma". There are a couple of valid remedial notes in here (like using files as targets) but the part about changing the default shell and inserting hacks that allow you to use spaces will just baffle anyone else and probably make your Makefiles non-portable (eg macs don't have a recent bash).

lloeki4• 7 years ago

macs don't have a recent bash

Macs don't have GNU make 4, only 3.81, so it's outright not portable with that > thingie.

alwillis4• 7 years ago

Use Homebrew [1] to install the most current versions of bash and make.

By default, it’s installed as gmake to not conflict with the existing make 3.81.

[1] https://brew.sh/

blattimwind2• 7 years ago

Macs don't have GNU make 4, only 3.81

Released April 1st, 2004.

zeveb• 7 years ago

probably make your Makefiles non-portable (eg macs don't have a recent bash).

Is there any point in supporting a platform which hasn't updated its major version of Make in over 15 years?

microtherion3• 7 years ago

Picking the platforms you support by which version of make they ship is a rather odd way of doing business.

OskarS• 7 years ago

The "replace the tab with magic character" thing particularly baffling. I agree that it was probably a dumb idea in the beginning for make to force you to use a literal tab character, but using these kinds of hacks is a terrible idea. It will just make it harder to read makefiles for normal people and it will make them less portable. Every sensible programming editor knows that make is particular about spaces and tabs and will make sure that it uses tabs to indent the recipes, so it's not an issue any more. If anything, using some other character will make it more confusing, because I'm guessing most editors will indent/syntax highlight it incorrectly.

It's not dissimilar from early C programmers who wanted C to be more like Pascal and put things like this in headers:

    #define BEGIN {
    #define END   }

and then never used curly braces. NO! DON'T DO IT!

primis• 7 years ago

Well now I have to try it

panpanna8• 7 years ago

I really don't understand his justification for such a significant modification:

Make leans heavily on the shell, and in the shell spaces matter. Hence, the default behavior in Make of using tabs forces you to mix tabs and spaces, and that leads to readability issues.

I agree with you in that replacing TAB with > makes things much less readable.

gnulinux• 7 years ago

When people write software, they don't need to target any and all platforms that can possibly operate on this software as input. If you want to only target GNU, that is perfectly fine. Just name your file `GNUmakefile` or put a comment at the header or in `README` to indicate you only support GNU.

KindOne• 7 years ago

macs don't have a recent bash

macOS Catalina now uses zsh.

alwillis4• 7 years ago

The default for new accounts in Catalina is zsh but if you upgrade, you'll still have bash: https://support.apple.com/en-us/HT208050

majewsky• 7 years ago

And you will never again pull your hair out because some editor swapped a tab for four spaces and made Make do insane things.

steps up to you and whispers in your ear

Or you could, you know... use a real text editor.

donatj• 7 years ago

I am really curious about how this would happen.

I have been working with Makefile’s, with a team, for close to ten years and have never encountered anyone accidentally using spaces. I guess someone editing it via GitHubs web ui or something.

bauerd2• 7 years ago

I can give an example.

Recently I was working on a project with multiple Makefiles, ie Makefile.{win,nix}. My Emacs config did not load makefile-mode due to the suffixes. As spaces are my default, I got spaces on TAB, iirc even after turning on makefile-mode.

GordonS6• 7 years ago

Happened to me just the other day.

I added a makefile to a project that didn't yet have one, and I didn't have makefile support added in my IDE.

Got some weird, unhelpful errors when trying to run make, and eventually realised spaces were being used instead of tabs.

The solution was to configure my .editorconfig file to use tabs for makefiles, and as a nice-to-have, to also install a makefile plugin in my IDE.

wostusername• 7 years ago

I've copied lines from a shell script into a makefile and it copied the leading spaces as well. I didn't really think about it and when I ran make I got a bunch of weird errors. I figured out what the problem was pretty much immediately. Overall this was just a minor annoyance for a minute or two, but it does happen. I'm not sure why people pretend that it doesn't.

I don't work with make much anymore (or Python or Haskell), but I've come to see the use of invisible characters to express structure as a mistake. It can make code real neat and tidy looking, but I'm not sure the trade off is worth it. In the past I've resorted to actually having my editor show whitespace symbols to make sure code that looks aligned actually is aligned.

godshatter2• 7 years ago

I ":set expandtab" in vim by default, I don't like mixing tabs and spaces. People have different opinions on this, I get it. So when I pop on over to the Makefile buffer, I have to remember to either ":set noexpandtab" or to copy a previous target down and edit it directly. Now I'll just swap out ">" instead of using tabs, which will make things simpler.

geofft4• 7 years ago

I've done it, thanks to using terminal copy/paste (either my actual clipboard, or GNU screen's copy buffer) instead of my editor's built-in copy/paste. Sometimes it's because I'm viewing a patch in a web UI or `git show` and pasting a few lines from it into my text editor.

But you notice your mistake very quickly and I've never sent code to code review with spaces instead of tabs.

rob74• 7 years ago

Ok, what's really insane is that this tool that does insane things if you accidentally use spaces instead of tabs is still the standard tool used to build software all over the world...

mixmastamyk3• 7 years ago

Indeed, folks make similar comments about Python too. Such a mix up happened to me one afternoon in 2002 or so when copy pasting code.

So, I configured my syntax highlighting to very subtly show whitespace characters. The editor keeps them consistent in every other case. Have never had the problem again since.

clarry4• 7 years ago

Ok, since we're sharing controversial opinions...

If you're going to rely on a fancy shell anyway, why not just throw make out of the loop altogether? That is, unless you're working on a big project where incremental builds really make a difference. (But IME, with a few exceptions, these are the projects that usually outgrow and abandon make anyway)

You can run cc via shell via make, or you can just run cc via shell. In the latter case, there's one less program (with quirks) to fight with, and more flexibility to do stuff that you can't easily bend make to do.

lloeki4• 7 years ago

Because the core proposition value of make is the dependency graph and incremental rebuild, which then trivially allows for parallelisation with -j<n>, and doing that with shell scripts is basically reimplementing that part of make, only badly.

Instead of pushing for makefiles to become shell scripts for the part you mention, I usually implement such things as support scripts (in shell, python, whatever) - who often come in useful on their own - and call those from make targets. IOW using the best tool for each job, with make as the orchestrator.

clarry4• 7 years ago

Because the core proposition value of make is the dependency graph, which then trivially allows for parallelisation with -j<n>

My core argument is that this rarely matters, and in the cases it does, it is often the case that the project's needs surpass what make provides anyway. Or they're using a very special version of make with very special supporting infrastructure, like bsd.mk.

I'm compiling some 100k lines of legacy C (for a custom operating system) plus compat shims that make the thing run on Linux and my very very trivial build script (plain posix compatible shell) takes less than 5 seconds to run in debug mode. Optimized release builds are a bit slower, but this hardly matters in day to day development. It would be trivial to add some parallelism with xargs.

Fwiw I can compile the same thing with CMake, and it's faster only sometimes. It often happens that a header changes and everything needs to be built anyway. And it often happens that the parallelism bites me back when one file gives an error which scrolls out of sight as a dozen other files are still being compiled in parallel.

I've wasted far more time fighting subtly broken makefiles (that don't pick up some dependencies and thus fail to recompile the things that need to be recompiled), and build systems with fancy declarative languages that don't present an obvious way to do what you do in one line of shell when you don't need to worry about a dependency graph.

jakewins• 7 years ago

Author here: Exactly this. We keep a `bin/` in the root dir with scripts/programs for more involved steps. Make gives us the dependency graph, incremental rebuilds and parallelization.

Sean1708• 7 years ago

Because the core proposition value of make is the dependency graph and incremental rebuild, which then trivially allows for parallelisation with -j<n>,

In my personal experience 90% of Makefiles I come across not only don't take advantage of these, but actually have to actively employ workarounds to avoid these things. If you're taking advantage of the things make has to offer then by all means use make, but if all of your targets are .PHONY and you're having to turn off make features just so you can put all your bash scripts in one file then maybe just put your bash scripts in separate files.

BeetleB• 7 years ago

Because the core proposition value of make is the dependency graph and incremental rebuild

So you are in agreement with your parent? He's suggesting make for these use cases.

The issue is that a lot of times people use Makefiles for putting convenient commands that have little to do with dependencies/incremental builds.

jorge-fundido• 7 years ago

If you're going to rely on a fancy shell anyway, why not just throw make out of the loop altogether?

I think can still add value in that scenario as the standard project entrypoint. Someone completely new to the project should be able to `make help`, `make install`, etc. Even if the make targets are simple wrappers to project specific tooling (npm, pip, sbt, etc). If the Makefile is kept simple, then users can treat it as a form of README and should be able to cherry-pick from it as they see fit.

But yeah, I share your sentiment about quirky, non-portable Makefiles potentially being anti-value.

clarry4• 7 years ago

I certainly like it when I encounter a new project, type make, and it just works.

This is rare these days.. honestly at this point there's so much tooling already around that one can't really ever take that for granted. Almost always, I need to look for some readme or look around and figure out what build system is in use (sometimes it's a common system plus custom stuff so just seeing a name you recognize isn't automatically going to mean the standard invocation will work).

In this case, it hardly matters whether I'm going to run make or ./build or make help or ./build help. A simple script, which I'm advocating for simple projects, can double as a form of README just as a Makefile can.

GordonS6• 7 years ago

This came up just a few days ago[0], and someone suggested you could use either of these:

    #!/bin/sh
    set -e

    case "$1" in
        build)
            ;;
        run)
            ;;
        clean)
            ;;
        *)
            echo "unknown: $1"; exit 2
            ;;
    esac

    #!/bin/sh
    set -e

    test $# -gt 0

    build() {
        :
    }

    run() {
        :
    }

    clean() {
        :
    }

    "$@"

Personally, I find makefiles to be ubiquitous and well-understood, and if I want to do anything complex, I can always put that complexity in a shell script that's called by a make target.

[0] https://news.ycombinator.com/item?id=21735176#21754331

jtaft3• 7 years ago

The second example is vulnerable to command injection.

./test.sh ls -la

simias2• 7 years ago

Besides performance, I do enjoy the feedback of incremental builds. It's not a huge deal but it's not uncommon that I spot a problem just because the build doesn't behave in the way I expected. Out of the top of my head:

- I forget to add a dependency to the Makefile, I modify said file and notice that `make` doesn't do anything, hinting that something is wrong. If the build system rebuilds everything every time I probably won't notice the problem immediately.

- I edit code in a project but run make in the wrong location (because I'm working on two projects at one for instance). Again, make will tell me that there's nothing to do when I make changes.

- I run the latest build and it behaves in an unexpected way, I'm thinking that maybe I haven't actually built the last version of the source so the code I'm looking at isn't what's being executed. If I run "make" and I see that it starts rebuilding some artifacts I know that I'm out of date and need to try again. If nothing happens then I need to stop looking for excuse and actually start debugging.

- I pull code written by other people (or apply a patch, or unstash old changes). Running make will give me some insight on what parts of the codebase have been impacted. If I see that it's rebuilding something I didn't expect to change it'll probably catch my eye and get me to double check what happened.

So I definitely wouldn't trade a Makefile for a shell script, especially since if you build system can easily be replaced by a simple script you'll probably be able to do just fine with a very basic Makefile.

Myrmornis• 7 years ago

It’s nice to use make to provide a uniform interface for diverse projects, including random personal projects. I.e. whatever project you’re in, in whatever language etc, you know what the semantics of `make build` and `make test` are going to be. Even if these recipes are trivial the uniform interface is a win.

sirn3• 7 years ago

If you're using GNU-specific Make features, please, please consider naming the file GNUmakefile instead of Makefile and use $(MAKE) for recursion. GNU make will happily pick up GNUmakefile before Makefile, and using $(MAKE) will remove a lot of headaches when people try to build your project outside of author's $PREFERRED_PLATFORM.

swiley• 7 years ago

Do you have a good guide for writing portable makefiles? I’m always using GNU make so I don’t notice it with mine but I always feel a little bad.

sirn3• 7 years ago

I don't think there's a guide, the only resource I'm aware of is The Open Group Base Specifications[1] which can be reference when in doubt. Almost other times I just install bmake alongside with gmake and try to get it work with both (it's still not perfectly conform to POSIX, since GNU supports some of BSD's make feature, and vice versa).

[1] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/m...

nipav• 7 years ago

The best way is reading the spec: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/m...

But to get you started Chris Wellons does have a pretty nice blog post about writing POSIX Makefiles: https://nullprogram.com/blog/2017/08/20/

asveikau3• 7 years ago

I use the *BSDs a lot and it's pretty common to install gmake from ports. I wouldn't feel bad.

kazinator• 7 years ago

The POSIX standard?

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/m...

If you don't use anything that is not there, or in a manner not covered by a requirement, you will likely have a highly portable result.

pmarin• 7 years ago

There is a very good tutorial by Chris Wellons

https://nullprogram.com/blog/2017/08/20/

nullifidian8• 7 years ago

Does somebody even use other make implementations anymore? I think such suggestions aren't relevant nowadays. Same goes for bash. Posix compatibility is overrated.

zingermc2• 7 years ago

It's still 100% relevant. I can only speak for FreeBSD, but this is a real pain when building from source (outside of FreeBSD Ports).

https://www.freebsd.org/cgi/man.cgi?make(1)

rurban• 7 years ago

On most other operating systems than Linux gmake != make. Darwin, bsd's, Windows, Solaris, ...

sirn3• 7 years ago

BSD make (pmake) is still very common, as it's the default make in almost all BSDs. Also Windows has nmake which is kinda-but-not-quite POSIX-compatible.

cies• 7 years ago

Learned some good tricks here. There's one that I use but did not find in the article. This one:

    # Borrowed from https://marmelab.com/blog/2016/02/29/auto-documented-makefile.html
    .PHONY: help
    help: ## Display this help section
        @awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z0-9_-]+:.*?## / {printf "\033[36m%-38s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST)
    .DEFAULT_GOAL := help

    .PHONY: load-terraform-output
    load-terraform-output: ## Request the output JSON from Terraform 
        some commands
        some more

It makes Makefiles a little more self documenting.

Now I issue a `make` (or `make help`) to get a listing of the documented tasks. Very helpful.

RVuRnvbM2e• 7 years ago

This is great, thanks!

bogwog3• 7 years ago

That's awesome! I've always been annoyed that there isn't a built-in `ls` like feature, but this is a really clean solution.

Hello71• 7 years ago

    make -p

the main problem is that make itself doesn't support a concept of "hidden files". also a problem is that GNU make comes with lots of implicit rules, so you really need something like make -R -r -p for this to remotely work.

manwe1502• 7 years ago

On the subject of good tricks, I like to put this debugging snippet in all my Makefiles. It supports typing `make print-VAR1 print-VAR2` and it'll dump the values of VAR1 and VAR2 (this could be extended to also show `$(flavor $star)` and `$(origin $star)` and other little details, but usually I just simply want the final value).

    define newline # a literal \n


    endef
    # Makefile debugging trick:
    # call print-VARIABLE to see the runtime value of any variable
    # (hardened a bit against some special characters appearing in the output)
    print-%:
            @echo '$*=$(subst ','\'',$(subst $(newline),\n,$($*)))'
    .PHONY: print-*

oso2k• 7 years ago

I like to use a `showconfig` rule so that `make showconfig` will display all public vars and give their default values. It ends up being a little like `./configure --help`. But I see now that I could tighten that up a bit with the above rule and something like `showconfig: print-VAR1 print-VARX`. Thanks!

m463• 7 years ago

Make is useful, but I really wish the implementation had evolved over the years, making stuff like this natural.

As an example, python has doc strings accessible via __doc__

  $ python
  >>> class foo:
  ...   '''description of foo'''
  ... 
  >>> foo.__doc__
  'description of foo'
  >>>

commandersaki• 7 years ago

I learned about this trick from the same blog post. I tweaked a little to suit my needs. But wow, it is such a game changer. This means all you need to write in auxiliary docs is 'make help'. That's it.

gonzus• 7 years ago

This is a great trick, thanks! Much appreciated.

rwmj3• 7 years ago

I wonder if anyone has ever generalized "make" so that it can operate on general dependencies and tactics rather than always on files? The sort of thing I mean is that you'd be able to write:

    url_exists(http://example.com/file): file
        rsync file example.com:/html

I've had a few attempts at this (most recently in 2013: https://people.redhat.com/~rjones/goaljobs/) but never really arrived at anything satisfactory.

q3k• 7 years ago

This has also been a holy grail for me. The closest I've come to this are custom starlark rules in Bazel.

lima• 7 years ago

This. Bazel is as close as it gets.

boris• 7 years ago

We did in build2[1]. Specifically, we have a notion of target type which may not be path or mtime-based. For example, one such target type is alias (something that is achieved in make with PHONY targets). So you could define, say, a url target type. Then you can implement a rule that knows how to perform a certain operation on certain target types (we've also generalized a notion of operation so in build2 test, install, dist are operations, not targets).

[1] https://build2.org

skore3• 7 years ago

Well, kind of.

I'm using make to build a content graph, part of that is http:// dependencies. The underlying concept is that you can define protocols (https would be one) and then teach my make pre-parser about building them. An example would be:

    myfile.js: https://code.jquery.com/jquery-3.4.1.js local.js
        uglifyjs $^ -o $@

The 'https://' is rewritten to map to a local cache directory (so it's little more than a glorified variable, but still more intuitive since protocols are so familiar from our use of the web), so the above code is actually something like

    myfile.js: cache/https/code.jquery.com/jquery-3.4.1.js local.js
        uglifyjs $^ -o $@

and that in turn invokes a rule that makes a HEAD request to the URL - if that is outdated against the local copy, that gets renewed. So, in regular make code, something like this:

    cache/https/%: cache/https/%.head
        curl -Ls https://$* > $@

    cache/https/%.head: cache/https/%.head.force
        rsync --checksum $< $@

    cache/https/%.head.force: FORCE
        curl -Ls -I https://$* > $@

I could see a way to turn this around, though. The concept of "protocols" is currently only GET in my setup, but I do plan on supporting the idea of POST, eventually. So what you're asking for is that, I guess.

I must say, though, that in my now three years of using (or abusing) make, "just make it another file" is surprisingly often the right next step. In the POST situation, it would simply be a local file that has direct or side-effects attached that do what you're expecting it to do.

lloeki4• 7 years ago

"just make it another file" is surprisingly often the right next step

This way it also often turns out that the "fetch"/"cache" part gets naturally separated from the rest, and that's super useful to track down any permanent or transitory issue.

rwmj3• 7 years ago

Perhaps the part I'm missing is how you are rewriting the https:// URLs to the paths. Do you preprocess the Makefile?

nerdponx• 7 years ago

There are literally hundreds of build tools and workflow engines that replace or extend some or all of Make. Many of them are domain-specific (especially to bioinformatics or machine learning), many of them are not.

rwmj3• 7 years ago

Any links to any of these tools?

floor_• 7 years ago

If you're doing anything other than a unity build (single compilation unit) you're doing it wrong.

chrismorgan2• 7 years ago

Concerning creating directories:

  out/foo:
  > mkdir -p $(@D)
  > …

I prefer to use this pattern, which lets you have many fewer `mkdir -p …` lines in the make output when you use it a bunch of times:

  out/foo: | out
  > …

  out:
  > mkdir "$@"

(Here using the > recipe prefix from the article, though you won’t find me doing that in real life—I like my tabs.)

jgrahamc3• 7 years ago

My take on making directories: https://www.cmcrossroads.com/article/making-directories-gnu-...

chrismorgan2• 7 years ago

That looks to be a good article, explaining the reasoning behind the different alternatives. What I wrote matches your Solution 4—that pipe is indeed a subtle beast! I’ve definitely combined this with fancier patterns and second expansion as well.

schnable• 7 years ago

I liked Make due to my perceived simplicity of it. Thanks to this article, I think I'll just use Rake next time.

pferde2• 7 years ago

I like make because it can be as simple as you want it, yet powerful if you need it to be.

megous• 7 years ago

For real simplicity I like ninjabuild.

I use it in cases where Makefiles would be a real mess to maintain and in general where I want to paralelize a set of file transforming jobs and have a nice progress indicator and ability to cancel and continue.

I just have a nice PHP generator class for ninja files and I build ninja files programatically in a real programming language based on whatever I like.

With this approach it's possible to fully parallelize builds accross however many different toolchains and SDKs with relative ease and without need to deal with quirks, or limitations of languages of various build systems or build tools that try to be semi-universal (like GNU make) but are not, or have a lot of non-obvious magic integrated.

And I also live how ninja tracks changes in its own build rules and automatically rebuilds targets that would be affected, which is what I always disliked about makefiles. If you change makefile rule you basically have to do `make clean ; make`.

sounds• 7 years ago

I'll second you on ninjabuild.

I just wish https://github.com/ninja-build/ninja/pull/1578 would get merged, so I can compose ninjabuild repos seamlessly.

wyldfire• 7 years ago

MAKEFLAGS += --no-builtin-rules
This disables the bewildering array of built in rules ...

Oh, wow, I totally disagree. It's good to be opinionated -- that is, unless you're wrong. ;)

In all seriousness I find it terribly useful to quickly create simple Makefiles and follow idioms like CFLAGS/CXXFLAGS/LDFLAGS/LDLIBS/etc.

Tyr422• 7 years ago

I think putting those rules in a standard lib you could include would be awesome. I would then be able to open up the file and see check if it was CPPFLAGS or CXXFLAGS, or whatever, and see how it's written. It'd be way more discover-able, AND it would make make much nicer to teach.

I have tried to teach make, but these implicit rules firing when I'm just trying to do a simple example can mask errors and create a lot of confusion.

Also, not looking at all the VCS versions of the file makes make run much faster, so being modular in what you imported would be a win for bigger systems too.

saagarjha3• 7 years ago

CPPFLAGS are meant to be passed to the C preprocessor, while CXXFLAGS are passed to the C++ compiler.

alxmdev• 7 years ago

I've recently been lightly toasted by GNU Make's implicit "old-fashioned" suffix rules, where it was looking for YACC files as prerequisites for my own generated C code. The manual suggests a workaround that clears the implicit suffixes and also allows you to add back the ones you want. For me, clearing all of them with an empty ".SUFFIXES :" rule was exactly what I needed.

https://www.gnu.org/software/make/manual/html_node/Suffix-Ru...

microtherion3• 7 years ago

If you think implicit yacc rules are "old-fashioned", let me introduce you to GNU Make's… implicit SCCS rules!

knorker• 7 years ago

So force GNU make and bash as shell?

No thanks.

anticristi• 7 years ago

I hope I'm not carried away by the example, but I would definitely not build Docker images using make. A multi-stage Dockerfile can take care of building, testing, packing, etc. source code a lot more predictably than any amount of make, by using known images of the tools involved (e.g., node, webpack, etc.).

Moreover, Docker's caching implicitly allows to declare dependencies with a lot fewer headaches than make. On the downside, it is all to easy to write something that busts Docker's cache early in the build process, rendering `docker build` super-expensive.

So my opinionated tooling: docker, git (to extract project versions and inject them into the image), bash (to glue everything together). Everything else belongs inside the Docker build.

GordonS6• 7 years ago

I like to use a simple makefile that runs `docker build` and `docker push` commands, automatically tagging images based on the current solution version and git head. That means I can simply do `make build|rebuild|release`, and get images all properly tagged.

But yeah, all the actual work is in the Dockerfile.

m_mueller• 7 years ago

I do just the same, but replacing docker by docker-compose. I like its Yaml files for managing settings around docker and plugging multiple containers together, you'd reinvent a lot of that by omitting it.

bogwog3• 7 years ago

This is a good list, but I have to disagree on the tab thing.

And you will never again pull your hair out because some editor swapped a tab for four spaces

How many editors out there in 2019 will automatically replace tabs with spaces by default? Unless you're editing makefiles in Microsoft Word or something, I don't see why a working code editor will do something you didn't tell it to do.

Vim, Emacs, Nano, Sublime Text, Kate/Kdevelop, Visual Studio/Code, Atom, Brackets, Text Mate, Scintilla/SciTE/Geany/etc, Programmers Notepad, CodeBlocks, Eclipse, and JetBrains all know not to mess with your tabs.

Rather than switch away from tabs so you can keep using a broken editor, the correct solution is to switch an editor that works. And if you configured your editor to replace tabs with spaces, and it doesn't give you an option to handle Makefiles differently, then that's a broken editor.

e12e• 7 years ago

I'd add: knows not to mess with your tabs in makefiles.

I generally use expandtabs in vim - but I still don't break makefiles. I suspect it's quite possible to force vim to expand tabs in makefiles.. But why would I want to?

Tyr422• 7 years ago

I 120% recommend MAKEFLAGS += --no-builtin-rules

If I could rewrite history, I would make those builtin rules nicely, explicitly, importable. No magic when I'm trying to teach make, please.

juped• 7 years ago

More derangements to undo when you want it to build on a different system.

donpdonp• 7 years ago

.ONESHELL is a huge help, and replacing the recipe tab is very interesting. Great post!

bobbyi_settv• 7 years ago

The problem with this:

    out/image-id: $(shell find src -type f)

is that if you delete a file from src (without also modifying or adding a file), make won't consider image-id to need rebuilding

microtherion3• 7 years ago

Another problem is that if one of the input files itself is generated, it might not get rebuilt after a "make clean", because the ultimate target no longer knows it's dependent on it.

dima55• 7 years ago

FYI, there's a patched Make providing (among other things) an interactive debugger:

https://github.com/rocky/remake

kevingadd• 7 years ago

Remake is wonderful, but note that many bad makefiles invoke 'make' directly instead of using '$(MAKE)', so remake won't work properly until those makefiles are all corrected.

morelisp• 7 years ago

"Opinionated" sure is right.

I love make, but a lot of this advice is targeted at being able to write "better" shell scripts in make. I don't recommend it. If you find yourself writing a shell for loop, you probably want instead to build a list of targets. If you find yourself wanting complex shell variable preparation, you probably instead want target-specific variables. ONESHELL is a good way to accidentally build some invisible dependencies into a recipe, and make it difficult to use custom functions or canned recipes.

If you do find yourself really wanting a shell, you'll probably also want "advanced" features like error traps, and you'll probably want to work with tools like shellcheck (imo critical for any shell script longer than one pipeline). Both are thwarted by baking the invocations into your make recipe. And the recipe still looks great - probably better! - if you extract any logic into a separate script (which then also opens up further possibilities, like including that tool itself in a shell pipeline in a make recipe).

mvaliente2001• 7 years ago

Another trick, create a `help` target:

    help:                      ## Show this help
        @fgrep -h "##" $(MAKEFILE_LIST) | fgrep -v fgrep | sed -e 's/\\$$//' | sed -e 's/##//'
    
    foo:         ## foo help
        # make help will print the names of the targets with 
        # the help message following ##

I don't remember the source, I think I saw it in a stackoverflow question.

Porthos9K• 7 years ago

I'm glad the author specified that this is mainly for GNU make, because I suspect that if I tried most of these using OpenBSD make I would have a bad time.

einpoklum• 7 years ago

To be honest - I don't believe I've had to write a single Makefile since I started using CMake.

At most I've had to tinker with existing ones - in which case I wasn't motivated to make them more elegant, as opposed to replacing them with CMake generation.

KerrickStaley• 7 years ago

My 2c on GNU Make is that you should not use it for new projects. There are better build systems, like Bazel, that are faster and produce consistent, reproducible results.

Make's lack of hermeticity means that it's easy to accidentally craft a Makefile where there are dependencies between targets that are not explicitly listed in the Makefile. When this happens, you can't be sure that when you change an input file and run `make`, all the dependent targets will be rebuilt. This means you often have to `make clean; make` to make sure that things get correctly rebuilt.

Bazel also supports things like caching of artifacts so that when you build, switch branches, build, switch back, and build, it can re-use built artifacts from build #1 so that build #3 becomes a no-op. With remote caching, this can happen across computers and users; when user 1 builds at SHA 123, and then user 2 later checks out SHA 123 and does a build, user 2's build will simply copy the cached artifacts over the network and do no local work. (There is some operational overhead to maintaining this shared cache however).

That said, guides like this are really helpful for maintaining and extending existing make-based build systems!

nrclark7• 7 years ago

Agreed that a pure Make approach is less than ideal for new software projects. It's great as an orchestration tool though. I use Make for all kinds of wrappers, because it gives built-in dependency ordering and tab-completion. Make is also a good backend for systems like CMake or Autotools.

I manage a couple of Yocto Linux-based firmware builds for my employer. Each project has a top-level Makefile that wraps the underlying build tools, so the build process is as easy as cloning the repo and running "make".

Make is also great for all of the post-build orchestration steps like "grab these files, rename this one, generate a little report, and make a tarball of the result". Steps that are each a little snippet of shell-script, and have some ordering dependency.

A 'build.sh' type shell-script would work too, but wouldn't include all of Make's built in features and dependency management. And you'd need to write stuff if you wanted it to parse specific target requests. At which point - why not just use Make?

So I guess I really like Make for the role of "be the top-level glue around other build systems". It includes dependency-management and file-generation rules, works out parallel ordering where possible, and comes with tab-completion on every desktop Linux.

Plus every developer who uses the command-line understands that a Makefile means "stuff can be built from inside this folder". They also probably know that "make" or "make all" is likely to run the build, "make install" will probably install it, and "make clean" will probably clean the folder. These conventions have deep staying power. And builds that follow them can be used by a lot of engineers with no further training or documentation required.

alwillis4• 7 years ago

I’ve never used Make in my life until a few weeks ago.

It’s surprisingly good for managing the build process for a web project that various build tools like Gulp, Grunt, etc. have been created to handle.

I finally ended up using npm scripts and it was easy to convert them into a Makefile.

armitron• 7 years ago

Don’t agree at all.

Make is the standard and available everywhere, Bazel is not. Everyone can/should be able to debug Make. Bazel is yet another adhoc solution in a problem space full of adhoc solutions. Which is great if one needs the features that Bazel offers, but I dare say that the vast majority of projects out there do not and are better served by Make.

Annatar• 7 years ago

This is chalk full of bad advice like "use GNU make and bash" that it's obvious to me it was written by someone who only knows GNU/Linux and doesn't know UNIX (or much else). Blind leading the blind again. Don't write GNU-specific constructs unless you have no other choice; use ksh instead of bash, it's portable and a standard and has been long before the GNU/Linux abomination. What nonsense I get to read...

nrclark7• 7 years ago

If somebody is targeting inter-Unix builds, Autotools is probably a better pick than hand-rolling a Makefile.

But realistically, Linux has captured 75% or more of the entire *nix market. A lot of developers spend their entire careers in Linux these days, and that trend isn't likely to swing back towards other Unix flavors anytime soon.

As long as a developer understands what users will be using their Makefile, it shouldn't matter if they target platform-specific features. The BSD ports tree uses a bunch of BSD Make features that aren't in GNU Make. And that's totally fine, because they know their target users.

Like let's say I'm writing a Makefile that wraps tools which don't run on anything other than Linux. Why would I care about portability across Make implementations?

If a dev doesn't know their users' environments (other than being POSIXy), probably they shouldn't be writing raw Make of any flavor. And if they do know their users, then it isn't a big deal to rely on some platform-specific features.

jrockway• 7 years ago

I think I have decided that I'm done with Makefiles. They are very tempting, because they follow naturally from interactive exploration. You see yourself writing a command a lot, and think "I'll just paste that into a Makefile". Now you don't have to remember the command anymore.

But the problem is that building software is a lot more than just running a bunch of commands. The commands represent solutions to problems, but if the solutions aren't good enough, you just make more problems for yourself.

The biggest problems I've had with Makefile-based builds are getting everyone using the repository the right version of dependencies, and incrementality. A project I did at work involved protos, and it was great when I was the only person working on it. I had a Makefile that generated them for Go and Typescript (gRPC-Web) and usually an incremental edit to a proto file and a re-run of the Makefile resulted in an incremental update to the generated protos. Perfect. Then other people started working on the project, and sometimes a simple proto change would regenerate the entire proto. Sometimes the protos would compile, but not actually work. The problem was that there is a hidden dependency of the proto compiler, protoc-gen-(go|ts), and the language-specific proto API version that controls the output of the proto compilation process. Make has no real way to say "when I say protoc, I mean protoc from this .tar.gz file with SHA256:abc123def456..." You just kind of yolo it. Yolo-ing it works fine for one person; even if your dev machine gets destroyed, you'll probably get it working again in a day or two. As soon as you have four people working on it, every hidden dependency destroys a day of productivity. I just don't think it's a good idea.

Meanwhile, you can see how well automated dependency management systems work. Things like npm and go modules pretty much always deliver the right version of dependencies to you. With go, the compiler even updates the project definition for you, so you don't even have to manage files. It just works. This is what we should be aiming for for everything.

I have also not had much luck with incremental builds in make. Some projects have a really good set of Makefiles that usually results in an edit resulting in a changed binary. Some don't! You make a change, try out your binary, and see that it decided to cache something that isn't cacheable. How do you debug it? Blow away the cache and wait 20 minutes for a full build. Correctness or speed, choose any 1. I had this problem all the time when I worked on a buildroot project, probably because I never understood what the build system was doing. "Oh yeah, just clean out those dep files." What even are the dep files? I never understood how to make it work for me, even after asking questions and getting little pieces of wisdom that seemed a lot like cargo-culting or religion. Nobody could ever point to "here's the function that computes the dependency graph" and "here's the function that schedules commands to use all your CPUs". The reason is... because it lives in many different modules that don't know about each other. (Some in make itself, some in makefiles, some in the jobs make runs... it's a mess.)

Meanwhile, I've also worked on projects that use a full build system that tracks every dependency required to build every input. You start it up, and it uses 300M of RAM to build a full graph. When it's done it maxes out all your CPUs until you have a binary. You change one file, and 100% of the time, it just builds what depended on that file. You run it in your CI environment and it builds and the tests pass, the first time.

I am really tired of not having that. I started using Bazel for all my personal projects that involve protocol buffers or have files in more than one language. The setup is intense, watching your CPU stress the neighborhood power grid as it builds the proto compiler from scratch is surprising, but once it starts working, it keeps working. There are no magic incantations. The SHA256 of everything you depend on is versioned in the repository. It works with traditional go tools like goimports and gopls. Someone can join your project and contribute code by only installing one piece of software and cloning your repository. It's the way of the future. Makefiles got us far, but I'm done. I am tired of debugging builds. I am tired of helping people install software. "bazel build ..." and get your work done.

An opinionated approach to GNU Make

Related Stories