GNU Make Jobserver Implementation

http://make.mad-scientist.net/papers/jobserver-implementation/

by krakensden • 10 years ago

59 20 10 years ago

amelius• 10 years ago

a traditionally hairy problem in UNIX programming: how to wait for two different types of events (signals and file descriptors) at the same time.

I've always thought one of the weak points of POSIX was that the event sources were not unified. Imho, is a pity the relevant committee didn't work towards a clean solution in the mean time.

okigan• 10 years ago

Same here! Also annoying is inability to wake up when any (vs all) child threads are done.

Even more annoying is that Windows API allows to wait on any combination of file/socket descriptor and/or message[1].

[1] MsgWaitForMultipleObjects https://msdn.microsoft.com/en-us/library/windows/desktop/ms6...

uep• 10 years ago

I know we're talking about POSIX here, but Linux does have the ability to wait on signals the same way it waits on file descriptors. It does this by also allowing signals to be delivered on file descriptors.

http://man7.org/linux/man-pages/man2/signalfd.2.html

JdeBP3• 10 years ago

Winding forward to the present: I recommend a look at kevent/kqueue.

lelandbatey• 10 years ago

Toward the very end, the author gives a brief one paragraph explanation of why they didn't use multiple threads:

    Obviously one idea is to use threads. It’s common to have a separate thread
    for dealing with signals and combining threads with semaphores, etc. would
    allow GNU make to wait on two different types of asynchronous events
    simultaneously. This was rejected because so far GNU make was single-threaded,
    and requiring multi-threading was deemed too much of a price to pay.

I'm confused as to why adding multi-threading would be quite so onerous. As I understand it, one could restrict the use of threads to that specific feature and thus avoid large architectural changes.

Do I not understand how Make is designed, or is "the price to pay" in reference to performance or some other metric?

0xcde4c3db• 10 years ago

It might be a portability concern. GNU make runs on a lot of platforms.

Samathy• 10 years ago

Surely the solution there would be to make it a compile-time option. Enable threaded jobs, or process jobs.

nullc• 10 years ago

It may be that the bulk of the codebase was not concurrency safe.

boris• 10 years ago

GNU make is a giant string-chopping hairball written in C over the past 30 years by numerous authors. Paul D. Smith (the current maintainer and the author of this article) has actually been doing a lot of codebase cleanup over the past years. But still any major change is a monumental task.

hyc_symas5• 10 years ago

As I noted in my own mention http://highlandsun.com/hyc/#Make I originally wrote that patch back in 1991. There was no such thing as threads back then. Even in 1999 the POSIX threads spec was only up to Draft 6 or so, not yet finalized. Nobody sane would write production code for such a moving target (oh, except maybe Microsoft, using Draft4 to create the Windows NT threads API).

JdeBP3• 10 years ago

That parenthetical history is wrong, at least according to Dave Cutler. He traced Windows NT's threading back to his work on VAXELN in 1983, years before POSIX was even thought of. He also was quite obviously deriving the NT API in part from OS/2, again not POSIX, given the whole "mutant"/"mutex" affair.

some1else• 10 years ago

We need to use Make for sensible (partial) rebuilds of static websites. I'm exploring ways to build a dependency graph out of Handlebars files. That might be enough to generate a Makefile and leave Gulp/Grunt out of the equation. Anyone doing anything similar?

saosebastiao• 10 years ago

Not for static websites, but I've definitely done very similar with Knitr for repeatable analyses in R (except for autogenerating the makefile). I create R scripts, markdown templates, database extract jobs, and even take care of sending the analysis results via email. I've left the job where I was responsible for those reports, but they're still in use with a distribution list that goes up to the SVP level at Amazon, so apparently they've been pretty successful.

jahitr• 10 years ago

GNU make is one of the most complete task manager I have seem.

But there are a few major problems with it:

Newies often complain a lot about the esoteric syntax.

It's written in C.

It doesn't gets updated as fast as it should.

These points make it not very suitable for our current time.

I belive we all just need to pick something like gulp and port the "good parts" into it.

There is no point in having 20 tools that do the same!

hyc_symas5• 10 years ago

"written in C" is an asset, not a liability. It is still the one language that is portable across the widest spectrum of platforms, as well as the most efficient high level language for execution.

GNU Make Jobserver Implementation

Related Stories