Parallel processing with Unix tools

http://www.pixelbeat.org/docs/unix-parallel-tools.html

by kiyanwang • 9 years ago

264 87 9 years ago

enriquto5• 9 years ago

do not forget about "make", that will traverse a user-supplied tree of filenames in parallel

pletnes• 9 years ago

Or any job in general. Also, make 4.x comes with the handy -O flag which makes the output seem "serial" instead of intermixed between processes.

dahart13• 9 years ago

And make -j will execute the rules & build targets in parallel. Once I built a general batch parallel system using make that could pause & resume parallel jobs by using each job's log file as the make target. Then I discovered gnu parallel and scrapped my project.

gnufx• 9 years ago

Dependencies in batch resource managers for distributed parallel jobs (or simpler ones) are standard practice. https://arc.liv.ac.uk/SGE/htmlman/htmlman1/qmake.html is an old adaptation of GNU Make which works within a job.

SamJson8• 9 years ago

Fun fact from https://www.gnu.org/software/parallel/history.html

[GNU Parallel] was originally a wrapper that generated a makefile and used make -j to do the parallelization.

anitil• 9 years ago

Gah story of my life - hours/days of grueling work trying to solve a problem, then someone say 'why don't you just use $tool_that_does_exactly_what_you_want_but_better?'. Code goes in the bin.

pixelbeat• 9 years ago

Very good point. I added a whole section to the article, implementing the counting lines example with `make -j`, which performs just as well as `xargs -P`

mtreis86• 9 years ago

Qt multithreading http://doc.qt.io/qt-5/threads-technologies.html

Aria for downloads https://aria2.github.io/

Pigz and pbzip2 for compression https://zlib.net/pigz/ http://compression.ca/pbzip2/

gjvc• 9 years ago

I have known situations where the compressed output of pbzip2 was not readable by some .NET library a customer was using (I'm afraid I can't remember which one). Fortunately an alternative multithreaded bzip2 implementation, http://lbzip2.org/, did not suffer from this problem so just in case...

vthriller3• 9 years ago

On multiple occasions I found lbzip2[0] to be faster than pbzip2

[0] http://lbzip2.org

Edit: here's a quick sample (file is read from tmpfs on a 4-core i7-5500U):

  $ time lbzip2 < linux-4.13.3.tar > /dev/null
  real    0m17.410s
  user    1m7.799s
  sys     0m0.283s
  $ time pbzip2 < linux-4.13.3.tar > /dev/null
  real    0m30.556s
  user    1m57.557s
  sys     0m2.169s

vthriller3• 9 years ago

Also, with modern enough xz-utils you can just use `xz -T` for parallel compression.

dahart13• 9 years ago

If you're zipping multiple files, is it better to pbzip2 each file, or to parallel bzip2 them? You wouldn't want to parallel pbzip2, would you?

vthriller3• 9 years ago

By compressing multiple (large) files sequentially you'd be able to gradually free some disk space much sooner, and less free space will be needed to write compressed data to.

alvil• 9 years ago

I like it using newLISP :)

  ;calculate primes in a range
  (define (primes from to)
    (local (plist)
        (for (i from to)
            (if (= 1 (length (factor i)))
                (push i plist -1)))
        plist))

  (set 'start (time-of-day))

  ; start child processes
  (spawn 'p1 (primes 1 1000000))
  (spawn 'p2 (primes 1000001 2000000))
  (spawn 'p3 (primes 2000001 3000000))
  (spawn 'p4 (primes 3000001 4000000))

  ; wait for a maximum of 60 seconds for all tasks to finish
  ; returns true if all finished in time
  (sync 60000) 

  ; p1, p2, p3 and p4 now each contain a lists of primes
  (println "time spawn: " (- (time-of-day) start))
  (println "time simple: " (time  (primes 1 4000000)))

  (exit)

unixhero• 9 years ago

One day I have to learn how to use xargs

limaoscarjuliet2• 9 years ago

You and me, both of us. I once combined rpm -e with xargs rpm -ql to the tune of getting all packages uninstalled from the system in one swift command.

It was a test system so all good. But I'm still not sure what I was thinking.

pwg• 9 years ago

The simplest use case for xargs is that you have a list of filenames and you want to feed that list to a command that only accepts filenames as CLI parameters:

I.e., 'find' generates a list of filenames.

But 'stat' (for example) only accepts file names as parameters on the command line.

You combine these two using 'xargs':

find -type f -print0 | xargs -0 stat --format="%y\t%n"

And you get an output of human readable modification times, a tab, the filename and a newline.

Note, the "-print0" to find instructs find to output null terminated filenames, and the -0 to xargs instructs xargs to expect to receive null terminated filenames. These switches (they may be specific to GNU coreutils) allow for handling filenames with any possible character in them correctly (i.e., spaces cause no troubles).

For this use case, remembering how/when to use xargs is easy. The more esoteric usages are the ones where "man xargs" is often handy.

cat1993• 9 years ago

It's pretty much 'apply' for shell arguments

https://en.wikipedia.org/wiki/Apply

use it many places instead of a for loop:

for x in ; do <cmd> $x done

is:

echo |xargs cmd

Assuming 'cmd' can take multiple arguments.

If not, use 'xargs -L 1 cmd' to run cmd 1x per arg.

GNU xargs has -1, but -L 1 is portable to xargs in the BSDs and others.. plus it's good to know -L in case you ever need to -L 2, etc.

using echo as a driver is not the best example, but anyhow.

cat1993• 9 years ago

also of note is:

-t : print the commands being run

-I %: interpolate file name in command string.

e.g. ls .mp4 |xargs -t -L 1 -I % scp % me@somebox:/movies

to scp individual files (replaced in %) to a destination, one at a time, printing each command as it is run.

Again, not the best example since you can

scp .mp4 me@sombeox:/movies

or rsync, etc. but you get the idea.

wyoh• 9 years ago

Rust Parallel is nice too: https://github.com/mmstick/parallel

agumonkey• 9 years ago

Performance of the actual program might be less relevant but it's nice to see it using 1% for itself compared to GNU parallel.

SamJson8• 9 years ago

The performance is apparently close to xargs (if speed is key, why not just use xargs?). Rust parallel does, however, have some issues that would rule it out for me:

https://www.gnu.org/software/parallel/parallel_alternatives....

tejtm• 9 years ago

and the humble pipe, another process may start the next stage before the previous one(s) have finished.

limaoscarjuliet2• 9 years ago

Consumer task will wait on pipe read and the producer will wait on pipe open, so it is synchronized properly.

grigjd3• 9 years ago

Need to be aware of the buffer size on the pipe. It can be one of those issues that never comes up until you just cross the threshold, then everything fails ungracefully.

termie• 9 years ago

Bash’s built-in wait is also handy when you want quick and simple parallelism. http://tldp.org/LDP/abs/html/x9644.html

philsnow• 9 years ago

I do this pattern a lot

    for x in $(seq 1 10); do long_process file${x} & done; wait

It doesn't take care of not saturating my cpu or anything; when I need to care about that then I try to remember how to use parallel.

enriquto5• 9 years ago

it's actually easier using parallel :

        for x in ...; do echo long_process file$x ; done | parallel -j 8

EDIT: and in your case, it is even easier with xargs, e.g.:

        ls files* | xargs -n 1 -P 8 long_process

SamJson8• 9 years ago

I would do:

    seq 10 | parallel long_process file{}

Or simply:

    parallel long_process ::: file*

gigatexal• 9 years ago

Command line goals.

Filligree• 9 years ago

For GNU Parallel, if you pass it -k it will buffer all of its subcommands' output and print it as if it had all been run serially, removing the need for those subcommands to output anything atomically -- a requirement that usually can't be guaranteed.

It's a capable tool, if somewhat... complicated.

protomikron• 9 years ago

Thanks, I did not know that.

This is useful if you are solving embarrassingly parallel problems that generate lots of log data (e.g. timings for benchmark) and write them to stdout/stderr.

AFAIK there is some guaranteed atomicity when writing PIPE_BUF<512 to a pipe, however then your sub programs shouldn't write more than 512 bytes of log-data at any point, I would guess.

dahart13• 9 years ago

parallel --will-cite

I love gnu parallel, but that citation thing is a bit of a drag on parallel. You only have to do this once, but I see why you'd write a blog post to include the flag and not have to discuss the issue.

I wish Ole would relax on this. It's not really that appropriate to demand that anyone using parallel for academics cite a magazine article. It's only appropriate for people doing research in parallel algorithms, and in any case it would make citing easier if there was a journal article to reference.

There's nothing wrong with asking nicely, and I try to spread the word about gnu parallel. But the heavy handed demand is annoying, even if I only deal with it like once a year.

bamboozled• 9 years ago

Strongly agree with this!

gioele• 9 years ago

For 99% of the use cases moreutils' parallel (AKA Tollef parallel) is an equal, or even better, alternative.

"Interesting" thread from 2012 about replacing moreutils' parallel with GNU parallel in Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=597050 (TLDR: skip to message #75).

dahart13• 9 years ago

Oh, I've seen the --tollef option in the help doc, but I still didn't realize there was a complete alternative to gnu parallel. Definitely will have to try.

Mr. Hess makes a lot of good points, most I think I agree with, so I can sympathize with his frustration.

The point about transferring files seems interesting. I haven't used gnu parallel in remote/ssh mode aside from simple tests, but that seems wicked useful for some situations. Is there a real and simple alternative to running gnu parallel on multiple remote hosts? Is that something the moreutils version can also do?

sillysaurus32• 9 years ago

If the author wants to extract ten seconds of your time once a year, that's just the price of admission. They're not uptight; they're trying to advance their career.

(Info: https://lists.gnu.org/archive/html/parallel/2013-11/msg00006...)

This is a very important problem. We devs love to make tools, and we love to do it for free. But you can't eat idealism. Parallel is a nice tool, but it's probably not as impressive as it seems when it comes to landing a job.

You're free to decry the author and to point out alternatives. It's practically tradition. But I identify with where the author is coming from. How would you feel if you spent a bunch of time on a tool and the top comment blasts it for reasons unrelated to its merits?

dahart13• 9 years ago

How would you feel if you spent a bunch of time on a tool and the top comment blasts it for reasons unrelated to its merits?

I love parallel, and I spread the word and participate in the PR that Ole is asking for.

If the author wants to extract ten seconds of your time once a year, that's just the price of admission.

The ten seconds is not the problem. The problems are:

- The citation nag notice is confusing people on their legal responsibilities. It has prevented people from using it due to the fear of liability it causes. (RE: "If you pay 10000 EUR you should feel free to use GNU Parallel without citing." ... "If you use '--will-cite' in scripts you are expected to pay the 10000 EUR, because you are making it harder to see the citation notice.")

- The citation demand goes against GNU policy. https://www.gnu.org/licenses/gpl-faq.en.html#RequireCitation

- Ole's citation is not from a scientific or peer-reviewed publication. It simply cannot be used in some contexts. In many contexts, a citation of Parallel is highly inappropriate.

- The small amount of time it takes to either cite Ole or read and understand this thread is largely irrelevant to the question of whether a citation is either warranted or appropriate or allowed (by the actual GNU license, by a user's employer, and/or by the publication they're submitting to). Understanding the license is a one-time event, and it's very important for the license to be legally clear. Parallel's citation notice is causing license confusion. The question is whether Parallel can and should be used without having to worry (forever) about the legal consequences of the contract I've agreed to by using the software.

- This approach isn't scalable. If other GNU tools, or other free software started using the same language as Ole, it would cause widespread problems. Mr. Hess is correct, this is very antithetical to Unix.

KGIII3• 9 years ago

I use Linux but I've never used parallels. At first, I thought you were making that up.

But, I looked at your username and realized I have seen you post before - and I couldn't imagine you being dishonest about this. You just don't seem like the type to make this sort of stuff up.

So, curious, I headed to Google.

Sure enough, and for those who don't know, the author wants you to cite parallels when you use it in conjunction with something that is going to be published. Now, to be fair, I've always included citation of the tools involved when I published. However, nobody has ever asked me to directly and nobody has nagged me to do so.

If anyone is unfamiliar:

https://www.gnu.org/software/parallel/

This appears to be the original thread about it:

https://lists.gnu.org/archive/html/parallel/2013-11/msg00006...

It's a bit taboo, I guess. You can pay money and not have to cite it. You can also feed it the --will-cite argument and it won't bug you. Still, it is in probably in poor taste. If nothing else, it is unconventional.

I didn't find a whole lot of complaints about it. The only hyperbolic outrage was, interestingly enough, someone on HN who was quite uptight about it.

I guess Ole is within his rights to do that, but it is certainly unconventional. I wonder if anyone takes his inflated number of citations as a negative against him?

From what I've now read, there don't seem to be a whole lot of complaints. Even the mailing list thread was short and didn't contain any drama. The only drama I found was the post on HN.

dahart13• 9 years ago

I have discussed this before, I really hope the hyperbolic outrage wasn't me though! ;) I like parallel, and this issue is relatively minor. But I admit the language of the nag bothers me just a smidge.

You can pay money and not have to cite it.

You can also use it without paying. The nag is neither a license nor contract, and the proposed money isn't a required fee. The language of the notice makes it sound like you have to pay, and makes it sound like part of the license.

I guess Ole is within his rights

I agree completely. And I'm in favor of him getting what he wants too! He's right that PR is helpful for his cause, and I appreciate his cause. I just wish he'd ask without the nag, and use softer language.

Using parallel isn't a reason to cite it, any more than using LaTeX or emacs would be a reason to cite LaTeX or emacs. And if gnu parallel counts as previous research, it's very unlikely he has to ask.

I didn't find a whole lot of complaints about it.

There are some, and I'd agree it's not a huge issue, by and large. The main discussions I've seen have been with package maintainers. It is a small issue, and the few times I have brought it up, I've had a ton of confirmation that it bothers other people a similarly: a little bit.

vermaden• 9 years ago

Just put this line into ~/.xinitrc or ~/.zshrc or other file that is executed BEFORE you start CLI:

echo 'will cite' | parallel --citation 1> /dev/null 2> /dev/null &

tokenizerrr• 9 years ago

That slows down the startup of my terminal, just to appease the developer of some tool I probably won't even use that session. No thanks.

The nagging notice should just be removed. It's not like I'm ever going to publish a paper anyway.

LeoPanthera• 9 years ago

It's GPL, right? Why hasn't anyone forked it with the citation requirement removed?

dahart13• 9 years ago

Good Q! I don't know, but I also feel like that could be a bit dirty, without having other reasons to fork. I might not want to encourage or support that.

There are some (non-fork) projects that provide the same functionality, with the stated motivation being in part because of the citation thing.

SamJson8• 9 years ago

You would then not be allowed to call it GNU Parallel due to possible trademark confusion. This also why we have names like CentOS (not RedHat Free) and IceCat (not Firefox Free).

There was a court case in Germany about this (for some CMS tool - IIRC) where a forker used a similar name. The verdict was pretty clear: Forking was OK (copyright law - permission by GPL), but keeping the name was not (trademark law - no permission by GPL).

jwilk4• 9 years ago

OK, that has killed all my desire to ever use this software.

Most users won't ever publish anything. Why alienate them?

dahart13• 9 years ago

I would totally recommend giving it a try anyway, I don't mean to turn anyone away. Gnu parallel is really nice. I use it a ton for things like batch image resizing. I wish the citation request would take another form instead, but it really is minor.

SamJson8• 9 years ago

Author explains background:

https://www.gnu.org/software/parallel/parallel_design.html#C...

wookayin• 9 years ago

This tool is also great: https://github.com/greymd/tmux-xpanes

npx• 9 years ago

I realize it's somewhat off topic, but I feel like Joyent Manta deserves an honorable mention. It's an S3 style object store, but you can spin up containers on top of objects and do massively parallel computations with Unix tools.

https://apidocs.joyent.com/manta/

zackmorris• 9 years ago

My biggest gripe with UNIX is that a command like:

(sleep 5 && echo hello) > test.txt

Causes a race condition where test.txt is created empty, then 5 seconds later written with "hello". Try it without parentheses to see it wait. UNIX doesn't use our intuitive notion of order of operations on its piping, it just runs everything simultaneously. This allowed for tremendous efficiency and concurrency but it's hard to fathom how much this has cost us in bugs and lost development time.

I feel like this was a lost opportunity because it prevented the Actor model (as seen in Erlang and Go) from really taking off decades ago. Perhaps this bug/feature was one of the motivations for commands like "parallel".

Does anyone have a general workaround for this problem? Some command that we could insert in the chain to force a wait, without having to install any external tools? Thanx!

Edit: I'm having a hard time explaining how insidious this race condition is for people who haven't encountered it yet. The gist of it is that file descriptors aren't opened when a pipe sends its first byte, they are opened when the shell command is interpreted. I'm also having a hard time finding examples, here is one I think, though there are many, many others: https://unix.stackexchange.com/questions/174788/am-i-hitting...

adamkruszewski• 9 years ago

I think it does, try it as:

sleep 5 && (echo hello > test.txt)

and you'll see it creates the file only after sleep ends.

chubot• 9 years ago

I don't understand what you're trying to say. I would expect these two commands to behave differently:

    (sleep 5 && echo hello) > test.txt
    sleep 5 && echo hello > test.txt

"our intuitive notion of order of operations on its piping" -- I don't know what your intuitive notion is, and there is no piping in your example. Piping would be:

    sleep 5 | echo hello  (this is an odd example, but it works)

I also don't see any race condition in your example. A race would be two processes accessing the same data.

They are redirects, and the way redirects like > and < work is to:

1. open a file

2. change the global process state

3. run arbitary commands, either builtin (echo) or external (ls). External commands require a new process and inherit their process state (descriptor table, where stdout and stdin are conected to) from the parent shell.

4. restore the global process state

5. close the file

If you understand that, it might be clear why the file test.txt is opened before sleep. There's really no other way it can work, given the fact that you can execute arbitrary commands inside the redirect.

I address some related things about redirects here: "Avoid Directly Manipulating File Descriptors in Shell"

http://www.oilshell.org/blog/2017/08/12.html

-----

The more unintuitive thing about shell is that what you wrote is technically better as:

    { sleep 5 && echo hello; } > test.txt

() doesn't mean "grouping" like in other languages, it means subshell. {} means grouping, and it has an odd way of parsing, where ; is required before }, but it isn't required before ). This is because () are operators and {} aren't.

I wrote about this here, "the subshell construct () is confused with the grouping construct { }":

http://www.oilshell.org/blog/2017/02/06.html

-----

EDIT: I will say that I was very confused by the syntax of > and < when I started learning shell. It does seem like ">" means "this goes into that". In fact I think I internalized that from MS-DOS.

But that doesn't explain what these mean, and why they are equivalent:

    tac >out.txt <in.txt
    tac <in.txt >out.txt

They are best read as prefix operators that modify the state of the process invocation.

What's even more confusing is that you can do:

    tac 1>out.txt 0<in.txt
    tac 0<in.txt 1>out.txt

Then they look like binary operators again. But the descriptor is "stuck on", it's actually part of the operator lexically.

Valid:

    tac 1>out.txt 0< in.txt
    tac 1> out.txt 0< in.txt

Invalid:

    tac 1 >out.txt 0 <in.txt
    tac 1 > out.txt 0 < in.txt

In other words, there can't be any space between the LHS file descriptor and the > or <.

The syntax of shell is horrible, and that's something I would like to fix with Oil shell. But I don't see the problem you are pointing out.

debacle• 9 years ago

(sleep 5 && echo hello) | > test.txt?

SamJson8• 9 years ago

Workaround:

(rm file; tac > file) < file

jwilk4• 9 years ago

Note that, somewhat counter-intuitively, what "wc -l" counts is the number of newline characters:

  $ printf 'foo\nbar' | wc -l
  1

There are arguably two lines in the input, but the result is 1.

Without this (mis)feature, "wc -l" would be more difficult to parallelize.

LambdaComplex• 9 years ago

The POSIX definition of a "line" is "A sequence of zero or more non- <newline> characters plus a terminating <newline> character"[0]. I don't think it's counterintuitive at all for POSIX utilities to respect that definition.

0. http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_...

frou_dh• 9 years ago

"terminator, not separator" is an easy way to remember it. I would call bar in GP's comment a fragment.

jandrese• 9 years ago

The only difference is you have to check the last character in the file to determine if it is a newline. Maybe the last 4 characters if it is Unicode.

catern• 9 years ago

known• 9 years ago

OleTange, author of parallel chips in https://www.reddit.com/r/programming/comments/5x39jh/countin...

YSFEJ4SWJUVU6• 9 years ago

The mentioned tool 'turbo-linecount' is, well, odd. It targets what I'd think is a very niche application of counting lines very fast (a domain usually limited by I/O speed), using a rather complex design... and then manages to throw much of the gained advantage away by using what is perhaps one of the slowest ways of counting newlines in a buffer.

Parallel processing with Unix tools

Related Stories