GNU Parallel 2018

by Tomte • 7 years ago

150 68 7 years ago

tbrock4• 7 years ago

It’s a shame that this tool is so incredibly useful but simultaneously so unergonomic to use that it requires a 12 page manual.

Every time I reach for it the magical incantations that finally get it working properly are so completely unmemorable that it’s begging to have a modern replacement (like httpie is to curl).

If someone wrote a version with a better interface it would be instantly adopted in preference to this. Might be a good weekend project.

feanaro• 7 years ago

I agree with the sentiment, but please, Rust, not Go. In fact, I remember seeing such a project, let me search...

Ah, here it is: https://gitlab.redox-os.org/redox-os/parallel

tbrock4• 7 years ago

Ok, whichever language.

krapht• 7 years ago

What? Why? Seriously getting fed up with the Rust lovers in Hacker News. It makes me hate Rust even though I actually really like it as a modern replacement for C++. Memory safety is only one aspect of proving program correctness. It's also less of an issue for garbage collected languages, and tracing GCs are fairly performant.

You don't see Coq users crapping on other languages for not proving their programs correct according to a formally defined specification.

sigmonsays• 7 years ago

lets not start a language war here

hi41• 7 years ago

Could you please elaborate on why Rust would be better than Go? Doesn’t Go offer good support for parallelism, which would be great use for implementing GNU Parallel.

kylek3• 7 years ago

Maybe not any better syntax, but xargs is usually installed by default in most distros, and I've never found parallel to be advantageous over it. I don't think this particular wheel needs to be invented again.

tbrock4• 7 years ago

xargs is great but doesn’t run the commands in parallel or chunk input does it?

ravoori• 7 years ago

xargs does nothing to prevent interleaving of output from simultaneously executed commands.

fmajid3• 7 years ago

I dislike parallel’s confusing options, but xargs does not report on progress.

lelf• 7 years ago

xargs is usually installed by default

xargs is part of POSIX, so it should be everywhere.

(-P/-n are not, but both GNU and BSD versions do support it.)

benbenolson• 7 years ago

Why does the language that it's written in matter? For most users, it's a compiled binary sitting on your system.

EDIT: Parent comment edited his post, leaving out his personal preference of language.

fmajid3• 7 years ago

Parallel is written in Perl, not a compiled binary. That may be what he was referring to. I also wish Perl (and ruby, and Java) would go away.

jarfil• 7 years ago

The biggest problem with parallel is not the implementation but the command line options specification. The shell has no generic way of defining multiple streams feeding multiple placeholders, so parallel ends up using its own syntax which has to be different in order not to interfere with the base shell syntax. If the same functionality was integrated directly into the shell, it could be less complicated and more generic.

tbrock4• 7 years ago

Exactly. The implementation is fine, it’s the interface that isn’t great.

faitswulff• 7 years ago

..a 12 page manual.

My first thought was, "12 pages isn't so bad." Then I loaded the page and realized you had a typo and meant _112_ page manual.

That is enough to scare me off of trying to use parallel for casual tasks.

LeoPanthera2• 7 years ago

I routinely use it for casual tasks and I've never read the manual. Don't confuse complexity with difficulty.

gnulinux2• 7 years ago

That is enough to scare me off of trying to use parallel for casual tasks.

Just because emacs is such a complex software, does that mean no one can start learning it? No. As far as GNU parallel goes simply just understand what `:::` operator does and you can start using it in your workflow today. Software is not complex for the sake of beginners, it is complex so experts have a higher variety of features.

mechanical_jane11• 7 years ago

Maybe the cheat sheet is made for you: https://www.gnu.org/software/parallel/parallel_cheat.pdf

yesco• 7 years ago

Well I had never heard of httpie before now so thanks for casually dropping that endorsement. Never realized how much I wanted something like this.

dahart3• 7 years ago

It’s a shame that this tool is so incredibly useful but simultaneously so unergonomic to use that it requires a 12 page manual.

It doesn’t require reading the whole manual, nor does the length of the manual have anything to do with parallel’s ergonomics.

Do you use bash or ssh? Have you looked at the length of those manuals? Most of parallel’s manual is advanced workflows the casual user doesn’t need, stuff like parallel commands run over a network of machines and collecting the results on the originating host.

Maybe you could show some examples of what you mean, because I think parallel is pretty easy to use. Personally I feel like it’s usually easier than Xargs, especially when dealing with tricky quoting. It does have some of it’s own syntax, like the triple-colon :::, and it’s really easy to use. Several pages of manual are about variable substitutions you can use if you want, they’re very similar to bash variable substitutions, and they’re all optional and super convenient. A bunch of parallel’s manual pages are devoted to simple examples, which is nice, and the kind of thing that’s missing from so many man pages.

it’s begging to have a modern replacement

I’m not sure what “modern” means. You could certainly propose some changes though, or show some examples of the kind of syntax that isn’t working for you. Httpie and curl are built to do different things.

Mizza2• 7 years ago

If you're interested, I've written a tool called `loop` which sacrifices some of the abilities of GNU Parallel for a far more intuitive interface (I think so, anyway.)

You can check it out here:

https://github.com/Miserlou/Loop

dahart3• 7 years ago

sacrifices some of the abilities of GNU Parallel

Like... the ability to run things in parallel?

Loop looks nice, don’t get me wrong. I think I’ll try it! It just seems like comparing it to Parallel is misplaced here, and a bad idea in general, since they’re not remotely related. Ole is comparing all these utilities to Parallel to show how general Parallel can be, and suggest you only need his one program and one syntax to emulate hundreds of different utilities: his argument is implicitly that people should use Parallel and not loop.

mechanical_jane11• 7 years ago

Comparison to GNU Parallel's syntax: https://www.gnu.org/software/parallel/parallel_alternatives....

degraafc• 7 years ago

Looks more like a souped up `watch` [1] than `parallel` to me. Nice tool!

[1] https://en.wikipedia.org/wiki/Watch_(Unix)

PeCaN• 7 years ago

it’s begging to have a modern replacement (like httpie is to curl).

This seems like a pretty weird example. All common tasks in curl are very straightforward and I've never even heard of httpie.

Let's please not change the interface to basic command line tools every few years because people want something ""modern"".

brians• 7 years ago

Common tasks for the web are easy in curl—though it will take me a long time to forget environment variables that could be set to false, true, or two. Httpie makes web apis easier, with better support for api keys and for hashing some headers into others, holding state across several command invocations, that sort of thing.

RubberSoul• 7 years ago

There are great videos for understanding how it works [0]. I watched a couple of those and after that was able to do exactly what I needed.

For me the trickiest thing is remembering how to deal with redirection. I find the --dry-run option to be super useful when setting up a command.

[0] https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

mechanical_jane11• 7 years ago

Yeah, quoting redirection is a bitch, but as long as you know GNU Parallel is just a program and not part of the shell, you can usually deduct which parts will be interpreted by the shell before GNU Parallel gets it.

I often create bash functions to avoid that shit, and then call the bash functions from GNU Parallel instead.

msla• 7 years ago

Realistically, the best language for this is C, because everything else is too heavyweight, requires too much of a runtime, or simply isn't available most places.

mechanical_jane11• 7 years ago

Does that also apply to platforms where you as a consultant for hire does not have access to a C compiler for that given platform?

It used to be an issue that non-free UNIXes did not include a C-compiler by default, and then C would really not help. I am not sure if that is the case anymore.

gioele• 7 years ago

For those afraid of the complexity of GNU Parallel, there is the tiny moreutils `parallel` (or Tollef's parallel) that covers 90% of the common use cases with a single-page manual.

https://manpages.debian.org/testing/moreutils/parallel.1.en....

mechanical_jane11• 7 years ago

But looking at the comparison, do you really find GNU Parallel that much harder to understand?

  parallel sh -c "echo hi; sleep 2; echo bye" -- 1 2 3

    parallel echo hi";" sleep 2";" echo bye ::: 1 2 3

  parallel -j 3 ufraw -o processed -- *.NEF

    parallel -j 3 ufraw -o processed ::: *.NEF

  parallel -j 3 -- ls df "echo hi"

    parallel -j 3 ::: ls df "echo hi"

And if you are looking for the single page overview, why not link to the cheat sheet: https://www.gnu.org/software/parallel/parallel_cheat.pdf

mechanical_jane11• 7 years ago

I think the cheat sheet is made for you: https://www.gnu.org/software/parallel/parallel_cheat.pdf

If you have a hard time getting it to work and remember how, then it is likely that the fundamentals of GNU Parallel are not clear to you. Chapter 2 ("Learn GNU Parallel in 15 minutes") of the book should take care of that.

a better interface

That is often a tough one. I, for one, like emacs' interface. Others despise it. So when you write 'better interface' I think it will be helpful if you were more specific.

0xFFFE2• 7 years ago

This is such a nice tool. Here is handy one liner I use it all the time.

parallel -a input_file_here -j16 --gnu 'ssh -q {} "echo 'Matrix has you'"'

mechanical_jane11• 7 years ago

Why not save some quoting:

  parallel --slf input_file_here -j16 --nonall echo 'Matrix has you'

0xFFFE2• 7 years ago

Sweet, this is more elegant, thank you.

LeoPanthera2• 7 years ago

Any discussion about GNU Parallel inevitably degrades into complaints about the requirements for "--will-cite".

Removing that requirement is a two line patch:

    4723,4724d4722
    <        1
    <        or

dahart3• 7 years ago

Or, instead of a two line patch with codes, you could just type one line: “parallel --will-cite”

saagarjha• 7 years ago

There are not citation requirements, as per the GPL. The text that the tool prints without --will-cite is merely a strongly-worded request.

gnulinux2• 7 years ago

There is no such requirement. If you read the text you'll see that author explains that's their only income source, so they ask you to consider citing them in your paper. It is an entirely reasonable thing for a software engineer to ask from their users.

bdowling• 7 years ago

It is an entirely reasonable thing for a software engineer to ask from their users.

It's also a reasonable thing for a user of free software to modify the software and improve it by removing this kind of garbage.

mechanical_jane11• 7 years ago

Why not simply do as GNU Parallel suggests in the message:

Run 'parallel --citation' ONCE and have the notice silenced.

I never understood the complaint: Is it because people do not _read_ the message?

edoceo• 7 years ago

For everyone that don't like this one, they publish a big list of alternative

https://www.gnu.org/software/parallel/parallel_alternatives....

edibleEnergy• 7 years ago

Dang that's some list of options :D http://git.savannah.gnu.org/cgit/parallel.git/tree/src/paral...

matt-attack• 7 years ago

I'm actually a huge fan on parallel. Use it often for various media processing pipelines. Um, you don't need to be daunted by the 112 page manual. A brief read of the man page has always sufficed. The best part is that every time I think of something new I need to do, it's always there.

nullc• 7 years ago

I like some of parallels' functionality but I wish it was more performant and that it was easier to avoid the complexity when you only want a few features.

As a result of parallel's performance and invocation complexity I find myself using xargs -P more frequently than parallel.

AlexDragusin2• 7 years ago

Immensely useful which I am forever grateful that it exists. I've made use of it thoroughly during a search engine indexer test I ran back in 2013, which allowed me to pretty much download and index (download + indexing in one go) an entire country (Romania) pages in less than 1 hour on a laptop with only 4GB ram and i7 Ivy Bridge processor (Zenbook UX21 a particularly).

All the components I've wrote are optimized to the core (short of going assembly). It allowed me to scale my operations up to 10000 processes continuously for the duration of the test. All this was done on a 4G connection with roughly 40Mbps bandwidth at the time on a virtualized (VirtualBox) CentOS LAMP installation.

It blew all my expectations, without parallel it would taken more than a month. Learned a lot from the experience developing it and maybe will make a post sharing what I've learned.

In regards to being complicated, well it isn't really when you consider its use scenarios, for these sorts of tasks, it's a indispensable tool.

Think of it this way: It is far less difficult than the difficulty of the tasks where you need to use it.

mkl• 7 years ago

How did you get the list of pages to download? Or was this spidering from some starting set?

One hour on that connection means a maximum of 18GB of data, which seems surprisingly small for a whole country's internet (not that I have any point of comparison), small enough that you could easily do interesting analysis of the whole lot.

AlexDragusin2• 7 years ago

I've started off a seed list (compiled from both Alexa and Romanian Wikipedia) and from there on spidering through links found on each of the pages. I needed only the pages written in Romanian language so right after download, a language checking would take place along with the ranking and finally indexing. Please note that only the HTML was downloaded, no images/css/js etc so the size would be considerably less than if you would normally browse the pages. The HTML/CSS/JS would be sripped off, leaving the content for analysis, ranking and indexing.

True and after that I realized how small the Romanian internet really is in the big picture, surely allowed me to put things into perspective. It's small enough that it would be doable to have a near realtime update of the index since less than 10% of sites update daily and less than 1% on hourly basis such as news sites.

It correlates with traffic details of top trafficked webs in Romania where one of the top sites gets around 200k uniques a day, a small amount when you consider the big picture. It is likely that today the amount of data would be larger but not by much.

GNU Parallel 2018

Related Stories