A small primer on xargs

by bitops • 14 years ago

48 19 14 years ago

DHowett• 14 years ago

"In particular, the presence of \0 in your data will throw xargs for a loop (no pun intended). In this case, it is necessary to use the -0 option. This is particularly relevant if you are using the find command and supplying the -print0 option."

This glosses over the fact that those two things are intended to be used together. Delimiting filenames with a \0 is superior to delimiting them with spaces or newlines, as filenames cannot usually contain ASCII NUL; xargs and find can therefore be used to remove heterogeneous lists of files.

Additionally, the example given may better be worded as "find . -name '* junk* -exec rm '{}' \;" or even "find . -name '* junk* | xargs rm". (as given it strikes me as a useless use of grep and is additionally fragile if ls is aliased to "ls --color=always", spewing ANSI into

EDIT: Please excuse the additional spaces in the find command arguments, they are a formatting avoidance feature.

kjetil• 14 years ago

Little known fact: the -0/-print0 idea is due to DJB.

darklajid• 14 years ago

If we're ripping this sample apart, I'd like to point out that

  rm *junk*

in this case would've done the same, without spawning lots of useless stuff.

xargs is cool. The example fails to show a decent use case. For me a line like that would indicate that someone is _not_ familiar with his tools. Similar to

  cat somefile | grep something

instead of

  grep something somefile

Pipes and processes aren't endangered or (in practice) limited, but we still don't need to be wasteful?

babarock2• 14 years ago

Furthermore, parsing the output of `ls` is a terrible practice[1] that makes for an even poorer sample.

A little further in the article, the author even suggests running

    `ls | grep junk`

(instead of the much simpler `ls junk*`. Do I need to point that out?).

I also know some people are going to retort that wasting a process or a pipe is not that bad. This discussion has been had over and over again, probably ever since Unix was invented.

In any case, the poor example chosen, as well as the flaky commands used, seriously hinder the credibility of the article.

[1] http://mywiki.wooledge.org/ParsingLs

lubomir• 14 years ago

Instead of -exec rm {} \; you could use -delete option as well.

babarock2• 14 years ago

This option is not portable at all. For instance, it won't work on Solaris' find. I strongly advise against the use of non portable options.

asdfaoeu• 14 years ago

If you still want to use xargs and ls perhaps avoiding \0's (perhaps so you can easily grep on the data) you can use "ls -Q | xargs rm". The advantage of that is you can still easily use grep on the data as it is still newline delimited but you won't have problems with spaces.

Also ls is never aliased to "ls --color=always" unless you've done that more likely is its aliased to "ls --color=auto" or similar which will work normally.

robertskmiles• 14 years ago

> more likely is its aliased to "ls --color=auto" or similar which will work normally.

My ls (which I haven't much reconfigured) behaves very differently depending on whether it's outputting to a shell or a pipe or redirect. Not just colours, but columning and filetype indicators as well.

djoe48• 14 years ago

ls | grep junk | xargs rm # is dangerous due to The separator problem https://en.wikipedia.org/wiki/Xargs#The_separator_problem.

rcthompson• 14 years ago

ls | grep junk | xargs trash-put

https://github.com/andreafrancia/trash-cli/#readme

webreac• 14 years ago

I try to use perl instead of sed, awk, bash each time the needs are beyond basic usage. This applies also sometimes for xargs for example with the following command. find . -name junk\* | perl -nle unlink

My favorite usage of xargs if for parallel operations: cat hostlist | xargs -P 0 -n 1 -I host rsh host gzip -9 log/*.log

slashclee• 14 years ago

xargs is so 20th-century. parallel[0] is way more awesome (especially with all the multi-core systems everybody has now). Cool bonus: all the things you learn for xargs work in parallel too!

[0] http://www.gnu.org/software/parallel/

wwwhizz• 14 years ago

I Was wondering what the difference with running parallel jobs using xargs -P was, but that's covered in their man-page: http://www.gnu.org/software/parallel/man.html#differences_be...

knewter• 14 years ago

A billion times, this. I just learned of GNU Parallel about 3 months ago and wish everyone knew about it. Have seen it used to stunning effect with a comically large pipeline (OCR patents) across a comically large number of EC2 instances. Very awesome indeed.

gizzlon• 14 years ago

"What xargs is doing is taking each line of input and applying the supplied argument or command to the line of input"

Actually it does not run command once for each line of input. This confused me for some time until I read the man page:

"The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). The specified command will be invoked as many times as necessary to use up the list of input items. In general, there will be many fewer invocations of command than there were items in the input. This will normally have significant performance benefits. Some commands can usefully be executed in parallel too; see the -P option"

Think you're looking for "-n 1" if you want that behavior..

leed25d• 14 years ago

I was going to respond on this very point myself. This blog entry is just plain wrong wrt the way that xargs works.

bitops• 14 years ago

OP here - thanks all for the comments - so yes, it looks like I don't know xargs as well as I thought I did. I'll update the post to be more accurate.

As for the points about a bad example, I am fully aware that it is extremely contrived. The point was to not confuse anyone with commands they don't already know. I'm wary of posting examples that necessitate people learning 4-5 new commands when their focus is learning just one.

A small primer on xargs

Related Stories