Find|xargs like a boss (Real Example)

http://www.r-bloggers.com/find-xargs-like-a-boss/

by g-garron • 14 years ago

41 20 14 years ago

noodly2• 14 years ago

Comments are closed on that page, so instead I will comment here.

There's no need to use xargs, and pipe to sh in this scenario. The same functionality can be achieved by using only find:

  PROCS=8;cuffs=`which cufflinks`;find `pwd` -name 'accepted_hits.bam' -type f -exec qsub -l ncpus=$PROCS -- $cuffs -p $PROCS -o {}-cufflinks -g genes.gtf {} \;

barrkel3• 14 years ago

Creating a text stream which you then pipe to a shell is a good way of debugging complex operations, especially potentially destructive ones that may need inspection to verify. It's a technique I use a lot, to the point that some of my more destructive scripts don't do anything - they just output shell commands, and it's up to me to pipe them to the shell.

noodly2• 14 years ago

You can achieve this also by using find and its -ok option, instead of -exec, that way find will ask for confirmation before doing anything.

icebraining• 14 years ago

xargs is very useful, although nowadays I usually use GNU parallel instead (it's essentially the same but with a few advantages: https://www.gnu.org/software/parallel/man.html#differences_b...)

makmanalp• 14 years ago

xargs -P is sort of similar

wging• 14 years ago

The link in the post to which you replied covers that; it's the section of the man page which deals specifically with differences between xargs and parallel. Having used neither of these tools before, I think two relevant lines from that link would be:

>xargs can run a given number of jobs in parallel, but has no support for running number-of-cpu-cores jobs in parallel.

>xargs has no support for keeping the order of the output, therefore if running jobs in parallel using xargs the output of the second job cannot be postponed till the first job is done.

brown9-2• 14 years ago

Came here to say this. GNU Parallel is great.

saurik• 14 years ago

I would have expected an article on how to use these tools "like a boss" to point out that if there is whitespace in the filenames you will run into problems unless you use "find ... -print0 | xargs -0 ...".

masklinn• 14 years ago

What I'd have expected in an article on how to use this tool is talk about `-exec` and `-exec +` before shooting straight for xargs.

barrkel3• 14 years ago

I very rarely use find -exec, because I usually want to massage the list of files further. So I have a simple utility, print0, which reads lines and outputs null-terminated lines. It turns any list separated with newlines into input suited to xargs -0. File names with embedded newlines are vanishingly rare, compared to those with spaces, so it has always worked out well for me.

zaptheimpaler• 14 years ago

This was one I used to backup all websites in a folder:

  find . -mindepth 1 -maxdepth 1 -type d | awk 'BEGIN {FS="./"}; {print $2}' | xargs -d '\n' tar czf backup1.tar

Note that you'll have to change $2 to the correct column number for the name field depending on your output.

Domenic_S2• 14 years ago

Why not just cp the parent directory?

narsil• 14 years ago

This command doesn't include the files in the parent directory.

An easier way to accomplish the original command uses globbing (like in bash):

    tar czf backup1.tar */

sgt• 14 years ago

For the record, these examples assume GNU xargs, so they won't necessarily work with the BSD xargs that comes with OS X.

zaphar• 14 years ago

The author would benefit from replacing his backticks with $(pwd) instead. You can nest this form arbitrarily deep as well as having it interpolate in strings or heredocs.

gcr• 14 years ago

$(foo) is a bashism; unsupported by different (older) versions of bash and especially sh

orra• 14 years ago

I don't know how well $(foo) is supported outwith bash in practice, but it is POSIX: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3...

Find|xargs like a boss (Real Example)

Related Stories