A Unix Utility You Should Know About: Pipe Viewer

http://www.catonmat.net/blog/unix-utilities-pipe-viewer/

by pkrumins • 17 years ago

201 63 17 years ago

catonmat.net

pkrumins2• 17 years ago

ask hacker news: what other utilities do you know that others don't? :)

gcv2• 17 years ago

I was happy to discover rlwrap. Not every REPL or utility-with-its-own-prompt out there supports readline by default.

jsrn3• 17 years ago

mmv (moves multiple files based on a simple pattern replacement language). Example:

    mmv "*.mp3" "old_#1.mp3"

prefixes all files in the current directory ending with .mp3 with 'old_'. Btw., the quotes around the arguments are necesssary because mmv uses some of the same metacharacters as the shell does.

Multiple substitutions are allowed, e.g.:

    $ mmv "*.*" "#1___#2"

renames blub.xyz to blub___xyz.

nuclear_eclipse3• 17 years ago

I've been looking for something like that for years!

mark_h• 17 years ago

Andrew Myers has a shell script called mvsed that does something similar, using sed to specify the patterns.

Found it: http://www.cs.cornell.edu/andru/useful.html

nickb• 17 years ago

Anyone know where to find a source tarball that compiles under Leopard? All I've found is a bunch of precompiled Linux RPMs...

Erwin• 17 years ago

There's a 'rename' script that comes with Perl that also does this, which may be useful if you want some really fancy renames. E.g. rename '$_=lc' *.MP3 to change case to lowercase.

jarekr• 17 years ago

Zsh has a module called zmv which lets you do things like:

zmv '(??)-(*).mp3' '$1 - ${(C)2:gs/_/ }.mp3'

Which would change "01-foo_bar_baz.mp3" into "01 - Foo Bar Baz.mp3"

yan4• 17 years ago

A short list of what I like and use:

the command 'tee': let's you see what is happening at an intermediate stage of a pipeline (I originally thought your blog post would be about tee)

the -S flag to objdump: If you're trying to learn how compilers work or assembly, the '-S' flag to objdumb is absolutely beautiful. If you compile a binary with debugging symbols (-g in gcc), 'objdump -S binary' will intersperse the assembly with the original code, letting you see what each line compiled into.

I use 'pushd' and 'popd' bash builtins. It lets you remember current directories and have a stack of locations that you've been. Very valuable when you're jumping around a large source tree.

Even though not a command, I usually follow ls by "-ltr". That arranges it such that the most recently modified file is on the bottom. Very easy to see what files were modified.

"du -sk * | sort -rn" for a nice listing of directory sizes.

Pressing "esc, ." (That's the Esc key, followed by a period. Not together) in bash would insert the previous command's argument. i.e.:

  bash# ls /some/really/long/path/to/a/huge/file.c
  bash# vi [esc, .]    # <-- that would insert the path
  bash# vi $_          # <-- same effect

'tree' is nice to picture a directory hierarchy.

gcv2• 17 years ago

Looks like the Esc-. trick works in zsh as well. I didn't know about it. Cool. Thank you.

With regard to pushd and popd: these tricks work in zsh also. In addition, (again, zsh-specific) you can set these options:

  setopt auto_pushd         # automatic directory stack
  setopt pushd_minus        # swap +/- for pushd and popd
  setopt pushd_silent       # no output after pushd and popd
  setopt pushd_to_home      # pushd with no arguments goes to ~
  setopt pushd_ignore_dups  # no duplicates in directory stack

With these options set, every time you change directories using cd, the directory goes into your directory stack, basically executing an implicit pushd. Then, move around the directory structure a little, and try typing 'cd -<Tab>' (meaning type 'cd -' and press Tab for autocompletion). It'll list entries in your directory stack by number (so -1 is your previous directory, -2 is the one before, and so on). It's a handy wrapper for bare pushd/popd functionality.

Bash might offer similar functionality, but I don't know much about bash.

nixme• 17 years ago

I find M-. (on most keyboards, hold Alt and press .) to be faster than Esc,. although using Esc as a "metafy-next-key" is useful for broken terminals.

_phred• 17 years ago

!$ is the substitution in most shells for the last arg of the last command. I use it all the time as it's really easy to type.

pwk2• 17 years ago

I find xargs handy, it lets you use the output of one command as arguments for another command. As an example, I have a little alias that I use to grep through my ruby/rails source trees:

  alias gr 'find . -name \*.rb -o -name \*.rhtml -o -name \*.erb -o -name \*.rjs | xargs egrep \!*'

(backticks can be used similarly, but can run into command line length limits)

pwk2• 17 years ago

Oh, and of course netcat (nc). Indispensable for debugging anything TCP/IP based.

  echo -n 'GET / HTTP/1.0\r\n\r\n' | nc news.ycombinator.com 80

Of course there are more specific tools for specific protocols, like curl or wget for http, but nc is the swiss army knife.

yan4• 17 years ago

nc is also awesome for quick and dirty file/directory transfer over the wire:

  host1 (receiving): nc -l 6123 > tar xfz -
  host2 (sending  ): tar cfz - directory/ | nc host2 6123

jacquesm• 17 years ago

not really a utility but a very handy one liner to check if someone is being annoying on a http server:

tail -1000 logfile | cut -d ' ' -f 1 | sort | uniq -c | sort -n

Bots should stand out in the last couple of lines and have a linecount that is substantially higher than a normal user.

cliffy• 17 years ago

Is there an equivalent utility for Windows?

newt0311• 17 years ago

Wouldn't windows have to have a usable shell first?

quoderat• 17 years ago

Agreed. Even powershell is so slow. And needlessly complex.

gnaritas4• 17 years ago

It does, it's called cygwin and it's the first thing you install after getting windows to boot.

likpok3• 17 years ago

SUA seems to be what my windowsy friends talk about. It's in newer versions of windows. It natively supports many UNIX apps, and has source compatibility with UNIX.

nailer4• 17 years ago

There's an object shell included by default in both Windows 7 and 2008.

There is none in Linux - you end up having to find regexs to match data rather than simply piping to where and asking for the field you want.

migpwr• 17 years ago

This sounds useful but I don't like the fact that it's named "pv". I've always associated "pv" with physical volume and lvm... might be a little confusing.

kaens• 17 years ago

You could create an alias for it, like

    alias eta='pv'

akkartik• 17 years ago

I made a graphical wrapper a few years ago -- call it from a terminal, but if you have X it won't munge stderr.http://akkartik.name/bash.html#pipeview

wmf• 17 years ago

For more pipe tricks, check out bidilink:

http://0pointer.de/lennart/projects/bidilink/

aston• 17 years ago

I just learned about the --forest switch for ps. If you use it, you'll get a tree representing all of the parent and child processes running rather than just a straight up list.

FlorinAndrei• 17 years ago

Or -f

As in:

ps axf

raamdev• 17 years ago

Thanks! I've been using a tool called pstree for the same thing (I didn't even realize ps had such a feature). It looks like the ps utility on OS X doesn't support --forest though, so pstree is still useful there.

http://freshmeat.net/projects/pstree/

Dobbs• 17 years ago

The --forest or as later mentioned the -f does not exist in the freeBSD version of ps (I'm assuming all bsd's).

If my understanding is correct OSX uses the freebsd utilities so that would be your reason.

ivank• 17 years ago

htop can also do this.

palish2• 17 years ago

--forest? Seriously?

Metaphors are sometimes confusing. Call it what it is. "Reflection" should have been called "type information", for example, and "--forest" should have been called "--nested".

yan4• 17 years ago

A process tree is known to almost everyone who uses Unix. A forest is, in my opinion, a good name for a collection of trees.

Reflection is the ability of something to introspect. The term "self-reflection" dates back to 1652[1], so why change it? "Type information" is inaccurate because reflection can mean more than just getting information about types, like interfacing with the runtime or getting information about the language implementation.

I don't think concepts should be misnamed at the expense of accuracy.

[1] http://www.merriam-webster.com/dictionary/self-reflection

lacker• 17 years ago

You're talking about a tool whose name is "ps". Unusual names for commands are par for the Unix course.

rkowalick• 17 years ago

In graph theory, a collection of disjoint trees is called a forest.

Keyframe2• 17 years ago

anyone have an idea how this might work in tandem with dd, for example dd if=/dev/urandom of=1GB.bin bs=1M count=1024

blasdel• 17 years ago

Easy!

  dd if=/dev/urandom bs=1M count=1024 | pv | dd of=1GB.bin

Keyframe2• 17 years ago

thanks!

pkrumins2• 17 years ago

or simpler:

    dd if=/dev/urandom bs=1M count=1024 | pv > 1GB.bin

ralph• 17 years ago

Sometimes, pv(1) doesn't help. For example, `tar cf - foo | bzip2 -9v >foo.tar.bz2'. You don't know the size of the data that needs to pass down the pipe. But I sometimes find watching tar(1) open the files to read is handy; `strace -e trace=open $(pidof tar)'.

A Unix Utility You Should Know About: Pipe Viewer

Related Stories