Unix Commands I Wish I’d Discovered Years Earlier

http://spin.atomicobject.com/2013/09/09/5-unix-commands/

by micahalles • 13 years ago

480 250 13 years ago

spin.atomicobject.com

I'm always surprised at the lack of general knowledge of Regular Expressions, so egrep would be a huge one for newcomers.

foobeer• 13 years ago

Never knew about the "cal" command. I especially like that you can quickly look up a previous or future month/year. >> cal 3 1973 quickly shows a calendar of March 1973

yogo• 13 years ago

Aliasing to 'cal -3' can also be good if you want a little continuity.

xrt• 13 years ago

The -3 arg is not available on OS X. Odd, since it's been in BSD Unix since forever. http://www.freebsd.org/cgi/man.cgi?query=cal

Shish2k• 13 years ago

you can even look up `cal sept 1752`

quchen• 13 years ago

Funny, 3-13 are missing here.

  > cal sep 1752
     September 1752     
  Su Mo Tu We Th Fr Sa  
         1  2 14 15 16  
  17 18 19 20 21 22 23  
  24 25 26 27 28 29 30

Edit: Oh, colour me surprised. https://en.wikipedia.org/wiki/September_14

mturmon4• 13 years ago

Right, you can use this to easily find the number of days in an arbitrary month in a script:

  % cal 2 1972 | tail +3 | wc -w
      29

parennoob2• 13 years ago

tail: cannot open +3 for reading: No such file or directory

What am I doing wrong?

This worked like a charm though:

cal dec 2002 | tail -6 | wc -w

(since it always takes up 6 lines, even sep 1752)

happywolf• 13 years ago

I recommend Quicksilver on mac, free and powerful. It lets you to find a lot of stuff (application, contact, email, etc.) and manipulate them (send, forward, print, etc.)

http://qsapp.com/

jentulman• 13 years ago

Also in that vein on Mac it's worth having a look at Alfred http://alfredapp.com

actionscripted• 13 years ago

Alread rocks. You can develop workflows for common groups of tasks and it integrates well with iTunes, 1Password, Terminal and other system actions/apps.

9999• 13 years ago

I second the Quicksilver recommendation. When you combine Quicksilver triggers (keyboard shortcut available regardless of what application you're in) with applescript, and fabric it's very powerful. I have a trigger that that restarts all of my local services using fabric and opens up new terminal windows running tails of the relevant logs.

The Applescript is this if anyone wants to do something similar:

tell application "Terminal" do script "tail -F /somepath/some.log" do script "tail -F /anotherpath/another.log" do script "sudo fab -f '/pathto/fabfile.py' local deploy" activate end tell

Here's a detailed explanation of setting up Triggers:

http://leafraker.com/2007/09/17/how-to-create-a-quicksilver-...

ck25• 13 years ago

I want a command that counts files in a tree like du does sizes but without having to pipe find through wc which is crazy for hundred thousand files.

Can't they just directly access inodes for high speed counting?

casca2• 13 years ago

Counting number of files is probably not a sufficiently frequently used case to justify a new standard command. What part of piping concerns you?

sp332• 13 years ago

It can be really slow, even for a few tens of thousands of tiles. I remember watching Nautilus's "properties" panel slowly count up the number of files and directories.

njharman• 13 years ago

Can't they just directly access inodes for high speed counting?

Don't know what you mean by that exactly. But, I'd be stunned if find is actually reading the actual files rather than just reading the directory entries in directory "files". Which is what I'd call accessing inodes "directly"

bendaid• 13 years ago

tree does this, don't know about suppressing the actual tree, but it tells you number of files and dirs printed at the end.

laumars7• 13 years ago

A regex match will do that easily enough:

    tree | egrep -o "[0-9]+ file(s|)"

edit: or you could just tail the output (should have thought of this first!)

    tree | tail -n1

ck25• 13 years ago

alas it does not show file count per subdirectory

laumars7• 13 years ago

It's not usually installed by default, but tree will do this: http://linux.die.net/man/1/tree

Available in most repositories, source code available at: http://mama.indstate.edu/users/ice/tree/

res0nat0r3• 13 years ago

There really isn't any way around this. The first pass over your data is going to be slow while the inode data is examined, but it will be cached for the next pass. If you do this before that data gets removed from the cache it will be much faster.

  imac:~
  $ time find . | wc -l
  419191

  real	0m37.865s
  user	0m0.385s
  sys	0m2.611s
  imac:~
  $ time find . | wc -l
  419195

  real	0m10.503s
  user	0m0.330s
  sys	0m1.430s

ck25• 13 years ago

This is a nasty hack where it is counting the output from find.

What do you do when find cannot handle the sheer number of files, argument too long from the linux hardcoded MAX_ARG_PAGES

We need a dedicated utility that counts inodes.

stormbrew• 13 years ago

If you're dealing with a truly large number of files you could use "find . -type f -printf 1 | wc -c" which would just print a single character to the pipe and count the characters. I really doubt this is significantly slower than one program that does both would be. It'll do roughly the same amount of work.

ajayss• 13 years ago

Try something like:

find . | sed -e 's=/[^/]*$==' | uniq -c

dredmorbius5• 13 years ago

It's not the pipe through wc, but statting the directory entries which slows you down.

If your locatedb is current, you can use locate for this:

    $ locate $PWD | wc -l

That runs in just a hair under a second (0.908s real) for the first instance. 'find' takes a few seconds on first pass (it's got to read from disk), but is surprising fast (0.227s real) on subsequent runs.

So I guess it depends on whether you're doing it just once or multiple times.

loudmax• 13 years ago

I have this in my .bashrc:

function lc() { if [[ "$#" -gt 1 ]]; then for DIR in "$@"; do echo -n "$DIR - " ; ls -AU1 $DIR | wc -l ; done ; else ls -AU1 "$@" | wc -l ; fi; }

Then "lc dir/" will print the number of files in the directory, and "lc dir/*" will print the number of files in each of the subdirectories. This exactly what you're looking for because it doesn't descend into the tree. If you're working with directories that may have tens or hundreds of thousands of files in them, it's useful to check first.

0x09• 13 years ago

du just uses the fts API, so if you want something directly equivalent (omitting options and error handling) it was only barely too long to post: http://pastie.org/8315296

Compared to the equivalent find, code #1 is much faster in this completely scientific best of several runs:

  $ time (find OSS/llvm -type f -or -type l | wc -l)
  real	0m4.681s
  $ time du -d0 OSS/llvm
  real	0m4.863s
  $ time ./fts OSS/llvm
  real	0m0.559s

Ostensibly because of the lack of stat calls. On the other hand the second paste and find with no arguments are equivalent.

edit: screwed up benchmarks

ck25• 13 years ago

Your code examples were very educational, thank you!

brown9-2• 13 years ago

I'm unreasonably excited about discovering xxd.

Luyt• 13 years ago

xxd is on OSX, but not on FreeBSD. Under FreeBSD, I always use 'hd somefile.bin'.

D9u5• 13 years ago

I have xxd on my FreeBSD setup.

    uname -rms
    FreeBSD 9.1-RELEASE-p19 amd64

Although the xxd man page doesn't have the BSD label, and is from 1996, so I'm not sure if xxd only recently found its way into FBSD.

    This manual page documents xxd version 1.7

    AUTHOR
       (c) 1990-1997 by Juergen Weigert

shabble2• 13 years ago

I've been told before that `od' is the most portable, although I prefer xxd in virtually all situations.

sebnukem22• 13 years ago

That's how I inspect a file in Vim: :%!xxd

It's even listed as a 'Convert to HEX' item in the Vim GUI menu.

marcosdumay2• 13 years ago

I never heard of it, untill today.

I can't remember how many times I implemented hexadecimal dumps and undumps because I didn't want to coerce od into working the way I wanted.

casca2• 13 years ago

TL;DR: man ascii, cal, xxd, ssh, mdfind

chrisdotcode• 13 years ago

`mdfind' in non-Mac UNIXs is spelled `locate' (and the corresponding `updatedb' updates the indexing db, obviously (although you don't have to do this often, as it runs as a cron job)).

Camillo• 13 years ago

mdfind is incomparably more powerful than locate.

mkramlich• 13 years ago

and for readers age 30-40+:

Summary: man ascii, cal, xxd, ssh, mdfind

:-)

mpyne• 13 years ago

I believe xxd is actually part of vim and not a Unix command itself. It's used to support hex-editing binary files, but you can see why it's useful in other roles as well.

D9u5• 13 years ago

The origins of xxd seem to be a bit murky, as I've seen others say the same thing about xxd being available only if vim is installed, but I've never seen a Unix system that didn't have vi(m) included.

    $ man xxd | grep vi
       Use xxd as a filter within an editor such as vim(1) to hexdump a	region
       Use xxd as a filter within an editor such as vim(1) to recover a	binary
       Use xxd as a filter within an editor such as vim(1) to recover one line

http://grail.cba.csuohio.edu/~somos/xxd.c

From what I've been able to find, vim was released in 1991, and xxd has been around since at least 1990, whereas vi was written by Bill Joy in 1976. (according to Wikipedia)

Interesting bit of trivia, perhaps someone with a good memory can clear it up for us?

clarry5• 13 years ago

There are no hits for xxd on the Unix Tree (http://minnie.tuhs.org/cgi-bin/utree.pl) so it's definitely not a traditional Unix tool. I'm sitting in front of a BSD system which includes nvi. I don't have xxd.

Amadan• 13 years ago

Well, at least on Debian it's in `vim-common` package. http://packages.debian.org/search?searchon=contents&keywords...

vidarh• 13 years ago

There was a time when I, annoyed at being thrown into vi(m) whenever something spawned an editor, rather than setting the EDITOR environment variable on login, removed vi and replaced it with symlinks to emacs on the servers I managed.

I'm a bit more tolerant of vi now, but just because I "never" see it courtesy of suitable EDITOR entries.

rabino• 13 years ago

I recently discovered whatis and I love it.

peterwwillis• 13 years ago

Yup, it's a great way to review a lot of commands without reading every man page.

Want to find all the tools on your system that use the curses library (for user-friendly interfaces)? Try this:

  for i in `\ls /bin/* /sbin/* /usr/bin/* /usr/sbin/* | sort -u` ; do ldd $i 2>/dev/null | grep -q curses && whatis `basename $i` ; done | grep -v 'nothing appropriate'

Don't know how your current path finds a program? Use which:

  pwillis@zippy:~$ which df
  /usr/bin/df
  pwillis@zippy:~$ which mount
  /bin/mount

Annoyed that /sbin and /usr/sbin aren't in your path? Use whereis:

  pwillis@zippy:~$ which ifconfig
  which: no ifconfig in (/usr/local/bin:/usr/bin:/bin:/usr/games:/usr/lib64/kde4/libexec:/usr/kerberos/bin:/usr/lib64/java/bin:/usr/lib64/java/jre/bin:/usr/lib64/qt/bin:/usr/share/texmf/bin:.)
  pwillis@zippy:~$ whereis ifconfig
  ifconfig: /sbin/ifconfig /usr/man/man8/ifconfig.8.gz /usr/share/man/man8/ifconfig.8.gz /usr/X11/man/man8/ifconfig.8.gz

jlgreco8• 13 years ago

The zsh builtin 'where' is pretty good too:

  > where ls
  ls: aliased to ls --color=tty
  /usr/bin/ls
  /bin/ls

Wicher• 13 years ago

Apropos 'which', let me show 'type -a':

  % type -a which
  which is a shell builtin
  which is /usr/bin/which

Convenient if you really want to know what you'll be calling.

hannibal5• 13 years ago

Unix and Common Lisp are similar in many ways:

1. Both are operating systems

2. Both are conceptually simple.

3. There is always nice command that does what you want but you somehow forget it. I can't believe that Common Lisp has just 900+ symbols but I routinely forget some of them when programming.

samatman• 13 years ago

Indeed! Apropos is our friend.

qu4z-22• 13 years ago

Or if apropos isn't on your system, man -k does the same thing.

... I think it works when apropos isn't installed. Possibly it just calls out to it.

Keyframe• 13 years ago

My favorite part is when you implement something you could've had running in a minute or so, but instead developed it for hours or days and it's still not as good as the existing solution. I often get that with C and libraries. There has to be a better way to search through existing stuff which is good and proven, not just for C of course.

philsnow2• 13 years ago

Haskell has hoogle [0], that lets you search by type signature. If you already know roughly what kind of function you want, it's a lot easier to sift through a tiny handful of (or even a single) functions that match a given type signature.

ISTR Forth had something similar, or maybe I'm just thinking that in Forth, words are almost uniquely defined by their stack annotations.

[0] http://www.haskell.org/hoogle/

grimgrin2• 13 years ago

Replaces the current day of the month with []:

     $ cal | sed "s/.*/ & /;s/ $(date +%e) / [] /"

        September 2013
     Su Mo Tu We Th Fr Sa
      1  2  3  4  5  6  7
      8  9 [] 11 12 13 14
     15 16 17 18 19 20 21
     22 23 24 25 26 27 28
     29 30

res0nat0r3• 13 years ago

Gcal will highlight the current date for you.

http://www.gnu.org/software/gcal/

nawitus2• 13 years ago

Hmm, the day is already highlighted for me (inverted color).

ape4• 13 years ago

You can turn that off with:

  cal | cat

grimgrin2• 13 years ago

It appears that on OS X 10.6 cal is from 2004, and on Ubuntu it's a 2009 build. The 2009 build highlights the day of the month.

acheron• 13 years ago

ctrl-R in bash was mine.

claudius• 13 years ago

Try C-x C-e – especially helpful if you want to paste stuff from websites.

acomjean• 13 years ago

nice! did not know that.

reginaldjcooper• 13 years ago

the last time one of these came up I learned about bash's vi mode:

    set -o vi

it's awesome

fsckin• 13 years ago

One of the more useful bits of ssh is not mentioned: remotely running commands.

Example:

ssh username@host "echo $HOSTNAME && sudo somecommand && cat somecommand.log"

There's probably a better way to do this, but in a pinch I can fix a problem on dozens of machines just by altering the host string.

Kurtz792• 13 years ago

Still about ssh :

"typing a password on your local machine once, then using a secure identity to login to several remote machines without having to retype your password by using an ssh key agent. This is awesome. "

Sounds awesome, but I really didn't understand what he means. Is he talking about using a certificate (-i option) to log in ?

oftenwrong2• 13 years ago

I am fairly sure they are referring to ssh-add

laumars7• 13 years ago

Yeah, you have a cert set up and put the public cert in the identities file for each server you want to connect to. Then just SSH into each box with agent forwarding ('-a' I think, but you're best to check that).

So you'll need to create an SSH cert first:

    ssh-keygen -b 4096 -t rsa -C "$USER created on $(hostname)" -f $USER.key

Keep the private key, that's essential you keep this safe as it's literally your key to access any servers you do the next step for.

On each server SSH, append your public key to the of /home/$USER/.ssh/authorized_keys (creating the file if it doesn't exist). It must be the public key!

There'll be better guides online for this stuff so I suggest you have a read through them before blindly pasting the stuff I've posted into your terminal. But it's all quite simple stuff once you've grasped the difference between a public and private key (if you weren't already aware - you may well be)

jfb7• 13 years ago

The single most useful thing about ssh to me is that it connects stdout/in across the channel (this is behavior inherited from rsh). This allows for e.g.:

  % (cd /foo; tar cpf - .) | ssh bar \(cd /baz\; tar xpf -\)

Or:

  % cat ~/.ssh/id_rsa.pub | ssh bar tee -a .ssh/authorized_keys

sopooneo• 13 years ago

Forgive my ignorance, but I don't know what what it means to connect "across the channel", but it sounds important. Do you know a link that explains the concept? I've failed to Google it.

dmckeon• 13 years ago

This is a bit of a pet peeve of mine, but cd/tar should likely be joined with && - in other words, in what directory do you want tar to (not) run if cd fails? So consider:

    % (cd /foo && tar cpf - .) | ssh bar '(cd /baz && tar xpf -)'

or explore something like:

    % rsync -aSHx --update --delete -e ssh /foo/ user@bar:baz/.

0x45696e6172• 13 years ago

ssh-copy-id is made for exactly this:

  ssh-copy-id  -  install  your  public  key in a remote machine’s authorized_keys

WestCoastJustin5• 13 years ago

Yeah, before I was heavily into automation (i.e. puppet) this was my go to command. Use a for loop and ssh to run things on a group of systems.

  for host in h-abc h-def h-ghi h-jkl h-mno h-pqr h-stu h-vwx h-yza; 
    do 
    ssh -l root $host "hostname; echo "1.1.1.1 server" >> /etc/hosts"; 
  done

This would iterate over hosts (h-abc .. h-yza) and run the commands "hostname; echo "1.1.1.1 server" >> /etc/hosts". Or maybe use a for loop and scp to push the files out and then run md5 to verify they are correct. It was a hack but useful for a small set of machines.

ps. if you are interested in learning about puppet, I have put together a screencast about it @ http://sysadmincasts.com/episodes/8-learning-puppet-with-vag...

philsnow2• 13 years ago

make a small change:

  for host in ... ; do ssh -l root $host "..." & done; wait

and it's very parallel. All that key exchange takes a while, so it's not ridiculously fast, but it's pretty good. If you've got ControlMaster set up already to be per-host, this might very well be very fast the second time.

dredmorbius5• 13 years ago

dsh (distributed/dancer's shell) is a wrapper around ssh which provides that capability. Plus you can define hostlists.

nawitus2• 13 years ago

I think most people know about this, but it's also good to know about scp. You can easily copy files using a ssh connection.

kbenson• 13 years ago

Scp doesn't get all file types correct (fifos, etc), while tar does.

Piped tar commands do, but are generally inferior to rsync (at least if you ever need to copy more than once). I believe I've heard (here) that piped tar commands can be useful for the first sync (possibly better compression, no overhead of file checksums), then rsync thereafter.

vowelless• 13 years ago

'screen' is absolutely crucial with ssh, too.

jlgreco8• 13 years ago

I find tmux to be a bit more pleasant to work with. In the past it didn't do reflowing upon resize, but it does that now.

daenz• 13 years ago

sh.py leverages this in command baking http://amoffat.github.io/sh/#baking

PuercoPop• 13 years ago

Protip, you can make emacs use mdfind with (setq locate-command "mdfind")[1]

[1] http://emacsredux.com/blog/2013/07/05/locate/ (setq locate-command "mdfind")

WestCoastJustin5• 13 years ago

If anyone is interested there were several great posts on Hacker News a while about about useful UNIX commands [1, 2, 3]. I have also created several screencasts about command commands like the following [4], and one about the Filesystem Hierarchy Standard [5].

  ls man pwd cd top ps df du cp mv rm mkdir rmdir less cat vi

[1] https://news.ycombinator.com/item?id=6046682

[2] https://news.ycombinator.com/item?id=5022457

[3] https://news.ycombinator.com/item?id=4985393

[4] http://sysadmincasts.com/episodes/13-crash-course-on-common-...

[5] http://sysadmincasts.com/episodes/12-crash-course-on-the-fil...

jlgreco8• 13 years ago

You should do a screencast of such infinitely useful classics as tac, tsort, and sl.

(Actually, I have found use for those first two on occasion. ;)

WestCoastJustin5• 13 years ago

Thanks for the suggestion! Actually, I initially wanted to do one on more advanced commands, but I thought I'd better lay a foundation first. I also want to cover using pipe (|) to chain these commands together. I'll make sure to include these.

jfb7• 13 years ago

tsort is one of my favorites, mostly because it's totally pointless until you need it and then it works exactly right. I love it.

Goladus• 13 years ago

Great links, though I admit when I saw the OP I was expecting content like that and was pleasantly surprised that they really were "Unix commands I wish I'd discovered years earlier", and not the standard array of unix tips+tricks that you can get by without for awhile but as you become more advanced become second-nature.

"Man ascii." The number of times that I wound up generating my own ascii table from whatever language I was working with...

"xxd" I always either used search in emacs hex-mode (which I find cumbersome and annoying but haven't taken the time to customize), or rolled my own search using a programming language. This would have come in handy SO many times back when I used to muck around with videogame save files and the like. I never even knew it existed nor how to discover that it did.

'cal' I learned about early on so never missed it and the 'ssh' stuff kind of fits into the regular "unix tips and tricks" category.

WestCoastJustin5• 13 years ago

re: ascii tables

You reminded me of something. There is actually a really cool and simple site called asciiflow [1], which I use all the time to draw diagrams for explaining things in email, etc. It's pretty cool, and was even submitted several times to HN [2, 3].

[1] http://www.asciiflow.com/#Draw

[2] https://news.ycombinator.com/item?id=2847177

[3] https://news.ycombinator.com/item?id=3598177

gpvos3• 13 years ago

xxd, together with xxd -r, is a rather practical way to turn vim into a hex editor.

gcb02• 13 years ago

i usually click those links, see the stuff and roll up my eyes. nothing learned, but upvote so more people learn about them anyway.

this time, the very first one `man ascii` got me. Genius. Never though about looking for that in man before. But in my defense, this was added in 1999 :)

gpvos3• 13 years ago

An ascii manual page appeared in Version 7 of AT&T UNIX.

I'm sure that was a bit earlier than 1999.

simonreed• 13 years ago

man units

Kurtz792• 13 years ago

I know it's not exactly an unknown command, but I didn't know about "sort" until last week.

It's freaking fast and convenient, sorts hadoop reduce results like a champ.

microtonal• 13 years ago

Wow, I use it (almost) daily. Especially useful with the '-u' flag or uniq.

WestCoastJustin5• 13 years ago

One of my favorite things about unix is that you can chain commands together with pipe (|). Say for example that you have a example data file that looks like this:

  # Deflection    Col-Force       Beam-Force 
  0.000              0              0    
  0.001            104             51
  0.002            202            101
  0.003            298            148
  0.0031           104            149
  0.004            289            201
  0.0041           291            209
  0.005            104            250
  0.010            311            260
  0.020            104            240

You can chain commands together to quickly get ball park figures. I do this all the time when reviewing log data to get simple counts and look for anomalies.

  $ cat data.txt | grep -v '#' | awk '{print $2}' | sort | uniq -c
      1 0
      4 104
      1 202
      1 289
      1 291
      1 298
      1 311

Let me break this down a little. I cat data.txt, which just prints the contents to the console, I use grep to remove the # header, then use awk to print the second data column, the sort the values for my next procedure, and lastly I use uniq -c, which takes the input and counts all the unique values. So you can see that "4 104" means that 4 instances of 104 were found. I use this often to look at httpd logs for IP addresses or to review sudo commands, etc. Very handy!

ps. there are optimizations that could be done here, but that what is great about unix and "one function" utilities. There are many ways to the same destination!

oftenwrong2• 13 years ago

Yes, there are some ways to simplify. You can get rid of the "UUOC". Instead of

     cat data.txt | grep -v '#'

you can do:

     grep -v '#' data.txt

or you can simplify further by moving the grepping into awk, so that the full command becomes

      awk '!/#/{print $2}' data.txt | sort -n | uniq -c

sort -n is good to use for sorting numbers. You could probably pull the sort | uniq -c into awk somehow, but I have already demonstrated the full extent of my awk knowledge. One of these days I will get around to actually learning it.

wiredfool2• 13 years ago

And once you've done that, join is a handy little utility to do sql style joins on your text files.

anonymous• 13 years ago

I find it fascinating that someone could be using something as complex and slow as Hadoop and not know of sort(1). Not faulting any lack of awareness, but it is really interesting that Google map/reduce and Hadoop marketing (word of mouth?) are so effective while the contents of /usr/bin are like buried treasure.

The funny part is that one of the early uses of Google's map/reduce approach was probably to distribute the job of sorting search results among many servers. And here you are sorting Hadoop results with something as small and simple as sort(1).

rfatnabayeff• 13 years ago

There are some other useful commands for organizing the stream of datas for pretty printing:

column - columnify the incoming stream into columns

pr - set up incoming stream for pretty printing including columnification along or across the screen

tr - sanitize input, collapse several delimiters into one

epsylon• 13 years ago

I use tr all the time when I want to check $PATH:

echo $PATH | tr : '\n'

joshcorbin• 13 years ago

One of my favorites: column -t

Examples include: $ df -h | column -t $ column -t -s: /etc/passwd

sebnukem22• 13 years ago

How can one use a Unix/Linux system without knowing SSH? How to login remotely without it?

codegeek2• 13 years ago

you can use telnet as an alternative even though it is not secure like ssh.

jfb7• 13 years ago

There are a large percentage of people who first encounter unix on single-interactive-user machines.

Toenex2• 13 years ago

Wow, now I feel old. I thought everyone got started with SunOS on a Sparcstation 2? Seriously, don't most people encounter it either through installing a Linux distribution or firing up a terminal on OSX? Or is even my-first-Unix a virtual experience?

Nick_C2• 13 years ago

There are free unix shells out there, google will find them. Don't expect anything fancy, but you will be able to try all the commands talked about here.

To connect to the shell from Windows, you can use the Putty program (google for it), although the shell documentation will almost certainly tell you how to do it.

codegeek2• 13 years ago

I love "tail -f" to see realtime updates on a file that I am writing to!!

shawkinaw2• 13 years ago

If you don't want to clog up your terminal, try 'less' and then hit 'F' to follow.

zipperhead• 13 years ago

Better yet use 'less +F'. It's as easy to remember as tail -f.

The advantage of using less here is that you can easily interrupt the tailing (ctrl-c). For example, to search. Then just hit 'F' again to restart tailing. But now your search pattern will be highlighted as the data streams by.

shawkinaw2• 13 years ago

I really love pushd/popd.

TomAnthony• 13 years ago

Ok, that is pretty freaking awesome. Thanks.

For others who don't know it: http://en.wikipedia.org/wiki/Pushd_and_popd. It essentially is a stack for the working directory, so you can get back to where you were previously quickly and easily.

clarry5• 13 years ago

For OpenBSD ksh I've got this hack

  alias d='[ ${#DIRSTACK[@]} -gt 0 ] && for i in `jot ${#DIRSTACK[@]} 1 ${#DIRSTACK[@]} 1`; do echo ${DIRSTACK[$i-1]}; done; echo $PWD'
  alias popd='[ ${#DIRSTACK[@]} -gt 0 ] && { cd "${DIRSTACK[${#DIRSTACK[@]}-1]}"; unset DIRSTACK[${#DIRSTACK[@]}-1]; }; d'
  alias pushd='DIRSTACK[${#DIRSTACK[@]}]="$PWD"; cd'

guard-of-terra2• 13 years ago

zsh has AUTO_PUSHD, guess what it does.

mafro• 13 years ago

jump just made it into oh-my-zsh:

http://jeroenjanssens.com/2013/08/16/quickly-navigate-your-f...

thejosh• 13 years ago

One of my favourite traceroute replacements, that is installed by default on Ubuntu: mtr !

salgernon• 13 years ago

Mac OS: pbpaste & pbcopy

For instance:

pbpaste | fgrep -i "`pbpaste -pboard find`"

To search the copy clipboard with the find clipboard.

greyfade3• 13 years ago

On X11 systems, there's also xclip:

    xclip -o -selection clipboard

deathanatos• 13 years ago

There's also xsel, which seems to be installed by default more often than xclip.

  … | xsel -b  # Copy.
  xsel -b | …  # Paste.

dredmorbius5• 13 years ago

I first ran into clipboard copy/paste on Mac. Discovered "xclip" on Linux. Aliased copy to 'xc' and paste to 'xp', later added 'xpc' to paste from the secondary (application) clipboard.

Makes all sorts of hackery possible.

FigBug• 13 years ago

In 2005 I wrote similar utilities for windows called ccopy, ccut & ppaste. I recently dug them up and put them on GitHub. https://github.com/FigBug/ccopyppaste

bndr• 13 years ago

I recently discovered ncdu (http://dev.yorhel.nl/ncdu/scr) as an alternative to du. Quite handy.

webreac2• 13 years ago

baobab ?

jlgreco8• 13 years ago

baobab is nice when you have X around, but ncdu has allowed me to get back up and running on more than one occasion.

why-el• 13 years ago

`cd -` to go back to the previous directory. Saved me LOTS of time over the past few months.

bizarref00l• 13 years ago

pushd and popd are very useful too.

nja2• 13 years ago

Wow. Thanks for this. I always cursed myself for forgetting to pushd, and now it turns out I didn't need to!

jamie_ca• 13 years ago

I recently found out that zsh lets you magically popd as well, as long as you're cding deeper into the tree. If you jump to a different subtree it will pop its internal stack until it finds a parent of the target dir.

kibwen2• 13 years ago

And `git checkout -` to checkout the previously checked-out branch.

tptacek• 13 years ago

Do yourself a favor: unzip the source for ascii.7 and change the order of the tables: hex, then decimal, then octal.

You'll thank me later.

jlgreco8• 13 years ago

I kind of like how the hex and char columns are next to each other though. Makes it easier to read those two columns together quickly.

ben0x539• 13 years ago

bash(?) thing I wish I'd discovered years earlier: diff <(command1) <(command2) instead of command1 > file1; command2 > file2; diff file1 file2; rm file1 file2

gnaritas3• 13 years ago

Yup, cool use of named pipes.

tripa• 13 years ago

Named pipes are filesystem objects you create with mkfifo.

What's in use here is more widely known as “process substitution”.

dredmorbius5• 13 years ago

Rather than 'man 7 ascii', there's the 'ascii' command itself.

It will also provide encodings for single characters: $ ascii a ASCII 6/1 is decimal 097, hex 61, octal 141, bits 01100001: prints as `a' Official name: Miniscule a Other names: Small a, Lowercase a

    ASCII 0/10 is decimal 010, hex 0a, octal 012, bits 00001010: called ^J, LF, NL
    Official name: Line Feed
    Other names: Newline, \n

clarry5• 13 years ago

Which Unix includes that command?

dredmorbius5• 13 years ago

Debian GNU/Linux. It's the 'ascii' package.

tmcb• 13 years ago

I'm quite impressed no one mentioned 'fg', 'bg', and 'jobs'. Since I got used to them, the number of open ptys in my screen dropped drastically.

talles• 13 years ago

Didn't know about the ascii one.

vectorpush3• 13 years ago

Command line globbing! For the uninitiated:

Let's say my pwd is `/home/projects` and i want to edit `/home/projects/a_huge_sprawling_app/940j_394/lol/flight_controller.rb`

  `vim **/flight_controller.rb`

opens our flight_controller.rb straight away. In terms of effort, this allows you to basically omit a `find -name` when you're in a rush to edit some damn file where the hell is it again damn we need to reevaluate this directory str...

----

Double bang to re-use your last terminal entry. One great use, taking all the pain out of forgetting your sudos

  `rm -rf /var/log`
  rm: cannot remove `/var/log': Permission denied
  `sudo !!`

All evidence of wrongdoing is now destroyed.

-----

Ok, here is an awesome one for users of the ultimate cloud IDE: Vim.

In your local ~/.ssh/config:

  `ForwardX11 yes`
  `SendEnv WINDOWID`

In your remote server's /etc/ssh/sshd_config:

  `AcceptEnv WINDOWID`

In your vimrc:

  `set clipboard=unnamedplus`

This has the effect of seamless yank and paste between local and remote vim sessions, no need for ctrl+shift+v.

I love linux.

ilikepi3• 13 years ago

FWIW, the double-star recursive globbing was added in Bash 4.0. My reading of the manual[1] suggests it is disabled by default, and can be enabled by setting the 'globstar' shell option. I believe it's enabled by default in zsh.

[1] https://www.gnu.org/software/bash/manual/bash.html#The-Shopt...

vectorpush3• 13 years ago

Thanks. Sometimes I forget that zsh is not what everyone uses. I should add zsh to the list, hah!

Typhon• 13 years ago

I just tried it on bash 4.2.25, and it worked, even though I just learned about it. Maybe not every OS enables the same default options for Bash (I'm using Lubuntu 12.10 at the moment)

parennoob2• 13 years ago

Re: command line globbing with globstar -- is there some way to get it to cache filepaths that are frequently accessed? It's quite slow for me (I usually use aliases for that, which I was contemplating getting rid of).

Thanks a lot for wising me up to the new hotness offered by bash 4+ (zsh is also pretty slow for me for some reason, even though people keep telling me to switch to that. Maybe I'm more impatient than most folks).

vectorpush3• 13 years ago

Sorry for the late reply. No built in way to cache globbed filepaths as far as I know. I did a little googling but to no avail. I suspect it wouldn't be too complicated to hack something together though. Another potential option: buy an SSD.

ja27• 13 years ago

xargs. Can't tell you how many times I wrote quickie C programs and awk scripts to do what xargs does.

http://sidvind.com/wiki/Xargs_by_example

marcosdumay2• 13 years ago

And find, that's a nice alternative to xargs when your args are a huge set of files.

penland• 13 years ago

Yeah, this is one of my favorite tricks as well . . .

smagch• 13 years ago

You can also use parallel instead of xargs when files are huge.

keithpeter2• 13 years ago

    history | grep <keyword>

I use this for <keyword> rsync or rdesktop when I have forgotten the long list of options & directories etc

I also use ctrl R <keyword> at the bash prompt but that only gives me the most recent match.

nja2• 13 years ago

Actually, after hitting C-R <keyword>, you can keep pressing C-R to search backwards through the history. Or C-S to search forwards. More info: (bash uses emacs editing mode by default) http://www.gnu.org/software/emacs/manual/html_node/emacs/Bas...

pbhjpbhj• 13 years ago

I also use ctrl R <keyword> at the bash prompt but that only gives me the most recent match. //

Are you sure, I'm using BASH on Kubuntu and I get previous matches by repeated use of ctrl+R.

Actually inspired by a previous post here I made a script "hs" that searches archived history files and shows me all past match lines. Before that I did what you do now.

keithpeter2• 13 years ago

Indeed it does! Excellent

general_failure• 13 years ago

my favorite is apt-file

Toenex2• 13 years ago

Thank you.

dagw2• 13 years ago

GNU Parallel. My new favorite command. I have probably written at least two or three ad-hoc, buggy, feature poor versions of that command before discovering it.

kibwen2• 13 years ago

I recently used xxd to illustrate the concept of text encoding, and put together a little vimscript to make it easier to visualize:

  "Toggle hex edit mode
  nmap <Leader>h :call ToggleHex()<CR>
  
  let g:hex_mode_on = 0
  
  function! ToggleHex()
      if g:hex_mode_on
          execute "%!xxd -r"
          let g:hex_mode_on = 0
      else
          execute "%!xxd"
          let g:hex_mode_on = 1
      endif
  endfunction

Stick this in your .vimrc, type something, use \h to convert it to hex, change a value, then \h to convert it back and observe how the text has changed. Not super useful, but a neat party trick.

wpietri• 13 years ago

A personal favorite: the file command. Through deep magic(5), it can describe the content of a lot of files.

jbaiter• 13 years ago

A great one I found out about recently is 'watch'. Periodically calls a command and refreshes the screen with its stdout.

csense2• 13 years ago

My favorite is the parallel jobs feature of xargs. For example, say you want to run a script you wrote called process-video.sh to do some processing on all the video files in a directory (extracting audio to MP3, converting format, etc.). You want to use all 8 of your cores. You could write a Makefile and run it with -j9, or you can do this:

   find . -name "*.flv" | xargs -n 1 -P 9 ./process-video.sh

This immediately forks 9 instances of process-video.sh on the first 9 .flv files in the current directory, then starts a new instance whenever a running instance completes, so 9 instances are always in flight. (I usually set to number of cores plus one for CPU-bound tasks, hence 9 for my i7 with eight cores [1].)

If you add -print0 to the find command and -0 to the xargs command, it uses null-terminated filenames (which does the right thing when filenames contain whitespace).

[1] Logical cores. Most i7's have four physical cores which become eight logical cores through the magic of hyperthreading.

thret• 13 years ago

May I ask, what do you gain by setting it to +1 cores?

csense2• 13 years ago

Even though CPU is usually the resource that limits throughput for video processing in my experience, each process will presumably do some amount of I/O as well.

If you have 8 cores running 8 jobs, then whenever one of those jobs needs to do I/O you have a core sitting idle while that job waits for the disk. If you have 8 cores running 9 jobs, then all your cores will still be fully utilized when a single job is doing I/O.

I think you actually want to do plus a small percentage, perhaps 5%, so ceil(1.05*N) jobs on N cores. That is, I have a gut feeling that 64 cores doing 65 jobs would still result in underutilization, 67-68 jobs would be better (provided the workload doesn't become I/O-limited with that much CPU power, and provided the number of videos to be converted is still much larger than the number of cores, so you don't run afoul of Amdahl's law).

These are really just rules of thumb based on my gut feelings and mental models of how the system works; it might be fun to actually benchmark it and see if my ideas correspond to reality. (You probably want to reboot first, or a bunch of unrelated I/O, to flush the disk cache.)

whitlock• 13 years ago

If you like xargs, but want more flexibility, I'd highly suggest GNU parallel. Such flexibility includes running jobs on multiple computers, running intensive command using all available CPU's (like xargs -P), and creating unique scripts to handle multiple parameters.

http://www.gnu.org/software/parallel/man.html

dagw2• 13 years ago

Parallel also lets you transparently run stuff on remote servers, automatically handling stuff like copying files back and forth. When I have some heavy ad-hoc data processing to do my new favorite trick is spinning up 50-100 ec2 spot instances, point GNU parallel at them and just fire and forget.

social_monad• 13 years ago

look at: man utf8 man syscalls

... and the best hint imo to process malformed filenames per 'find -print0' is:

find ... -print0 | { while read -d $'\0' fil ... # use variable fil }

Regards

guard-of-terra2• 13 years ago

xmlstarlet is a very nice obscure utility which you will love if you touch moderate amounts of XML.

gnaritas3• 13 years ago

Yes it is!

helloTree2• 13 years ago

My biggest improvement was about finding out about .inputrc and configuring the Bash to use VIM-keybindings which is pretty handy if you are used to the editor. Also the following mapping from ESC to pressing jf via

imap jf <ESC>

was very nice as I find it much more ergonomic. And to use it in Bash I have

set editing-mode vi set keymap vi

$if mode=vi set keymap vi-insert "jf": vi-movement-mode $endif

set show-all-if-ambiguous on

aikinai• 13 years ago

I have my Bash set to vi keybindings, but I had no idea it was possible to remap them. Now I can finally stop hitting escape in Bash too. Thank you so much!

helloTree2• 13 years ago

Yeah it is great if you can use similar commands across applications. Btw. there is also a nice history inside VIM (implemented as a separate buffer) that can be accessed using q: so you can scroll through the history with jfkl. E.g. I use that for accessing files: If you are working on a project you will probably edit similar files. If you use one "programming" vim sessions where you always edit files via :tabe you will find all the recent edited files in your history and using q:/ you can search for them pretty fast. So over time you have automatically a working environment adapted to your current project.

malkia• 13 years ago

I have the bad habit of trying out everything in /usr/bin (linux, OX) and in c:\windows\system32 (and further on, digging Program Files / Applications / etc.).

This is how I discovered myself "fsutil" (Windows) :)

wildmXranat• 13 years ago

Less +F with search highlighting mentioned in comments is my new fave command. I was beholden to 'tail -f', not that it sucked.

mpu• 13 years ago

ssh, come on...

gajomi• 13 years ago

Glad to learn of cal!

cheese1756• 13 years ago

For the ssh section, I'd also include sshfs. It's super handy when you want to use local (or GUI) applications for a file on a server, or just to mount your home computer as a drive while you are away. I personally use it to mount my backup server's drive on my Chromebook, effectively giving myself a 500GB hard drive whenever I need it.

DavidWanjiru• 13 years ago

I'm feeling rather good about myself because although I'm a complete novice at linux, I knew the cal command. Yay me! I read about it here: http://linuxcommand.org/tlcl.php

Kiro• 13 years ago

When do you need man ascii and xxd? I'm a front-end developer so I have no clue about these low-level things but I'm interested and would like to learn.

F30• 13 years ago

There are a lot of usecases for bot of them, mostly debugging I's say. So I'm gonna tell you what I used them for in the past months:

The ASCII table might be useful when you're dealing with encoding problems and want to know e.g. what exactly happened in a broken URL encoding. I used both of them for binary exploitation in the security class I took at university: When utilizing buffer overflows, it's important to get every byte at the right place of your payload. Sometimes, you also need to treat it as a string and that's where ASCII comes in.

These are only two specific usecases. As I said, there are a lot more. Development of binary file parsers could be another.

cmhamill• 13 years ago

This is a very silly use, but I use `ascii` when I can't distinguish between 0 and O in a font ? `ascii 0`, for example, clears that right up.

doktorn• 13 years ago

strings - find the printable strings in a object, or other binary, file. Good command to know of if you want to snoop around in binary files.

ngcazz2• 13 years ago

Brace expansion to create a backup:

    cp {file,.bak}

reduces to

    cp file file.bak

qu4z-22• 13 years ago

Brace expansion is great! My understanding, though, is that you'd need to write:

  cp file{,.bak}

Was it a typo, or is there some feature/special-case I don't know about?

ngcazz2• 13 years ago

Oh, you're right. Thanks. (Can't edit the original comment anymore)

ruok0101• 13 years ago

One little known, but very handy tip:

At any password prompt, if you knowingly mis-keyed your password, rather than pressing return, getting the "your an idiot" message (often after an annoying 1 - 2 second delay), and having to retype the command, you can actually press ctrl-U, which is the "Clear the line" bash command, and start over typing your password again.

rockybutler• 13 years ago

For those of us using emacs-style bash keybinds. For vi-style keybinds, it's something more like "esc dd".

PS if you are a Vim user, you owe it to yourself to try the vim-style keybinds. They make a world of difference.

mncolinlee• 13 years ago

Along with the SSH recommendations, I would include some of the great helpers for ssh agent like keychain.

https://wiki.gentoo.org/wiki/Keychain

Unix Commands I Wish I’d Discovered Years Earlier

Related Stories