Unix tricks

http://mmb.pcb.ub.es/~carlesfe/unix/tricks.txt

by shawndumas • 13 years ago

623 223 13 years ago

mmb.pcb.ub.es

carlesfe14• 13 years ago

So happy to see this on HN! Actually I took most of the tips from unix threads here and r/commandline, I highly recommend that subreddit if you like this kind of tricks!

Edit: also, @climagic: https://twitter.com/climagic

Second edit: I'm enhancing the file with these comments, so if there are any inconsistencies between the txt and a commenter, it's my fault.

adlpz• 13 years ago

Hostiàs! Topopardo! I used to follow your podcasts long ago :D. Anyway, nice list of tips.

ciupicri• 13 years ago

> SMB is better than NFS.

More details would be nice.

carlesfe14• 13 years ago

NFS basically breaks the client computer until the host responds, while SMB detects the I/O error earlier and, at least, doesn't hang the terminal.

Other than that, SMB allows for permissions and a bunch of other features. It is, basically, a more modern and robust protocol. Does NFS shine for some use cases? Indeed. Would it be the first choice for most ones? Nope.

Disclaimer: my sysadmin skills are not so great, I'm talking as a user

EwanToo2• 13 years ago

NFS has 2 mount types, soft and hard.

http://tldp.org/HOWTO/NFS-HOWTO/client.html

Soft mounts report errors immediately, hard mounts hang.

And for permissions, NFS provides everything under the sun that you could possibly need via ACLs. NFSv4 is a very modern protocol, much as SMBv2 is (not SMB though, it's awful).

Linux has had NFSv4 for what, 6 years now, at least, and even v2 and v3 had some limited ACL support?

http://wiki.linux-nfs.org/wiki/index.php/ACLs

easytiger• 13 years ago

i cannot possibly disagree more. Are you using NFS from 1999?

jameswyse• 13 years ago

I've found NFS to use much less cpu than SMB (on raspberry pis running XBMC at least)

cowmix3• 13 years ago

I second this..

If you use any NAS device with a weak CPU and you have a choice of CIFS or NFS, the NFS transfer rates are normally 25% faster and the load is much less on the server.

shrikant• 13 years ago

I third this. I was attempting to set up my Raspberry Pi for sharing a big HDD that was plugged into it on my home network, and the NFS mounts appear to weigh lighter on my vanilla Raspbian setup.

And fwiw, if there's some sort of error with the mount, the terminal lets me know right away.

lobster_johnson• 13 years ago

I believe NFS is a less chatty protocol with fewer roundtrips. SMB is not a particularly well-designed protocol overall.

My only evidence is anecdotal. Back in ~1998 we had Samba running on our small Linux file server using Windows NT desktops via 10mbit Ethernet. It was dog slow, not just for browsing but on sequential things like file transfer. We installed NFS mounts on the same box, and it turned out to be lightning fast. I don't remember just how fast, but it made everyone go "wow".

Nowadays, with NFS over fast 100mbps or gigabit Ethernet the latency difference probably is not significant enough to make a difference. I prefer NFS just because its more Unixy.

gnu8• 13 years ago

1) The caveats of NFS can be crippling if you are a rubbish sysadmin. NFS requires a more thorough understanding than what you can get from a tip sheet.

2) Samba is single threaded, performance will suffer serving SMB from a Linux machine. For this reason it would be better to serve SMB from a Windows machine.

3) Using a foreign protocol between homogenous computers when the native protocol will do is non-ideal. NFS is the right thing when sharing filesystems from Linux to Linux.

duggieawesome• 13 years ago

:set spell is pretty neat. Will use not when coding, but when writing blog posts.

nathan_long• 13 years ago

If you put that in, for example, `~/.vim/ftplugin/markdown.vim`, it will be used on all markdown files. (`../ftdetect/markdown.vim` controls how Vim determines that a file is markdown.)

jamesmiller5• 13 years ago

I turn that on when editing Git commits.

  autocmd FileType gitcommit setlocal spell

niggler4• 13 years ago

1) `pgrep` is a standard utility that does what his `psgrep` does and much much more.

2) htop is a cpu and memory hog -- every time I've used it I noticed it takes 6+% CPU time

3) there's an awk trick to do the `sort | uniq` recommendation that works on 10+GB files (single pass):

    awk '!x[$0]++'

4) Passwordless keys are dangerous -- use ssh-agent to save the password of the keys

vvpan2• 13 years ago

On my computer htop consumes about the same amount of resources as top, just tested.

carlesfe14• 13 years ago

Not trying to contradict you, just some explanations

1. I prefer 'psgrep' because it covers 99% of my use cases for pgrep (ps axuf | grep $NAME)

2. htop is very nice, come on! Would not let it running on the background for hours, but it's nicer than top

3. Note taken, thanks!

4. Is ssh-agent really safer than using passwordless keys? Just asking, I'm curious

niggler4• 13 years ago

1. `man pgrep` is your friend (you save two greps, and TBH the `grep -v grep` should be a hint that there's a better way)

2. In my experience on Debian (granted this was in 2010), there is a noticeable performance difference between `htop` and `top`.

4. ssh-agent stores the password in memory and is erased on reboot. OTOH If you use a passwordless key file, anyone can use it if they have the key.

ominous_prime4• 13 years ago

re: 4: not only is an ssh agent by far safer, but most agents now allow you to set a timeout on a key, so it's not indefinitely saved in memory.

A passwordless key gives anyone with acces to that file, access to the login associated with it. If that file is inadvertently exposed (oops, checked it into github...), any machine you have a login on must be considered compromised.

JoeAcchino• 13 years ago

3) if all the lines of the 10+GB file are actually unique, wouldn't awk keep the whole file in RAM? For files larger than my RAM could this leave my system unresponsive because it's thrashing on swap?

niggler4• 13 years ago

The sort | uniq method literally needs to sort the file and pipe it to uniq, a far more memory-intensive operation than the single-pass awk check. You can write your own hash function in AWK if you think you may overstep memory, but of course you risk hash collisions. It's a tradeoff.

I tried it on a 1U server with 24GB ram a few years ago and found that the sort was thrashing at the 10GB file size while AWK handled it easily

alexkus2• 13 years ago

Yes it will, and a bit (read: a lot) more than 10GB as it needs to store the contents of variable x in a hash table (with the corresponding hash key and value of the counter). There's no other magic way it can 'know' whether a particular line has been seen before. You can't rely on hash keys alone as the hashes aren't guaranteed to be unique.

For files with relatively few duplicates it's going to be a lot slower than sort | uniq.

Trying it on a 128MB file (nowhere near enough time to test a 10GB file) filled with lines of 7 random upper case characters[1] (so hardly any duplicates):-

    $ wc -l x.out
    16777216 x.out
    
    $ time ( sort x.out | uniq ) | wc -l
    16759719

    real    0m17.982s
    user    0m42.575s
    sys     0m0.876s
    
    $ time ( sort -u x.out ) | wc -l
    16759719

    real    0m20.582s
    user    0m43.775s
    sys     0m0.688s

Not much difference between "sort | uniq" and "sort -u".

As for the awk method:-

    $ time awk '!x[$0]++' x.out | wc -l

has been running for more than 20 minutes and still hasn't returned. For that 128MB file the awk process is also using 650MB of memory (according to ps). Will check up on it later (have to go out now).

This Linux machine has ~16GB of memory so the file was going to be completed cached in memory before the first test. All things considered equal the awk method will be roughly O(n) (e.g. linear against file size) and sort/uniq will be O(n log n). So, theoretically, the awk method will eventually surpass the sort method because it's having to do less work (it's only checking for a previously seen key rather than sorting the entire file) but I'm not sure the crossover will be anywhere useful if the file doesn't contain many duplicates.

Repeating it for a file containing lots of duplicates (same 128MB file size but contents are only the 7 letter words consisting of A or B, so only 128 possible entries):-

    $ time awk '!x[$0]++' y.out | wc -l
    128
    
    real    0m1.207s
    user    0m1.192s
    sys     0m0.016s
    
    $ time ( sort y.out | uniq ) | wc -l
    128

    real    0m14.320s
    user    0m31.414s
    sys     0m0.428s
    
    $ time ( sort -u y.out | uniq ) | wc -l
    128
    
    real    0m12.638s
    user    0m30.366s
    sys     0m0.188s

Notice that "sort -u" doesn't do anything clever for files with lots of duplicates.

So awk is much faster for files with lots of duplicates. No great surprises. When I get a chance I'll repeat it for a 1GB file and a 10GB file (with lots of duplicates otherwise the awk version will take far too long).

1. Example contents:-

EPQKHPH DLJCROB WICVGQY MHWTPSR HMPNECN

daGrevis• 13 years ago

Only one Vim tip? I'm disappointed.

Check out this resource!

http://www.rayninfo.co.uk/vimtips.html

carlesfe14• 13 years ago

Great resource! Thanks for the link.

Most of my vim tips are on my .vimrc (my dotfiles here https://github.com/carlesfe/dotfiles/blob/master/.vimrc) and on the links on my homepage: http://mmb.pcb.ub.es/~carlesfe/#programming_h3. The .txt that op linked feels a bit out of context :)

DrinkWater• 13 years ago

Good lord, thats a big list! Thanks!

bramswenson• 13 years ago

SMB is better than NFS in Unix Tricks? Shame Shame Shame. Possibly just wrong. Like the rest though. Thanks!

carlesfe14• 13 years ago

I know, that's controversial... but I've had zero problems with SMB and many ones with NFS. Just my two cents.

appplemac2• 13 years ago

tar czf - . | ssh destination "tar xz"

To pipe all the contents of your current directory (including dotfiles) to the destination machine.

Greetings from LSI-UPC!

carlesfe14• 13 years ago

I use that everyday, but didn't think about adding it to the list. It's on now, thanks!

PS: hey, we're neighbors! :)

deafbybeheading• 13 years ago

Why not rsync?

inyourtenement3• 13 years ago

They're both useful. tar over ssh is very fast for large amounts of data. rsync is only fast if most of the data already exists.

cowmix3• 13 years ago

I would change this to:

tar czf - . | ssh destination "cd /remote/dir; tar xz"

appplemac2• 13 years ago

There is also tar -C /local/dir -czf - . | ssh dest "tar -C /remote/dir -xz"

zengargoyle• 13 years ago

You will one day punch yourself for not being safe.

    tar czf - | ssh destination 'cd /remote/dir && tar xzf -'

Test that `cd` or one day you'll end up extracting in the wrong place.

kbenson3• 13 years ago

I used to use this, because it's awesome that it got all the file types correct (sockets, for example).

Then I learned to love rsync -av. The benefits are many-fold.

nathanstitt3• 13 years ago

Whut? ctrl-r? How have I missed that? No more 'history|grep foo' for me!

btipling• 13 years ago

Also ag is faster and better than grep: https://github.com/ggreer/the_silver_searcher

pavel_lishin2• 13 years ago

Try this in your .inputrc:

    # Bind the up arrow to history search, instead of history step
    "\e[A": history-search-backward
    "\e[B": history-search-forward

No more "^rls" to search for ls in your bash history; just type "ls" and start hitting the up arrow.

dunk0102• 13 years ago

This is the best tip ever.

nathanstitt3• 13 years ago

Thanks for the suggestion. I used to have a very customized .bashrc with nice little things like that, but have decided to stick with standard stuff for things that work off of muscle memory.

I got tired of sshing into a new box and half the things I'd type wouldn't work properly until I remembered to copy my settings files over, which seemed more trouble than it was worth for a short-lived s3 instance.

That's why I was excited to discover ctrl-r. It's a built-in method of searching history that I can remember and it'll work everywhere.

lucastheisen• 13 years ago

If you are using 'set -o vi', then you can just use 'ESC' -> '/<somePartOfCommand>' then use 'n' to iterate in reverse order through the commands that match the vi regex. really powerful and useful.

setrofim_• 13 years ago

> Compile your own version of 'screen' from the git sources. Most versions have a slow scrolling on a vertical split or even no vertical split at all

or just use tmux

tomsthumb• 13 years ago

Just to note, splits in tmux and screen behave differently. iirc, in screen you have a set of splits and fill them in with different ports (kind of how vim thinks of viewports). so technically you can have the same viewport open twice on your monitor and the other will mirror the workings of the one that your are working in right now. in tmux each 'tab'/window is a set of splits, which act more like sub-windows, and don't really share across multiple spaces.

crusso3• 13 years ago

The special bash command I'm most often asked about by shoulder surfers is !$.

It substitutes the last argument in the previous command into the current one.

For example:

  $ ls /some/long/path/somewhere/looking/around/
  <output>
  $ cd !$
  cd /some/long/path/somewhere/looking/around/

yan• 13 years ago

I find ESC, . (esc, then press period) to be more intuitive. (alt+period works as well)

kps2• 13 years ago

An arguably better (and slightly more portable) way to accomplish this is the special variable $_ , which expands to the last argument of the previous command. Since $_ is a variable rather than a history substitution, it still works when there is no command history (e.g. in scripts) and allows all the usual variable expansion forms, for example ${_##*/} to extract the last path component.

crusso3• 13 years ago

That's good to know, although !$ is easier to type since you only need to depress the right shift key, whereas yours requires a quick shift on the opposite side. Makes a big difference when you're trying to quickly type "rm -rf !$" as root. :)

vilgax• 13 years ago

Better than that for me is "Alt+.", much easier to type. It can be also combined with number like "Alt+2+." will insert second argument. Similarly "Alt+0+." will insert last command. http://linuxcommando.blogspot.in/2009/05/more-on-inserting-a...

micampe• 13 years ago

If you are on OS X and use Terminal.app “Alt-.” won’t work because Alt is used for alternate characters. You have two options: enable “use option as meta” in the app settings (but you lose the extra characters) or use “Esc-.” instead.

Yes, I know about iTerm. I don’t want it.

philsnow5• 13 years ago

In my bash/readline/whatever, Alt+<N>+. gives the Nth-from-the-end argument to the previous command. so,

    $ echo a b c d
    a b c d
    $ echo # pressing <Alt+<2>+.> here inserts 'c', not 'b'.

nicholassmith• 13 years ago

I like that one and `!!` for the inevitable point I've forgotten to put sudo in before a command.

nbouscal• 13 years ago

I find myself typing `sudo !!` on a daily basis. You'd think I'd eventually learn to remember to type sudo the first time, but nope.

LandoCalrissian• 13 years ago

That is really a good one, thanks.

jevinskie• 13 years ago

Is there one for all but the first argument?

ampersandy• 13 years ago

!!:2-$

globalpanic• 13 years ago

find . -name "file-wildcard" -exec "string" {} ";" -print

is something I use a lot - plus xargs sometimes

martinced• 13 years ago

I use variations on "find" so often that I've created several little commands "fij" (find in any java source), "fit" (find in any text / org file), "fix" (find in any XML file) etc. which I use all the time

prakashk• 13 years ago

You might want try `ack` then. (http://betterthangrep.com/).

On Ubuntu/Debian systems, it is packaged as `ack-grep`.

Symmetry• 13 years ago

Lots of good tricks, though I replaced a lot of them with "Use fish instead of bash" http://ridiculousfish.com/shell/

bradbeattie• 13 years ago

Of similar note, http://www.commandlinefu.com/commands/browse/sort-by-votes indexes a good number of these tricks.

firebones• 13 years ago

Wanted: a lint for your history that analyzes your commandline usage, suggesting these types of tips based on your historical use.

  history | commandlint

ninetax• 13 years ago

Where can I find commandlint?

vvpan2• 13 years ago

That's a hypothetical program (note "Wanted").

implicit_cast• 13 years ago

Bash and zsh support a surprising amount of emacs editing functionality. Cursor navigation (M-b, M-f, C-a, C-e), text selection (C-space), copy/paste (C-w, M-w, C-y, M-y), and undo (C-_).

jofel• 13 years ago

Instead of

  find . -type d -exec chmod g+x {} \;'

you can usually use

  chmod -R g+X .

which gives additionally the group execute permission to files which have already user/everyone execute permission.

jff• 13 years ago

His example changes permissions for directories only, which I believe your example does not.

tedchs• 13 years ago

Yes, but the find command will only affect directories (-type d).

jcr3• 13 years ago

This is bad juju.

  'find . -type d -exec chmod g+x {} \;'

If you happen to have a malicious directory named (without double quotes):

  ".;sudo rm -rf /"

You'd be stuffed.

It's better practice to use the '-print0' flag of find(1) and pipe the result into xargs(1) with the '-0' flag set. For safety, it's best to also use '-r #' for expected number of arguments, '-n' to stop xargs from running without arguments, and "-J %" so you can use quoting.

  find . -type d -print0 | xargs -0 -n -r 1 -J % chmod g+x "%"

I believe some on implementations '-n' is unnecessary when '-r #' is specified, but on other implementations it's is the maximum.

EDIT: thanks for the vim tip on for finding spelling mistakes!

neoteric3• 13 years ago

This is simply not true:

    $ mkdir '; echo woops'
    $ find . -type d -exec echo {} ';'
    .
    ./; echo woops

As you can see 'woops' is never echoed.

EDIT: The reason being the shell is never involved in this process, and the shell is what is responsible for splitting commands on semicolons/newlines.

jcr3• 13 years ago

You are assuming the exact versions of the shell and find programs that you use are the only ones that exist. It may not be a problem on your exact system, but it can be a problem elsewhere.

mixedbit• 13 years ago

ack-grep is great for searching source code

sepeth• 13 years ago

Yesterday, I discovered the_silver_searcher:

https://github.com/ggreer/the_silver_searcher

I think you might like it.

kibwen• 13 years ago

- 'cd -' change to the previous directory you were working on

To my surprise, this also works with git:

  git checkout -  # to checkout the previous branch

VeXocide• 13 years ago

Instead of

    ProxyCommand ssh -T host1 'nc %h %p'

you can use

    ProxyCommand ssh -W %h:%p host1

which uses ssh itself and therefor also works on machines where netcat isn't installed.

carlesfe14• 13 years ago

Hi, I tried that but my ssh version doesn't have the -W flag. What would you suggest?

argarg2• 13 years ago

Unfortunately OSX uses a badly outdated openssh version just like many other unix tools and it doesn't support the -W option. You could try upgrading openssh using homebrew if you want.

Ensorceled2• 13 years ago

Weird, I've been using vi/elvis/vim/MacVim as my primary editor since 1984 and I hate vi key bindings for shell; I always use the emacs bindings.

AaronBBrown• 13 years ago

I'm the opposite. Every time I log into a server that doesn't have my bashrc, I immediately have to `set -o vi` or else I'm useless.

Ensorceled2• 13 years ago

That's why I love *nix ... options!

growt• 13 years ago

If I may add a trick:

ctrl-z - stops a program

bg - sends the stopped program to the background

fg - gets the program back to the foreground (interactive mode)

very useful in editor sessions or when you want to get rid of the endless download/scp that is blocking your terminal

sp332• 13 years ago

You can also use "jobs" to see background and stopped tasks. If there are multiple jobs, you can use "fg 2" to foreground the second job on the list, etc.

niggler4• 13 years ago

Note that each job gets an identifier (which you can see by running `jobs`). Other commands like `kill` can work with the id number by using %[id]. For example:

    $ some_command
    ^Z (hit control z)
    $ some_command_2
    ^Z (hit control z)
    $ jobs
    [1]-  Stopped                 some_command
    [2]+  Stopped                 some_command_2
    $ kill %1
    $ jobs
    [1]-  Terminated: 15          some_command
    [2]+  Stopped                 some_command_2

dminor• 13 years ago

Way back in my college days, one of the student admins took down the CS department's server by forgetting the % and killing process 1 by mistake.

ottbot• 13 years ago

Another nice job management tool is disown, which lets you log out of your session without killing the job (similar to starting the command with nohup)

mpyne• 13 years ago

And also, you can pass the -h flag to disown to leave the job in the job control table (i.e. you can still bring it to the foreground/suspend it/etc.), but still skip sending the SIGHUP on logout.

signa11• 13 years ago

i normally have this :

fork() { (setsid "$@" &); }

in myzshrc, and then start stuff with :

fork firefox

philsnow5• 13 years ago

I have my terminal emulator configured to set the URGENT ICCCM hint when it sees a bell character. When I realize a command I just entered is going to take a while, and I want to go look at something else on another workspace, I do this:

    $ alias b='echo -e "\a"'

    $ long_running_thing
    ^Z
    $ fg ; b
    [long_running_thing resumes]

Then when the job completes, the terminal bell will ring and my window manager will get my attention.

etrain• 13 years ago

His last trick - compressed fie transfer without intermediate state:

  'tar cz folder/ | ssh server "tar xz"'

Can be pulled off with two flags to scp - and you get to see progress as a benefit!

  scp -Cr folder server:dest/

kbenson3• 13 years ago

rsync -av folder $USER@$SERVER:/destination/path.

scp won't copy certain file types correctly. Rsync really is better here, and not just because of that.

stevvooe3• 13 years ago

You need a -z to activate compression. But, nonetheless, please always use rsync when copying host to host.

cowmix3• 13 years ago

I don't think your method preserves ownership, file perms, etc.

ominous_prime4• 13 years ago

Tar can transfer more filetypes and attributes than scp can (even using -p option). `scp -p` only transfers mode, mtime and atime; you lose ownership, extended atributes, symlinks, and hardlinks.

You will also get better compression with tar (or rsync), as it is compressing the files directly, and not just the ssh stream (-C is just passed on to ssh).

I did the tests years ago, but a quick google found someone who tried to test the various combinations: http://www.spikelab.org/transfer-largedata-scp-tarssh-tarnc-...

jerf3• 13 years ago

In particular, scp is mindblowingly slow on lots of small files. I independently rediscovered the tar-pipe trick while sitting there watching scp laboriously copy thousands of 100-bytes so slowly I could count the files as they went by. That should not be possible, even at modem speeds. Fine for moving one file, OK for directories of very large files, not suitable for general usage where you might encounter a significant number of smaller files.

maw• 13 years ago

If security isn't a big consideration (read: you control both machines and the network), you go even faster with netcat.

On the receiving machine, in its destination directory:

    nc -l 6789 | tar xvf -

And on the sending machine, from its source directory:

    tar cvf - . | nc receiving-machine 6789

netcat varies a bit from distro to distro, so you may need to adjust these command lines a bit to get it to work.

dspillett2• 13 years ago

Add pv (available in many standard repos these days, from http://www.ivarch.com/programs/pv.shtml if not) into the mix and you get a handy progress bar too.

65a• 13 years ago

Here's one it took me a while to figure out...

Hung SSH session (such as wifi out of range)?

Type Return-Tilde-Period

philsnow5• 13 years ago

Also, if you've done this:

    you@somehost:~$ ssh otherhost
    you@otherhost:~$ some_command

If you hit ^Z right now, it will tell the shell on otherhost to stop some_command and give you the shell prompt on otherhost back.

If instead you wanted to stop the ssh process and get the shell prompt on somehost, hit Tilde ^Z (you don't have to hit a new Return, but ssh only notices these escape sequences after a Return).

Also if you use ControlMaster and have a few xterms open all with sshs to otherhost, and then you exit the first ssh you happened to have open, it will seem to hang and not give you your prompt back. What's happening is that that ssh process is the "control master" and it's still open because you've got other sshs to the same host open. Hit Tilde & to background the ssh process and get your terminal back.

Yes, you could also hit Tilde ^Z and then 'bg' the ssh process.

init0• 13 years ago

Md version for readabilty https://gist.github.com/hemanth/5109020

mikegioia2• 13 years ago

'ssh -R 12345:localhost:22 server.com "sleep 1000; exit"' forwards server.com's port 12345 to your local ssh port, even if you machine is not externally visible on the net.

This one blew my mind

malloc2x3• 13 years ago

Better to use, e.g.:

  ssh -fN -o ServerAliveInterval="240" -R 2222:localhost:22 example.com

(And on example.com ssh -p 2222 localhost)

This lets you easily keep the tunnel open long-term. Why?

  -f    Background the ssh process (don't need nohup)
  -N    Don't run any command
  -o... Make ssh do the work of keeping the session alive forever

jackalope2• 13 years ago

The ServerAliveInterval option might also help with another issue mentioned in the post:

'sshfs_mount' is not really stable, any network failure will be troublesome'

To avoid trouble with remote backups and other long running processes, I always add the following to the end of ~/.ssh/config or /etc/ssh_config:

    Host *
        ServerAliveInterval 300

carlesfe14• 13 years ago

But the remote host can limit your ServerAliveInterval, however most hosts don't close your session if there is something running. On our clusters this is the only working solution, and I tried both, trust me.

mikegioia2• 13 years ago

This is excellent, thank you!

jonstjohn• 13 years ago

Thanks for calling that one out - had never tried that before and works great. Putting into my bag of tricks ...

gcv• 13 years ago

Learn to use your shell's globbing features instead of overusing find. In zsh, you can do 'print -l /*.(c|cc|h|hh)' for example (I'm sure bash has an equivalent).

oftenwrong2• 13 years ago

For me, the glob gets a lot of use.

oftenwrong2• 13 years ago

    ** glob

aw3c22• 13 years ago

does that work for say 2143789 files?

klibertp2• 13 years ago

I actually hit a limit quite a few times already - I don't remember if it was the shell that complained or the command (mv, cp, ...) itself, but I know I couldn't execute the command. find with xargs or -exec worked, however.

unfletch• 13 years ago

    '!!:n' selects the nth argument of the last command, and '!$' the last arg

A lot of people know about "!$" (which is shorthand for !!:$), but that's just the tip of Bash's history expansion. I use these things all the time. One of my favorite keystroke savers is adding :h, the head modifier, to !$. For example:

    $ cp file.txt /some/annoyingly/deep/target/directory/other.txt
    $ cd !$:h
    $ pwd # => /some/annoyingly/deep/target/directory

Once you understand how each component works it's easier to put them together into new (to you) combinations. For example, once you know that !$ is shorthand for !!:$, it's not a huge leap to reason out that you can use !-2:$ to get the last argument to the 2nd-to-last command. Or !ls:$ for the last arg to the most recent `ls` command.

I also prefer to do substitution with the :s modifier rather than ^ as suggested at the link, for consistency's sake:

    $ echo "foo bar"
    foo bar
    $ echo !!:s/bar/baz
    foo baz
    $ echo !?bar?:s/foo/qux
    qux bar

Relevant Bash manual pages:

http://www.gnu.org/software/bash/manual/html_node/Event-Desi...

http://www.gnu.org/software/bash/manual/html_node/Word-Desig...

http://www.gnu.org/software/bash/manual/html_node/Modifiers....

stakecounter• 13 years ago

That first example is great, thanks. I run into that all the time. Up until now I've been doing this:

$ cp file.txt /some/annoyingly/deep/target/directory/other.txt

$ cd !$ (No such file or directory)

(up arrow and then backspace to directory)

cschol2• 13 years ago

Alt + . – use the last word of the previous command.

    $ cp file.txt /some/annoyingly/deep/target/directory/other.txt
    $ cd then press Alt + .
    $ pwd # => /some/annoyingly/deep/target/directory

ralph3• 13 years ago

No, that gives the last word, as you say, we want the dirname of the last word.

    $ echo foo/bar
    foo/bar
    $ echo !$ !$:h
    foo/bar foo

pjungwir• 13 years ago

Another nice ending is `:p` to print the command instead of executing it. I use this if I'm doing something complicated and I want to make sure it's right. Or if I'm saying `!-n:foo` with n>2. Then just up-arrow and enter to run it for real.

kejadlen• 13 years ago

I use magic-space for that in my inputrc:

  $if Bash
    Space: magic-space
  $endif

Basically does the same thing as :p, but after a space instead of enter.

Edit: fixed formatting.

cawel• 13 years ago

Here is a detail when using '!!:n' with ':p' when trying to iteratively construct a complex command. I want to emphasize the use of the up-arrow (which will show the interpolated arguments), as opposed to rewriting exactly what you wrote in the previous command, since the usage of ':p' will be interpreted by the shell as a command in itself:

    $ echo a b c d
    a b c d
    $ echo !!:2:p
    echo b
    $ echo !!:2
    -bash: :2: bad word specifier # there was only 1 argument in last command

However:

    $ echo a b c d
    a b c d
    $ echo !!:2:p
    echo b
    $
    <up-arrow pressed once will give the following prompt> 
    $ echo b

absqua• 13 years ago

  shopt -s histverify

to show the expanded command before executing it. Then just enter. I never get these right the first time.

sciurus• 13 years ago

A sometimes handy addition to the s modifier is g, which replaces all instances of the pattern instead of just the first.

    $ echo "foo foo"
    foo foo
    $ echo !!:s/foo/bar
    echo echo "bar foo"
    echo bar foo

    $ echo "foo foo"
    foo foo
    $ echo !!:gs/foo/bar
    echo echo "bar bar"
    echo bar bar

hiroaki• 13 years ago

  $ !!

runs the previous command.

This is especially useful:

  $ !!
  $ sudo !!

DanHulton• 13 years ago

sudo !!

Perform the previous command as root. Great if you continually forget which of your scripts need to be run as root and which don't.

rburhum2• 13 years ago

I freaked out when I found out about the "screen" command several years back. "screen" starts a virtual screen that you can detach from with "ctrl-a d" and you can log out, login from a different machine/session and reattach with "screen -r". it has history so you can run long running commands and reattach 3 days later to continue from where you left off as if you had been logged in the whole time.

dirkk0• 13 years ago

and you can do things like

  "screen -X -S minecraftserver -p 0 -X stuff "say test $(printf '\r')"

base698• 13 years ago

Tmux is the new screen. tmux allows for emacs or vim key bindings to run around the buffer and search, splits horizontal and vertical and easier configuration: http://tmux.sourceforge.net/

revscat• 13 years ago

Yup! You might want to try tmux, though. I used screen for many years and switched over to tmux about 2 years ago, and have been really happy with it. It (tmux) is also maintained more regularly now; screen development seems to have stagnated.

Either way, enjoy!

rburhum2• 13 years ago

I will checkout tmux. Thank you!

re_todd• 13 years ago

Tmux is similar, here is a cheat sheet that helped me move from screen to tmux: http://www.dayid.org/os/notes/tm.html

artursapek• 13 years ago

Regarding set -o vi, which I love, is there a way to have it load my vimrc as well? I have custom bindings I would love to have.

McUsr• 13 years ago

This was a nice very timely page sitting with my private repo in mercurial, and the other one at github…

I discovered help <builtin> some months ago, and that was a great boon really. Like help "test" so I didn't have to go the rather large bash man page.

Here is a little shellscript for displaying a man page on Mac Os X (gman). (If you then click on on of the links on the man-page, it may pop up in your default browser).

  #!/bin/bash
  if [ $# -lt 1 ] ; then
  	echo "gman takes a man page, if found and formats it into html."
  	echo "Usage: gman [manfile]"
  	exit 2
  fi
  a=`man -aw $* |head -1`
  if test  x$a = x ; then
  	echo "Can't find $1"
  	exit 1
  fi
  # Figures out if it is a normal man page or something else (gz).
  b=`man -aw $* |head -1 |grep "gz"`
  echo $b
  if test  x$b = x ; then
  	groff -man $a -Thtml >|/tmp/tmp.html
  else
  	gzcat $b |groff -man -Thtml >|/tmp/tmp.html
  fi
  qlmanage -p /tmp/tmp.html >/dev/null 2&>1

D9u• 13 years ago

- Use 'apt-file' to see which package provides that file you're missing

Ummm, "apt-" isn't a "Unix trick..." It's specific to linux distros which use the "Aptitude" package manager.

Linux != Unix

iso-8859-1• 13 years ago

Aptitude is just a front-end to APT. And APT (package tool) is just an interface to dpkg or RPM (package managers).

siavosh• 13 years ago

Is there a command/short-cut to storing the path(s) returned from a find command? Currently I do something like:

find . -name somefile.txt -print

Then copy and paste the results manually into a cd command for example. I feel like there's a much better way somewhere.

ralph3• 13 years ago

cd `find -name foo`

Tyr42• 13 years ago

xargs?

Here is an example from the man page: >For example, the following command will copy the list of >files and directories which start with > an uppercase letter in the current directory >to destdir:

                   /bin/ls -1d [A-Z]* | xargs -J % cp -rp % destdir

You can use this with find really nicely.

find . -name somefile.txt -print0 | xargs -0 -J % cp -rp % destdir

Note the -print0 and -0 flags (zero). This will use a null byte as a separator instead of spaces to avoid failing on files with spaces in their names.

FiloSottile• 13 years ago

    * Read on 'ssh-keygen' to avoid typing passwords every time you ssh

He meant ssh-agent?

jabo• 13 years ago

No he did mean ssh-keygen which generates a public and private key pair for you. Run ssh-keygen on your local machine and copy the public key to ~/.ssh/authorized_keys and you'll be able to login to the server without a password using ssh -i /path/to/private/key user@host

LiveTheDream• 13 years ago

That approach is insecure, however, because anyone with the private key now has access. When running ssh-keygen, you should add a passphrase to the key, then add the key to ssh-agent so you don't need a password for the account, nor do you need to type the key's passphrase constantly.

woodchuck642• 13 years ago

zsh: setopt extendedglob allows you to use numeric ranges on globbing with < >:

mv p1080<100-300>.jpg folderx/

to move p1080100.jpg through p1080300.jpg to a new folder.

Still not available on bash or more common shells?

LukeShu2• 13 years ago

Bash:

    mv p1080{100..300}.jpg folderx/

1amzave• 13 years ago

Though that's not strictly a glob, but brace expansion (it will expand to all the numbers in that range regardless of whether or not files with those names exist). bash does have 'shopt -s extglob', which enables a number of useful globbing extensions, though I don't believe there are any numeric ones among them.

woodchuck642• 13 years ago

TIL, thank you. Here it is from the manpage:

A sequence expression takes the form {x..y[..incr]}, where x and y are either integers or single characters, and incr, an optional increment, is an integer. When integers are supplied, the expression expands to each number between x and y, inclusive. Supplied integers may be pre‐fixed with 0 to force each term to have the same width. When either x or y begins with a zero, the shell attempts to force all generated terms to contain the same number of digits, zero-padding where necessary. When characters are supplied, the expression expands to each character lexicographically between x and y, inclusive. Note that both x and y must be of the same type. When the increment is supplied, it is used as the difference between each term. The default increment is 1 or -1 as appropriate.

dunk0102• 13 years ago

Here's a really great presentation I found (probably on HN) some years ago. http://www.ukuug.org/events/linux2003/papers/bash_tips

All extremely useful, my favourite being the .inputrc rebindings of up and down to search history. Takes a little getting used to but is great once you are (good luck to anyone else trying to use your terminal though ;))

artagnon• 13 years ago

Feels very outdated.

1. Use zsh, not bash. AUTO_PUSHD, CORRECT_ALL and tons of other options make some tricks redundant. Also, the zle M-n and M-p are more useful than C-r imo.

2. Use tmux, not screen.

3. Use z (https://github.com/rupa/z), not j.py.

4. Use cron, not at. Or even systemd timer units, if you're so inclined.

5. Use public-key authentication and keychain, not password-based SSH.

6. Don't send emails from command-line naively; you have no control over the headers. Use git-send-email or similar.

7. Consider using something slightly more sophisticated than Python's SimpleHTTPServer to share files/ folders. One example: woof (http://www.home.unix-ag.org/simon/woof.html)

laichzeit0• 13 years ago

I take it you don't work on many disparate unix systems on a daily basis :) I find most of the time I'm lucky if there's even bash installed on the remote system, it's usually ksh. tmux? way too new. Python? Nope. Perl is the only scripting language I'd wage my balls on beyond awk if I hope to reuse the script again.

Sad, but I find this is generally the case in extremely large enterprises where there is a mix of AIX, HPUX, Linux and Solaris being used due to years of weird procurement decisions. Sigh.

carlesfe14• 13 years ago

Yes, that's exactly why those tricks "feel outdated". It's because they're made to run on systems from the early 2000s

rcfox• 13 years ago

Isn't at just a cron helper for one-off jobs?

emidln2• 13 years ago

Sadly this isn't usually the case

philsnow5• 13 years ago

I've given tmux two separate tries and both times it had screen corruption issues with curses apps. screen is tried (and tried, and tried) and true.

I've also invested enough time in learning bash over the years that zsh is not a net-win for me. I've switched to using it on some systems but I seldom use more than what's available in bash. I would switch back to bash on these systems for consistency, but I feel like I've already sunk too much time in to this experiment of using zsh.

don't be a zsh/tmux hipster; there's nothing wrong with "good enough". this lesson has played out several times as businesses / software with arguably better execution / implementation loses out to existing players that have been around a while.

Edit: though, if you're relatively new and haven't been using bash/screen/whatever for decades, I don't think anybody would call you a hipster for using zsh/tmux instead of bash/screen. The marginal utility of learning tmux or zsh is much higher for somebody who hasn't already used other stuff forever.

klibertp2• 13 years ago

I would recommend fasd (https://github.com/clvv/fasd) as a more feature complete alternative to z.

xiaoma• 13 years ago

One of my new favorites-- use ctrl-r autocompletion to grab a command close to what I want from history, tack on a character that breaks it if necessary and run that.

Then use fc to get that command back in a vi editing context, change what I want and :wq

The resulting command is immediately executed. This process can be really fast compared to manually constructing long piped commands.

networked• 13 years ago

Related earlier discussions:

https://news.ycombinator.com/item?id=5022457

https://news.ycombinator.com/item?id=4481234

A pretty good thread on Reddit:

http://www.reddit.com/r/linux/comments/mi80x/give_me_that_on...

Edit: see https://news.ycombinator.com/item?id=3257393 for a discussion of the above thread.

While we're at it, consider using weborf [1] as an alternative to Python's SimpleHTTPServer for simple file sharing. I found it able to saturate a gigabit ethernet connection when hosted on a Core 2 Duo ULV laptop with an SSD.

[1] See http://galileo.dmi.unict.it/wiki/weborf/doku.php?id=start. It's available from the official repos in Debian and Ubuntu with

  sudo apt-get install weborf

Invoking it is dead simple:

  weborf -b ~/dir-to-share

dfc• 13 years ago

RE: weborf

I will have to look at weborf. I have always wished debian packaged publicfile, similar to djbdns or even dbndns. As it is I am still looking for a "djbdns-like http server" that is apt-get installable and actively maintained.

I will never understand why gnome-user-share depends on apache...

keenerd• 13 years ago

Consider webfs too. It is even simpler than weborf.

http://packages.debian.org/search?keywords=webfs

hackerpolicy• 13 years ago

If you work with something like this

  /this/is/some/very/nested/directory/

and you need to move to almost the same structure

  /this/be/some/very/nested/directory/

then you might benefit from

  function bcd {
    cd ${PWD/$1/$2}
  }

jbp• 13 years ago

As an emacs user I map "C-p" "C-n" for history search in bash

In .inputrc

  "\C-p": history-search-backward
  "\C-n": history-search-forward

Also I found the following settings to be very useful:

  # List the possible completions when Tab is pressed
  set show-all-if-ambiguous on
  #tab complete without needing to get the case right
  set completion-ignore-case on

marekrud• 13 years ago

GNU parallel: easily substitute of file extensions + parallel execution, e.g.

    ls *.png | parallel convert {} {.}.jpg

emmelaich• 13 years ago

I see people often have aliases for ls -ltr | tail but ls -lt | head might be a lot faster with a lot of files.

runejuhl• 13 years ago

When creating excessively long oneliners (you know, the kind that should actually be a script, because you know you're going to find a use for it in a few weeks), the following key combo is golden:

    ^x ^e

It opens up the exported EDITOR with a tmp file containing whatever is on the command line.

Using SSH, especially on campus/in a train/other places where a wifi connection doesn't long, mosh is really a godsend. In places where mosh isn't practical or available, the following combos are really good to know:

    <RETURN> ~ .    # end ssh connection
    <RETURN> ~ ?    # show available commands

Paired with autossh which will reconnect by itself, it really takes the pain out of traveling while doing remote work.

Oh, and when you need to know what the decimal value of 0x65433 is, it's good to know that bash can do stuff like that:

    $ echo $((16#65433))
    414771

Reading the bash man page is not a bad idea in itself...

andr3w321• 13 years ago

find . -name "*" | xargs grep "hello" 2>/dev/null

Searches all the files in current directory and all subdirectories for files that contain "hello". Add -l option to grep to only display the filenames instead of filename and match.

marvwhere• 13 years ago

i'm working with unix since years, but only on my server, and now for a little bit more then a year now on osx for 2 days a week.

and since one month i have my first own macbook. totally helpful to get in touch with some magic in the console.

thank to everybody, who makes my working life easier =)

SEJeff• 13 years ago

I can't believe this past has sort | uniq when GNU sort (important distinction, the BSD version and hence OSX can't) has a -u flag so sort -u == sort | uniq

lri• 13 years ago

OS X does have sort -u. -u is also in POSIX (http://pubs.opengroup.org/onlinepubs/9699919799/utilities/so...).

nonamegiven2• 13 years ago

"Add "set -o vi" in your ~/.bashrc to make use the vi keybindings instead of the Emacs ones." Better to do this kind of thing in .inputrc, as:

set editing-mode vi

(or set editing-mode emacs) because any application that uses readline gets to use those settings. So for example you get command line editing in various command line apps. bash uses readline, so you'll get that. The python repl will give command line editing with .inputrc set as above.

  $ apt-rdepends -r libreadline6 |egrep -ve "^ " |wc -l
  8995

psql (postgresql) and mysql (mysql) are really handy with command line editing.

"'ctrl-x ctrl-e' opens an editor to work with long or complex command lines"

If you've set -o vi, or set editing-mode vi in .inputrc, then on a command line type:

  esc-v

(Escape key to get out of insert mode, then the 'v' key) That will open a full vim session for editing your complex command line.

:wq

exits vim and gives your command to bash to execute.

carlesfe14• 13 years ago

Oh, wow, you just made my day. I love vim-style line-editing.

james2vegas• 13 years ago

'set -o vi' is more likely to something that isn't bound up tight with readline, like /bin/sh

codygman• 13 years ago

s/wq/x/g

nonamegiven2• 13 years ago

Duly noted. :)

rogerd• 13 years ago

I really could have used that "chmod g+X * -R" the other day! Thanks for the tips.

dmackerman• 13 years ago

Did a quick and dirty format for easy reading: https://gist.github.com/dmackerman/5117156

UNIXgod• 13 years ago

% :(){ :|:& };:

grn• 13 years ago

Personally I put the following in my .bashrc:

    function pushcd {
        if [ $# -eq 0 ]; then
            pushd ~ > /dev/null
        else
            pushd "$@" > /dev/null
        fi
    }

    alias cd='pushcd'
    alias b='popd > /dev/null'

Then cd saves the history of visited directories and b navigates backwards, e.g.:

    ~$ cd /tmp
    /tmp$ cd /usr
    /usr$ b
    /tmp$ b
    ~$

Unix tricks

Related Stories