Shell Productivity Tips and Tricks

https://blog.balthazar-rouberol.com/shell-productivity-tips-and-tricks.html

by pcr910303 • 6 years ago

353 95 6 years ago

blog.balthazar-rouberol.com

andreareina• 6 years ago

Process substitution deserves a mention: http://www.tldp.org/LDP/abs/html/process-sub.html

tcoff91• 6 years ago

fzf fizzy history search with ctrl-r is my favorite new shell trick. Fzf is a wonderful program

gimboland2• 6 years ago

fzf's great. I use it for a quick/easy git branch menu, complete with preview of recent history of each branch: https://pastebin.com/R91bP2qa

rwnspace• 6 years ago

I too prefer fizzy history. It's like Sodastream for your shell!

Skim [0] is a nice alternative to fzf [1], mainly for the fact you can call additional commands dynamically like so [2]:

    sk --ansi -i -c 'rg --color=always --line-number "{}"'

[0] https://github.com/lotabout/skim [1] https://github.com/lotabout/skim#difference-to-fzf [2] https://github.com/lotabout/skim#interactive-mode

ptzz• 6 years ago

Would that correspond to piping the output of the additional command to fzf?

lcnmrn• 6 years ago

I like fd more. fish already has autocomplete.

raykooyenga• 6 years ago

I max out my history sizes as well as have bash functions to dump/persist them to time stamped archives on command or on logout.

saagarjha2• 6 years ago

(That’s

  export HISTSIZE=
  export HISTFILESIZE=

for those wondering how to do it.)

xpuente• 6 years ago

xargs should be there. The most useful tool IMO.

based22• 6 years ago

https://en.wikipedia.org/wiki/Xargs

JNRowe4• 6 years ago

As zsh is mentioned at various points throughout the article I'll add that zargs¹ is often a great alternative to xargs. Combined with zsh's awesome globbing and filtering functionality it is often far nicer to work with than find.

1. https://github.com/zsh-users/zsh/blob/master/Functions/Misc/... or zshcontrib(1)

jerzyt• 6 years ago

I think zsh is the best shell ever, unfortunately, I've reverted to bash, because it's too much hassle to install zsh everywhere I work. zsh globbing is on another level.

lolive• 6 years ago

Whoever can point me at a robust method to save/restore bash histories of a tmux session would save my life!!! [And yes i use tmux inside tmux]

Klasiaster• 6 years ago

I use this to have all shells share a history concurrently - which sometimes is confusing if you press "↑" and have an unexpected (possibly dangerous command) but I'm used to it now:

  shopt -s histappend
  PROMPT_COMMAND="history -a; history -c; history -r"

See more about it here: https://unix.stackexchange.com/questions/1288/preserve-bash-...

notyourday• 6 years ago

About a year ago I went further into this rabbit hole and right now I think I am on a re-iteration of a system I have not changed in about 4 months. It came to me after watching too much sci-fi that had clicky-clicky-typie-typie instant recall.

It occurred to me that every hack around history just nibbles at the edges probably because all of them are bolted on a design when the systems were slow. Systems now are fast and have good connectivity ( at least to themselves ) hence:

"All logins across all systems that have the same security domain share single common command history."

I use bash but it should be adoptable to anything. Here's how it works:

1. There's a database. I use redis because it is fast and has SETNX. [I may switch this to a redis queue to be able to archive commands as well.]

2. Using PROMPT_COMMAND variable I execute a function that pulls the last executed command using "history 1" and pushes it into the database. SETNX ensures that only a command that has not been seen before is stored. In the same transaction I trim the number of commands stored in the database to 5000.

3. Ctrl-R is bound to fetching last 5000 commands from the database, piping it into "peco" which I use with the selection line on a top ("fzf" does not support the top line so I don't use it here. "peco" does not support certain header skips/treatments, so i use "fzf" in other places )

4. On a failure to push into the database, system does nothing as the data is in a local history.

5. On a failure to pull from the database, system feeds the result of the local history into the "peco".

This system gives me ability to access the same command line history across my laptops, workstation, VMs, random Linux devices that i have running at home. Our ops-team uses it across all the monitoring workstations

bhaak2• 6 years ago

I'm using a variant of this: https://spin.atomicobject.com/2016/05/28/log-bash-history/

Logging the entire bash history in files sorted by day: export PROMPT_COMMAND='if [ "$(id -u)" -ne 0 ]; then echo "$(date "+%Y-%m-%d.%H:%M:%S") $(pwd) $(history 1)" >> ~/.logs/bash-history-$(date "+%Y-%m-%d").log; fi'

I haven't overloaded ctrl-r yet to search in these files but that would be a possibility.

kortex4• 6 years ago

Histdb is amazing, but requires zsh. I have one continuous history. I'm working on figuring out multi-host history for One True History.

https://github.com/larkery/zsh-histdb

I haven't tried it, but the keywords you are looking for is bash history db/sqlite. E.g.:

https://github.com/thenewwazoo/bash-history-sqlite/blob/mast...

based22• 6 years ago

vi :x

grep -n ... vi +§

jpxw• 6 years ago

If you want to remove a sensitive command from your history, you can simply edit your $HISTFILE history file and remove it.

It’s already too late at this point. Anybody on your machine can read the commands you are running, for example with ps. You should instinctively avoid entering anything in plaintext into a terminal which you don’t want other people to see. Any command line tool worth its salt will provide alternative ways of passing secrets (using configuration files for example).

cassiet• 6 years ago

How exactly?

jedimastert3• 6 years ago

I think what OP meant was closer to "if it could have been entered into the bash history than there's other ways it could be seen".

Most CLI programs that need sensetive information as input should either do it interactively (a la sudo), as standard input, or configuration file. If worst comes to very worst, you can put the sensitive info in a text file and use backticks. For example:

  some-command --username=jedimastert --password=`cat secret.txt`

tyingq• 6 years ago

Not a real solution, but you can assign new values to argv/argc to hide things from ps.

With Perl, for example, assign to $0.

$0="what /bin/ps will see";

rrmm• 6 years ago

The worst is when you expect a password prompt so you type the password and hit enter, but you mistyped the original command and you end up typing your password in plaintext onto the command line. oops.

McKayDavis• 6 years ago

If you're using `bash`, A quick remedy for this is:

  unset HISTFILE
  exit

This prevents `bash` from updating the `~/.bash_history` file on exit with the command you don't want memorialized. (assuming you don't also have some hook that updates your history file after every command)

fitzn3• 6 years ago

I understand what you're saying but if anybody on your machine can read the commands, can't they also read your configuration files? Or are you saying a different user on the machine can run `ps` and see my processes but not necessarily my portion of the file system? My thought is if the file is not encrypted on the disk, then any desktop application can read it. So, while I agree that preventing a user from reading bash history is not worth it if they can read your processes, aren't configuration files (if unencrypted) just as insecure?

mturmon• 6 years ago

In a Unix system, files have 3 levels of permissions — for the user who owns the file, the group-of-users that owns the file, and any system user.

So, a given config file can have permissions so that the file owner can read and write, but other users cannot. Like your ssh keys.

But ‘ps’ can be run by anyone, and it can typically access the whole command line you used.

jlgaddis• 6 years ago

Anybody on your machine can read the commands you are running, for example with ps.

This is why I always mount /proc with the "hidepid=2" option (as well as the "gid=<n>" option to ensure that doesn't apply to the members of a specific group, such as the "admins" of the box).

perlgeek• 6 years ago

This doesn't have to be true. You can mount /proc with hidepid to prevent users from seeing other users' processes: https://access.redhat.com/documentation/en-us/red_hat_enterp...

capableweb• 6 years ago

Indeed! One thing that at least works on zsh is to prefix the command with a space, then the command doesn't go into the history file.

So if you do `date`, it'll end up in the history file but if you do ` date` (one space prefixed), it won't. One `clear` later and it's like the command was never run.

kubanczyk3• 6 years ago

Two big ones missing. After ctl-r history search:

  ctl-o ctl-o ctl-o   re-execute the history one by one
  
  alt-. alt-. alt-.   last argument (better ux than that !$ business)

rkangel• 6 years ago

I didn't know either of those, thanks!

celeritascelery• 6 years ago

No matter how many times I read articles like this I always manage to learn something new. For example this time I learned about brace expansion. Will save me retyping a bunch of commands with different text.

kreetx• 6 years ago

Same here. I though I'd seen most of them by now but apparently not :)

echo ${MY_VAR:-some_default} is another class of useful functionality.

jldugger• 6 years ago

sadly, i know how that works only because it was the recommended long term fix from an RCA

alufers2• 6 years ago

The single best shell productivity tip for me was to use z:

https://github.com/rupa/z

It automatically makes a list of directories you frequently use in your shell and if you type `z a-small-substring-of-the-directory-path` it does a cd to that directory. (e.g. I type `z and` and it changes my current directory to $HOME/Work/Projects/android`). Saved me tons of typing.

Spivak• 6 years ago

A somewhat less fancy version of this is $CDPATH.

   export CDPATH="/path/to/my/projects:$HOME/Work/Projects"
   # From anywhere.
   cd andr<tab><enter>

mceachen2• 6 years ago

Note that CDPATH can break shell scripts that assume `cd` hasn't been adulterated.

A huge block of "unalias" calls at the beginning of shell scripts used to be a common practice, but they don't seem to be found in more widely-used tools these days.

kubanczyk3• 6 years ago

TIL, magnificient! It doesn't work on the minimal Centos 7, so you need:

    yum install -y bash-completion
    bash      # a fresh run

JNRowe4• 6 years ago

CDPATH doesn't require the export as it is local to the shell session. Simply setting it will work as you'd want and won't pollute the environment for subshells and scripts.

And zsh users get another advantage here. You can use assign to the cdpath array or the CDPATH colon-delimited string, they are both treated the same internally. The advantage is being able to `cdpath+=...`, much like adding to $PATH with `path+=` or removing from PATH with `path^=`. Check zshbuiltins(1) for how to make your variables work like this, or how to use typeset -U to remove duplicates automatically, or $thousand_other_cool_things.

marcuskaz• 6 years ago

Agree, so useful! A port is also available for PowerShell: https://github.com/vincpa/z

andrelgomes2• 6 years ago

Have you checked out Warp drive for zsh? https://github.com/mfaerevaag/wd It is more manual than z, but takes out the learning phase. Will check out z though!

alufers2• 6 years ago

It depends on how you work, I have a lot of small projects which come and go, so setting up the paths every time I start or finish something would be inefficient.

Although I might adopt it for different weird system configuration directories which I often forget the path to.

__Joker• 6 years ago

Sound similar to autojump. https://github.com/wting/autojump

fdw2• 6 years ago

The advantage of autojump - at least for me - is that it doesn't depend on the shell, as it is implemented in Python. This makes it possible to read and write the data elsewhere, too, independent of your shell.

For example, I integrated it into ranger (see my history, I don't want to spam HN with this project).

gimboland2• 6 years ago

I use autojump, but I liked the idea of having an fzf-powered menu pop up when invoked without any args (which is not how autojump behaves). So I made rustled this up (zsh): https://pastebin.com/aLmqJVRy

kortex4• 6 years ago

This is my workhorse, 'to'.

No argument: start fzf (this really should read "to $(fzf)"), jump to that dir.

One arg: if the path is a real file/dir, jump to it, else use zoxide (z in rust) to soft-match it from history.

https://github.com/xkortex/bashbox/blob/master/aliases/core....

alektorophobiae• 6 years ago

set -o vi to use vi on the command line.

jwilk• 6 years ago

Toggle your cursor between its current position and the beginning of line

That's quite not what ^X^X does. The bash man page says:

Swap the point with the mark. The current cursor position is set to the saved position, and the old cursor position is saved as the mark.

mstudio• 6 years ago

fishshell is my favorite productivity enhancer for the terminal: https://fishshell.com/

The primary reason I chose over zsh/bash was that autocompletion just works. If you install it, I highly recommend oh-my-fish as well, which allows you to add packages for look/functionality: https://github.com/oh-my-fish/oh-my-fish

If you're a zsh user, they have a similar framework: oh-my-zsh: https://github.com/ohmyzsh/ohmyzsh

ASTP001• 6 years ago

I really enjoyed using fish for a short while, but soon I learned that it was not POSIX compliant. This was a deal breaker for me since my colleagues would often share scripts that wouldn't just work for me. I use zsh nowadays.

JNRowe4• 6 years ago

I love zsh, but it isn't without its own strange quirks that make compatibility exciting at times. In a thread from a fortnight ago¹, RANDOM is handled very differently for example. Try `echo $(echo $RANDOM) $(echo $RANDOM)` in zsh and bash.

I'm unsure whether minor differences that catch you out occasionally are worse than huge differences that you always have to think about. For me zsh still wins ;)

1. https://news.ycombinator.com/item?id=22841259

LEARAX• 6 years ago

Well, you could always just use a shebang or call bash/sh directly on the file.

linsomniac• 6 years ago

I tried fish for a while and loved the history recommendation mechanism, but the 'vi mode' emulation had enough annoyances that I went back to zsh. From the bugs filed on it, it seems like the vi fixes weren't really possible.

I did install a similar history recommender in zsh and have been pretty happy with that.

stygiansonic• 6 years ago

Another vote for fish. I would say another benefit is that it has a reasonably great out of the box experience with little customization needed to get a lot of its benefits, eg awesome autocomplete.

The downsides have been mentioned, namely the lack of compatibility with shell scripts not written for it. In the same vein, until somewhat recently, operators like && did not work for it.

It’s still my daily shell, and I really only use something like bash when having to remote into a production server.

fitzn3• 6 years ago

The article links to this at the bottom, but just calling it out again because I think it's so useful. I am using git all the time---probably a majority of my terminal relates to git in some way. I think the git bash completion script is extremely useful. https://github.com/git/git/tree/master/contrib/completion

It's incredibly easy (and transparent) to "install" and saves a ton of keystrokes.

mceachen2• 6 years ago

On more handy set of emacs-derived keystrokes:

Ctrl-a ctrl-k to clear the current command, or redo a hidden password prompt if you know you fat-fingered it.

ctrl-a jumps the cursor to the beginning of the line, and ctrl-k deletes from current position to the end of the line.

The mnemonic to remember these: "a" is the first position in the alphabet, and jumps you to the first position. "K" is short for "kill.”

wazoox• 6 years ago

Ctrl-u (delete everything from the cursor to the beginning of line) is one less stroke for the very same result :)

usrme• 6 years ago

Ctrl+U is also really handy for clearing password (or any other input) lines where you know you made a mistake, but don't want to just spam Backspace until you think you've gotten to the beginning.

Axsuul• 6 years ago

Any ideas why ctrl+k doesn't work in my iTerm2? All other readline shortcuts work.

asicsp2• 6 years ago

I rarely use Ctrl+r to get command from history, because most of the time I know the starting characters of the command I need. So, I type those characters and use arrows keys to match from history. This is enabled via these settings in .inputrc

    # use up and down arrow to match search history based on typed starting text
    "\e[A": history-search-backward
    "\e[B": history-search-forward

I also changed couple of setting wrt Tab autocompletion:

    # when using Tab for completion, ignore case
    set completion-ignore-case on
    # single Tab press will complete if unique, display multiple completions otherwise
    set show-all-if-ambiguous on

codegladiator• 6 years ago

You can use ctrl+r for commands which are similar (say ssh to n number of servers, so the up/down would still take some amount of browsing the options)

Just tag commands with a # at the end

$ complex command here # tagHere

Later use the tag with ctrl+r

I do this all the time

ssh 127.0.0.1 # master

asicsp2• 6 years ago

yep, ctrl+r is better suited for such cases or whenever you have to match a command other than starting characters

and I think there are some fuzzy matching options (like fzf) that can be added to ctrl+r

tasuki2• 6 years ago

When I used bash, I had all of your settings on (had more, some others I can live without - these four I can't). I still have them in .inputrc for programs that use readline!

I have since switched to zsh. Up/down to search history, in ~/.zshrc:

    [[ -n "${key[Up]}" ]] && bindkey "${key[Up]}" history-beginning-search-backward                                              
    [[ -n "${key[Down]}" ]] && bindkey "${key[Down]}" history-beginning-search-forward

Autocompletion in zsh is a different beast altogether :)

seemslegit3• 6 years ago

Best shell productivity trick ever: don't use the shell for anything more complicated than launching executables and use a reasonably modern programming language to achieve all other tasks.

wasdfff• 6 years ago

What is a reasonably modern language? Everything seems to be at least 20 years old now.

bhaak2• 6 years ago

OP might have made their statements tongue in cheek.

But there's the truth in there that even programming languages that are 20 years old had a completely different mindset applied to their design process than the original Unix shell which was much more ad-hoc.

40 years ago IT was a completey different environment than 20 years ago.

serf• 6 years ago

no thanks.

'rename .txt _1.txt *' is quite a few lines in a language that's super simple, like Python, and it turns into code golf with even more terse languages.

I'll stick to the terminal.

kortex4• 6 years ago

anything more complicated than launching executables

mv is a simple executable. GP is talking about stuff like generating reports or sophisticated orchestration in .sh

jabl• 6 years ago

I'd love a good answer here. Shell is clunky and annoying as hell, but a "better" language like python is very verbose for typical shell stuff like process management (processes, pipes, signal handling), small text handling tasks like regex matching or string splitting.

Perl is maybe the closest here, but it certainly has it's fair share of annoyances, and I more or less dropped perl for python decades ago and don't particularly feel like going back.

seemslegit3• 6 years ago

Not really, with python there's a habbit-acquiring curve but you end up with longer lasting more readable and maintainable assets that can be modularized refactored etc. perl is write-only and has little going for it afaic but ymmv.

usefulcat• 6 years ago

If you care about your command history, I would advise against doing this unless you are truly a sed expert:

    $ sed -i '/secret-command/d' $HISTFILE  # deletion of history line containing 'secret-command'

Because if you get it even slightly wrong, you could easily hose your entire history. If it were me, I would just use an editor to delete that line (or two, if timestamps are enabled).

wazzaps• 6 years ago

And if you know don't want a command in your history, prefix it with a space.

arendtio• 6 years ago

Today, was actually the first time that I used the vim time travel feature. I noticed that I had deleted a line by accident somewhen during the last few minutes. So I typed

  :earlier 5m

and copied the line. Afterward, I typed

  :later 5m

and was back where I started, ready to paste the missing line :-)

ryangittins• 6 years ago

On mornings when I'm feeling particularly lazy in the office, I'll do

  :later 8h

and commit the code before heading home. ;)

fdw2• 6 years ago

Some plugins that saved me lots of typing with zsh:

https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/histo... looks through your history and only shows entries that started with the same characters that you already typed. Similar to Ctrl-R with fzf, but still similar enough that I use both.

https://github.com/zsh-users/zsh-autosuggestions also looks through your history for commands with the same start, but it will automatically suggest them for you to use. If you often use the same commands, this saves quite some typing. For bonus points, I hooked it up so that Ctrl-Return will execute the suggestion.

https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/per-d... To make the first two even more powerful, I split my history based on the directory I'm in. This means that it's only suggesting me commands I entered before in this directory.

zeppelin101• 6 years ago

I'm a huge fan of the first 2 plugins, but I didn't know about the per-directory history plugin. I might put that in my toolbox. Thanks for the recommendation!

kortex4• 6 years ago

My productivity stack:

Zsh (with all the autocomplete plugins)

Oh-my-zsh zsh-autosuggestions: https://github.com/zsh-users/zsh-autosuggestions

Tmux

Histdb: https://github.com/larkery/zsh-histdb

Zoxide (replace cd): https://github.com/ajeetdsouza/zoxide

Fzf

Rip (replace rm): https://github.com/nivekuil/rip

My personal dotfiles: https://github.com/xkortex/bashbox

Mosh (replaces/augments ssh)

What I'm currently looking for is a better multi-host logging solution.

l0b0• 6 years ago

**

is not "expanded to all files and directories in the children directories, with a depth limit of 1." It's hard to see using `ls` because `ls` doesn't just list its arguments, it also lists files inside its directory arguments. Use

  echo **

and you'll see it just expands to the same as

- all the files in the current directory. Or type

**

and press Ctrl-x, Ctrl-* (glob-expand-word) to expand it.

After `shopt -s globstar`

**

expands to all files recursively.

Also, `echo $dir | tr '[:lower:]' '[:upper:]'` should be `printf '%s' "$dir" | tr '[:lower:]' '[:upper:]'` to handle spaces and other weirdness in file names.

(If anyone knows how to escape asterisks in HN it would be helpful to know :)

kmarc• 6 years ago

My productivity tip:

Sit down, and read through `man bash`.

Thank me later.

hattori• 6 years ago

Not built-in but hstr (https://github.com/dvorka/hstr) and tldr (https://github.com/tldr-pages/tldr) are awesome.

Shell Productivity Tips and Tricks

Related Stories