Better Bash History (2012)

https://sanctum.geek.nz/arabesque/better-bash-history/

by Pete_D • 7 years ago

267 83 7 years ago

sanctum.geek.nz

tapoxi• 7 years ago

highly recommended for fellow ctrl-r addicts: https://github.com/junegunn/fzf

umvi• 7 years ago

These ones are pretty good too:

https://github.com/cantino/mcfly

https://github.com/dvorka/hstr

graton2• 7 years ago

I also recommend FZF. Really very handy for doing CTRL-R searches.

I also use it for doing interactive git rebases. I have an alias for doing a fuzzy rebase interactive (frbi):

   frbi = !git rebase -i $(git log --pretty=oneline --color=always | fzf --ansi | cut -d ' ' -f1)^

kbd8• 7 years ago

This is a great idea, thanks for it! I've done similar things with using fzf to git add, git diff, and so on, but I hadn't thought of using it to pick a commit to rebase from.

I made a few changes so I figured I'd post them back here in case you're interested:

    pick-commit = # equivalent to your git log... cut...
    rif = "!f() { rev=$(git pick-commit); [[ $rev ]] && git ri $rev~; }; f"

"rif" = "rebase interactive fzf/fuzzyfind". The function and test just makes it exit cleanly if you escape out of the filter instead of giving "fatal: invalid upstream '~'" (and I changed it from ^ to ~ because I think ~ is usually what one should use over ^, but I'm willing to be corrected). And obviously I already have 'ri' aliased to 'rebase -i'.

While I'm here, here are my similar git aliases for adding, diffing, etc. files using fzf:

    pick-status-files = "!f() { git status -z | xargs -0n1 | cut -c4- | fzi --print0 | xargs -0to git "$@"; }; f"
    af = !git pick-status-files add
    # and so on for diff, difftool, checkout, etc.

Oh, and 'fzi' (fuzzy-inline) is a small shell script that just calls:

    fzf --height 30% --reverse --multi "$@"

barbuk• 7 years ago

fzf is the most fun and productive tool added in my dotfiles in recent years.

Mix it with https://github.com/rupa/z or https://github.com/skywind3000/z.lua for fuzzysearch/cd

And you can do cool scripts, like:

- search your vim edited files with highlighted preview: https://github.com/BarbUk/dotfiles/blob/master/shell/complet...

- search and open your chrome history: https://github.com/BarbUk/dotfiles/blob/master/shell/complet...

aaronharnly• 7 years ago

I set up my bash config to store history separately for each working directory (ie separate history files that I store under a $HOME/.bash_history_d directory) – so as I cd into a project directory, the history is populated with the commands that I've run within that directory.

I find this super-helpful to remind me of the context for projects – cd in, then up-arrow to whatever that command is I usually run to launch this server, grep this logfile, etc.

There are some downsides – sometimes I want to run the same command in directory X and also in subdirectory X/Y, but I love doing it this way.

Anyone else done something similar?

zwieback• 7 years ago

That's clever, I'd like that. Can you have another key to recall the global history, e.g. all the combined histories like you would normally see?

dllthomas4• 7 years ago

I do something similar, but instead of current directory I have a notion of context that is carried by an environment variable set in a GNU screen process (and therefore inherited by its children) which drives logic in my .bashrc.

https://github.com/dlthomas/config-files

dual_basis3• 7 years ago

Fish does this by default (and much more as well).

dasl• 7 years ago

I use zsh, and I have a thing in my profile that skips writing any `cd` command to history. Instead, it will rewrite the command using the absolute path of the directory I `cd`'d into and write that to history.

Thus, all of the `cd` commands that get written to my history use absolute paths and are re-usable no matter where I am.

It would be cool if this sort of functionality were generalizable to all commands where you type a relative path, not just `cd`.

https://github.com/dasl-/settings/blob/3143bbfe23bd75c3e35c7...

smacke• 7 years ago

You might enjoy checking out 'fasd', which, among other things, creates an alias ('zz') with autocomplete functionality for changing to frequently-used directories.

https://github.com/clvv/fasd

dual_basis3• 7 years ago

In fish, suggested autocompletions are contextual. For example, if I am in /home/dual_basis/projectA and I type `cd some/subdir` then a bunch of time goes by when I come back to /home/dual_basis/projectA and start typing `cd` it will suggest `cd some/subdir`, even if I have typed a bunch of other `cd` commands while in other directories.

This isn't specific to `cd` commands either. It's useful if you are `rsync`ing different directories to different places, or checkout different `git` branches, etc.

tyingq• 7 years ago

That seems like it would be tricky for it not to screw something up.

Like, for example, would it handle this properly?

$ (cd /source;tar -cf - .) | (cd /dest;tar -xf -)

Yen• 7 years ago

I've done the same thing, and I've found it super helpful.

My particular configuration is at https://github.com/YenTheFirst/dotfiles/blob/master/.bash_cd...

90% of the time, the bash history is super relevant to the current context. A small amount of the time, I remember that I issued a command, but not in which directory - in these cases, I can just search the history directory, and get both the context of what the command was, and also where I was when I originally issued it.

chubot4• 7 years ago

Thanks, reading over that code was useful. In particular I realized that my Oil project (a bash-compatible shell [1]) doesn't support PROMPT_COMMAND, which tons of people appear to be using.

https://github.com/oilshell/oil/issues/320

It seems like a weird hack but apparently it works for tons of things!

[1] Latest status: Success With the Interactive Shell http://www.oilshell.org/blog/2019/02/05.html . Oil runs a bunch of interactive programs like bash-completion, virtualenv, and git-prompt.

inciampati• 7 years ago

This is lovely, and I should integrate it with the chronology based approach I've been using.

I keep an infinite log by appending a timestamp to the beginning of the lines written to my history file, then merging this into a growing log with an hourly cronjob that runs sort -n | uniq on the file and the new version of the shell history. This way the shell history file is just a buffer. It doesn't get too big to cause corruption or other issues with reverse search on the command line. I just grep my way through the big history file to figure out what I was doing and when.

saagarjha• 7 years ago

But doesn’t that mess up the order of reverse-i-search?

GuB-42• 7 years ago

zsh-histdb stores history inside a sqlite database instead of a simple text file.

The good thing is that it includes context: current directory, session, date.

I use it together with ssh-suggestions which can use SQL queries to find the most relevant command. You can program rules such as: use the most frequent command in that session, if you none do it for the current directory in the last 6 months, if not, search globally.

dylan604• 7 years ago

isn't that totally opposite of the *nix philosophy of everything is a text file?

dudouble• 7 years ago

Thank you for this. It always amazes me when I come across something that would be so useful to me, and yet so obvious in hindsight.

lscotte• 7 years ago

Anyone else done something similar?

I'm not, but now that you mention this, I think I will. I already use direnv all over the place, and this would be a pretty natural extension. Thanks!

_em_• 7 years ago

Could you share details on how did you setup this. This sounds super useful

euske• 7 years ago

I set PROMPT_CMD to the following function:

    function log_history {
        echo "`date '+%Y-%m-%d %H:%M:%S'` $HOSTNAME:$$ $PWD ($?) `history 1`" >> $MYHISTFILE
    }

which records a timestamp, host name, shell pid, working directory and exit status for each command.

unhammer• 7 years ago

I use one bash history (and fasd history) per tmux session: [link redacted]

narnianal2• 7 years ago

It's a great idea. If you additionally also like working with chroots you might find containers an interesting topic.

Basically a container let's you wrap a part of your filesystem, network, process space in a separate world of its own. And in that world you have only the context related stuff like history, processes, libs etc. Additionally most container environments come with tooling that enables you to make everything reproducable, so that you can share your container with others.

The great thing is you can always live in global space, while switching between what that global space means.

danappelxx• 7 years ago

How would one go about setting up a container?

mistrial9• 7 years ago

can anyone please explain how BASH history on current Ubuntu, is different when logging in from different desktop machines, to the same login ? the dot-local directory or dot-gconf directory is part of this ? consternation..

O_H_E• 7 years ago

By default, Bash only records a session to the `.bash_history` file on disk when the session terminates. This means that if you crash or your session terminates improperly, you lose the history up to that point.

I kept loosing my history consistently on my new installation because of this. I usually put my laptop to sleep, and pretty much never intentionally terminate my x-session except on kernel updates. But because its a cheapo laptop with a crappy battery, I would occasionally run out of battery when asleep or it just wont wake up.

barrkel• 7 years ago

I have a fairly expensive ThinkPad and it also runs out if battery if asleep for more than about 4 days, and frequently freezes on suspend or resume when it's been both suspended and resumed docked and undocked. (Linux, of course.)

thenewwazoo• 7 years ago

Because this comes up every so often, I'll post this link again: I wrote a little widget that stores your bash history in a sqlite db. I haven't touched it in a while because I now rely on fzf and this doesn't integrate with it, though maybe I could make it do...

https://github.com/thenewwazoo/bash-history-sqlite

chubot4• 7 years ago

This is cool, somebody recently suggested that I add history to sqlite in Oil [1]. I looked at your code but I didn't see how it integrates with bash?

Most people seem to be using the PROMPT_COMMAND hook for stuff like this -- are you using something else, or did I miss it?

https://github.com/oilshell/oil/issues/320

I will add PROMPT_COMMAND to Oil, but I'm curious if there is something else I should add to support end-user customization.

One thing that might be interesting is to support a hook for the ! syntax, so the sqlite DB could be searched. I think that would solve some of the problems you mention in the code's comments, and maybe the fzf issue. (Although that literally just occurred to me, I don't promise to do this :) )

I haven't looked at how fzf works exactly. But if you want to chat about it please chime in on Github :)

[1] Latest status: Success With the Interactive Shell http://www.oilshell.org/blog/2019/02/05.html . Oil runs a bunch of interactive programs like bash-completion, virtualenv, and git-prompt!

skissane• 7 years ago

This is cool, somebody recently suggested that I add history to sqlite in Oil

I think a shell with an embedded sqlite, which makes sqlite accessible to scripts as shell builtins, and in which command line history is stored in a sqlite database instead of a plain text file, would be pretty cool. You could store command history (and maybe even other stuff) in something like $HOME/.shell_db As well as just a list of lines, other useful information could be stored for each line, such as current directory, timestamp, username (maybe I su to root and am running commands as root but still going into my own account's history database), hostname (if $HOME is on NFS, putting aside the issue that databases, including sqlite, don't work well over NFS due to file locking issues), etc.

enriquto2• 7 years ago

somebody recently suggested that I add history to sqlite in Oil

what a disgusting, perturbed mind! Shell history is probably the kind of data that fits better as a text file. It is naturally textual and it can never grow too long, even if you are typing commands continuously for your entire life. Please, keep the dirty hands of sql far, far away from the shell!

kbd8• 7 years ago

Bash's broken history is what finally got me to switch to Zsh. The history behavior you want is: 1. each line of history is appended incrementally, but 2. each shell session only has access to its own history unless you specifically reload the history into your session.

Every combination of history commands in bash didn't do the right thing[1], but Zsh's works as desired.

[1] eg. putting 'history -a' in PROMPT_COMMAND as the article recommends appends your session's entire history each time, IIRC.

cortesoft• 7 years ago

See, what I want from my history is every command I have ever run, so that ctrl r can find anything I have done.

kbd8• 7 years ago

Yes what I wrote doesn't exclude that. The only distinction is that you generally don't want all your sessions' histories interleaved. I.e. when you up-arrow, you only want your current session's history, without random things from other active sessions in there too. If you do want "everything you've ever done" including things from other sessions that started after your current session, it's easy to reload your history into your session. I have a 'rlh' (reload history) alias for that.

jackewiehose2• 7 years ago

[1] eg. putting history -a in PROMPT_COMMAND as the article recommends appends your session's entire history each time, IIRC.

No, only the last command. Very handy because that way you can start a new terminal and have your current history available immediately. I also have an alias r='history -n'. With this you can load the history of another terminal session directly into the current session.

kbd8• 7 years ago

Just tested it and 'history -a' behaves how you say. Maybe it was an older version of bash that had this behavior.

Edit: now I'm obsessing about reproducing this behavior from years ago! If anyone knows what I'm talking about I'd be grateful for some validation :)

matthewbauer• 7 years ago

`setopt inc_append_history_time` is super useful for Zsh users. This gives you some sharing of histories between shells.

stilist• 7 years ago

Since everybody has their own way of handling it, mine is

* Generate a new history file for every session (https://github.com/stilist/dotfiles/blob/master/dot_sh/sessi...)

* A 'histgrep' script that searches across all the files and highlights matches (https://github.com/stilist/dotfiles/blob/master/dot_sh/bin/e...)

MarcScott• 7 years ago

I've had this in my .bashrc for years. I thought it came from this post judging from my commments, but looks different, so I wonder whether it came from HN comments when this was previously posted.

  ## Better bash history
  # Avoid duplicates
  export HISTCONTROL=ignoredups:erasedups
  # When the shell exits, append to the history file instead of overwriting it
  shopt -s histappend
  # After each command, append to the history file and reread it
  export PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND$'\n'}history -a; history -c; history -r"

mrzool• 7 years ago

I love this blog and learned so much about the Unix toolkit and the philosophy behind it by reading it. Too bad it doesn’t seem to be much active anymore, but there’s a lot of great material in the archives that pretty much never gets old since its focused mostly on standard Unix software. The author (Tom Ryder) is an very talented writer with a clear and concise style.

A couple of plugs for your consideration:

- I integrated all of these history options in my project Sensible Bash [1], an attempt at saner Bash defaults

- I made an ebook [2] out of a series which appeared on this blog, titled “Unix as IDE”.

[1] https://github.com/mrzool/bash-sensible

[2] https://github.com/mrzool/unix-as-ide

pkaye• 7 years ago

We have some servers where the user home directory is on nfs. To keep separate history for each server, I add the following to .profile.

export HISTFILE="${HOME}/.bash_history.$(hostname)"

bhaak2• 7 years ago

I just recently stumbled over https://spin.atomicobject.com/2016/05/28/log-bash-history/

Using PROMPT_COMMAND to log every command in a separate timestamped log file. This allows you to use a much better format than bash history has to be in.

Easily readable to see what you've been doing and extremly efficient to grep for.

Has been discussed on HN before: https://news.ycombinator.com/item?id=11806553

jefftk• 7 years ago

I do:

    promptFunc() {
      # right before prompting for the
      # next command, save the previous
      # command in a file.
      echo "$(date +%Y-%m-%d--%H-%M-%S) \
        $(hostname) $PWD $(history 1)" \
        >> ~/.full_history
    }
    PROMPT_COMMAND=promptFunc

This is in addition to standard bash history, and is much more reliable. Being able to see not only what command I ran but what directory I was in and when I ran it is very useful.

(I wrote about this here: https://www.jefftk.com/p/you-should-be-logging-shell-history)

deeg• 7 years ago

I do something very similar (just saved in files by month) and it has saved my bacon a few times. I was recently asked to re-run a process that I ran years ago. It would have taken me hours to re-discover how to run the process but instead took a few minutes after grepping the commands.

Every developer should do this.

314• 7 years ago

Yeah, this is how I do it - although I append the hostname to the end of the filename and sync the collection between my machines so that I can grep for history on any machine.

Edit: went to check and apparently I've overcomplicated this somewhat (it's about ten years old) but it is nice if empty lines are not logged. It's an old habit to ctrl-c then hit return repeatedly if a terminal is not responding.

  PROMPT_COMMAND=storehist

  storehist ()
  {
    CMDCNT=`history 1 | sed -e 's/\w* *\(.*\)/\1/'`;
    if [ "$CMDCNT" != "$LASTCMDIND" ] && [ -n "$LASTCMDIND" ]; then
        DATE=`date '+%Y%b%d %H%M'`;
        if [ "$LASTCMDIND" != "$CMDCNT" ]; then
            echo "$DATE $HOSTNAME:$BASHTTY $CMDCNT" >> ~/.custom_history;
        fi;
        LASTCMDIND="$CMDCNT";
    fi;
    LASTCMDIND="$CMDCNT"
  }

naggie• 7 years ago

I have a global bash history, having set up bash/zsh to [immediately share without any annoying race conditions.][2] I use FZF mapped to CTRL+R to search through my history.

The unique thing is that I use a [python script (in my dotfiles)][1] to clean the history in a more advanced way than is possible with HISTIGNORE.

The script removes duplicates, leaving the most recent copy and removes lines matching several regexes.

[1] https://github.com/naggie/dotfiles/blob/master/scripts/clean... [2]: https://github.com/naggie/dotfiles/blob/master/home/.bashrc#...

n8henrie• 7 years ago

I was recently planning to write something just like this. Thanks for sharing.

IronBacon2• 7 years ago

If one is only interested to remove duplicates (your script does more) I've found this one liner in a StackExchange post: awk '!x[$0]++' ~/.zsh_history

pwagland• 7 years ago

Note that this only keeps the first copy, not the most recent copy. To make it so that it only keeps the most recent, you would need to run this through `tac` twice.

commandersaki• 7 years ago

Probably the most useful thing I've done regarding bash history: https://eli.thegreenplace.net/2016/persistent-history-in-bas...

ConSeannery• 7 years ago

To echo what everyone else is saying, get FZF. Most of these linux productivity things are just some dumb pet project that solves a specific problem that a specific person finds annoying. FZF is not this - it makes something you do hundreds of times a day (history and file opening on zsh, bash, fish, whatever) indisputably better. If you find yourself sitting there tapping ctrl+r multiple times like an idiot for some thing that may or may not be there, do your stubborn beardy ass a favour and install fzf.

You can implement some or all of the tips in this HN post, it'll make fzf results better.

shriek• 7 years ago

Yup, I tried the context aware history for couple of days and it got annoying pretty fast. I have everything in one file which I can just FZF reverse search and bam. No need for infinite up arrows.

kbumsik• 7 years ago

Yeah, these kinds of hacks usually end up with unmaintainable mess when cumulated. Just use fzf.

mceachen• 7 years ago

https://github.com/junegunn/fzf for the uninitiated

ausjke• 7 years ago

that's exactly what my setting is for history, nice post. The only difference is that I unset HISTCONTROL, don't recall why I did that.

    shopt -s histappend is the default already I believe.
    shopt -s cmdhist is also the default now.
    HISTSIZE=-1 will be unlimited, which is fine for today's computers.

zonidjan• 7 years ago

PROMPT_COMMAND is probably already used for something else by your distro.

HISTFILESIZE and HISTSIZE can be set to -1 for unlimited.

Read man bash.

bhaak2• 7 years ago

HISTFILESIZE and HISTSIZE don't solve the abysmal bash history file handling when you have more than one bash open.

I would also like to know which distro clobbers user settable env variables.

omeid2• 7 years ago

I use fzf for bash history. Fuzzy history search and much better TUI.

https://github.com/junegunn/fzf

tenken• 7 years ago

Meh. I prefer to leverage SQL.

I have yet to find a better way to store and search bash history than:

https://github.com/tkf/rash

More recent bash sqlite history attempts all seem to fall short than this implementation, they store less metrics or have other caveats.

romka2• 7 years ago

I remember reading this article back when I was still using Bash. I've finally got the history I like only after switching to ZSH.

You can get decent history in ZSH just by setting the right options. If you want great history, you'll need to write a bit of code. Right now my history works as follows.

History is written to disk after every command. Up and Down keys go through the local history (from the same session). Ctrl+Up and Ctrl+Down go through the shared history (from all sessions). Ctrl+R also uses shared history (it's easy to add another binding for local history but I don't have it). Pressing Up/Ctrl+Up after typing something will go over history entries that have the matching prefix. For example, `git<Up>` will show the last command from the current session that starts with `git`. Here's my config: https://old.reddit.com/r/zsh/comments/bsa224/how_to_setup_a_....

In addition, my history is stored in Git. History from the local machine comes first, and from other machines second. This is fairly easy to do it Bash, too.

Better Bash History (2012)

Related Stories