Curate your shell history

https://esham.io/2025/05/shell-history

by todsacerdoti • 1 year ago

156 84 1 year ago

kstrauser• 1 year ago

I see the benefit, but... that looks like work. From the post:

Why save vim /etc/rc.conf when sudo vim /etc/rc.conf is what I meant 100% of the time?

Good point, but AFAIK all shells search history by recency by default. Whether I search with Fish's up-arrow, or with Atuin's ^r, the first result will be the match I executed most recently. In this case, that means I'd have to type `sudo vim /etc/rc.conf` correctly one time and then that's the first version of the command I'll see next time I look for it. And if it's something I do often enough for it to matter, but still manage to screw it up frequently, I'll turn it into an alias or function.

This is the kind of busywork that feels like an ADHD tarpit to me. No. I absolutely do not need to optimize my shell history, lest I end up with a beautiful, tiny history file that's free from the detritus that would have come from me spending that time actually doing my job.

corytheboyd• 1 year ago

Agree, the recency sort sort ends up normalizing history over time anyway. Use fzf with your history to see more than one match a time anyway, and you can at least glance through a few of the iterative commands you went through if your history isn’t “normalized” yet. I like keeping shell history dead simple— it’s a stack of executed statements. Period.

I do pull very often run things into functions and source them into every shell session. Usually only parameterized things that aren’t trivial to use reverse search for.

Cockbrand• 1 year ago

I've been following a similar approach for a while. In bash, I have `HISTCONTROL=IGNOREBOTH` activated (or `setopt hist_ignore_space hist_ignore_dups` in zsh - most shells have an equivalent setting), and I just prepend most commands with a space. So only the more or less important stuff ends up in .bash_history or .zsh_history. This has the added nice effect that I don't accidentally trigger something destructive when I browse through my history.

I also spend most of my online life in incognito windows, at least for sites that don't absolutely require a login. This keeps my browser history clean from all the disposable pages that I only visit once, and I take care to do only the more meaningful stuff in a regular browser window.

Rediscover2• 1 year ago

Two prehensile thumbs up - your HISTCONTROL or

HISTIGNORE='m:??:info STARHERE:info:[bf]g:exit:[bf]g %[0-9]:help STARHERE:date:cal:cal ????:exec env ENV\STARHERE' (replace STARHERE with an asterisk)

generally works for me in bash(1).

belden2• 1 year ago

Oh crazy, I didn’t know about this aspect of bash.

The line above set up automatic history ignoring for the colon-delimited shell globs; eg

fg %3

won’t be recorded to shell history, since it’s matched by one of the globs.

remram3• 1 year ago

  HISTIGNORE='m:??:info *:info:[bf]g:exit:[bf]g %[0-9]:help *:date:cal:cal ????:exec env ENV\*'

alecco• 1 year ago

  HISTCONTROL=ignoreboth
  HISTIGNORE='rm *:ls *:cd *:cp *:builtin *'

The history file has ALL the lines prefixed by space. I curate it a lot, with most frequent/recent commands near the end. And I add a lot of comments to the file so things are easily searched. And search for things, for example /rsync to find the rsync quick backup snapshot lines so first comes one with -n to check what it would do, then if it looks OK I press 'n' (vi mode) and get the same line without -n, or another 'n' and I get the same without -n and with --delete.

    (cd ~ && [ -d ~/backup_mnt/ ] && rsync    -xav --delete --exclude='backup_mnt' --exclude='.debug' --exclude='.cache' "$HOME" ~/backup_mnt/backup )   # rsync DELETE
    (cd ~ && [ -d ~/backup_mnt/ ] && rsync    -xav          --exclude='backup_mnt' --exclude='.debug' --exclude='.cache' "$HOME" ~/backup_mnt/backup )   # rsync NO DELETE
    (cd ~ && [ -d ~/backup_mnt/ ] && rsync -n -xav          --exclude='backup_mnt' --exclude='.debug' --exclude='.cache' "$HOME" ~/backup_mnt/backup )   # rsync LIST

For very long lines I always explain it in a comment.

When trying to modify a command, I remove the space, and when done I search for saving history and editing with '/_hi':

    history -a ; vi ~/.bash_history && history -r

and voila.

I also keep the similar lines aligned with spacing to note the differences quickly. (e.g. the -n above)

When exiting a shell and there's nothing interesting, history -r; exit

Also always I go to the end of the file before quitting vi so next time I'm where it last ended and before the appended new commands.

It takes work upfront and discipline but the everyday use of shell becomes very fast and clean.

mock-possum• 1 year ago

Wouldn’t LLM-driven contextual fuzzy search of your shell history be more useful than manually keeping it clean?

I need less things to do, not more. Let the robot eat my history and make smart suggestions if it thinks I’m trying to do something similar to something I’m done before. That’d be actually useful to me.

kccqzy• 1 year ago

It would be too slow. LLMs incur too much latency so using LLMs gets you out of the flow. No way to compete against a simple substring or subsequence search.

kergonath• 1 year ago

Why use a LLM when standard fuzzy search does the job, at a fraction of the computational cost?

tonymet• 1 year ago

i like where you are going. even a simple full text search with ranking would work well.

https://github.com/atuinsh/atuin

c0l0• 1 year ago

I've been doing this for years, and my shell history is usually only a few lines/commands long when I start a new session. I like it that way. My browser's roughly the same - I always start firefox in "private browsing mode" (my system's default browser application is a shell wrapper around firefox that takes care of that), and I very consciouly use a non-private instance to "soft-bookmark" stuff that gets committed to my browser history. Actual bookmarks are for all the stuff that I consider really important not to forget about. A few sites (such as HN) have the privilege of me visiting them in non-private browsing mode while authenticated via cookes.

People might find that weird, and it sure is a tad inconvenient when random, low-priority websites cosplay Internet Fort Knox with very short and annoying MFA login timeouts and methods, but for me, the benefits (I'm less susceptible to tracking and can click on stuff with less angst that I might somehow leak data from my nigh-eternal browser session to some site or another, and I get to start relatively fresh in each session) outweigh the costs.

127dot12• 1 year ago

Nowadays EVERY site pretends being Fort Knox. Everyone's got a captcha for you.

jsphweid• 1 year ago

Where I work someone wrote a system to allow one to save every command they ever typed and make it available for searching via cli or web app. I opted in. It's probably one of the most useful tools I've ever used.

ethersteeds• 1 year ago

Centrally? Can you browse others'?

kriro• 1 year ago

How many credentials get exposed this way? That would be my main concern.

csmattryder• 1 year ago

I use fish with the sponge plugin that omits any command that doesn't return an error code 0. Combined with ones like `z` to jump around directories, I reckon my history is pretty tidy.

Anything important and convoluted, I'll shove it in the fish config for quick access.

eddd-ddde• 1 year ago

Fish + fzf integration is Wonder for working with history. Alt+c will show a fuzzy finder for changing directors.

cobertos• 1 year ago

I like keeping _all_ my shell history, with timestamps. Even the fuckups. It's come in handy a few times now

* Looking back at my old process for onboarding into some new technology. I can easily see how my technique evolved as I optimized for more correctness (e.g. cp -r vs cp -rp vs rsync with whatever flags does everything I want, I know I want the most recent incantation)

* Figure out the exact process I did for a backup or other movement of data when I want to ensure I have copied X, Y, and Z. Sometimes my shell history has saved me where my notes have failed.

jhgorrell• 1 year ago

I have most of my shell history back to 2005(?). Each terminal gets its own new history file.

99% of the time, I never look at it, but when I do need to look at it, it has been great. My boss once asked me: "What args and screening file did we use when we made that one-off DB 4 months ago?" Was able to check and confirm it was correct. Or for personal use: "Where did I move that folder of pictures?"

PeterWhittaker• 1 year ago

I opted for a single history across all sessions on any given host: On my main machine, the first of 54,434 entries is timestamped 2020-08-22:12:39, while on the machine on which I do most development at the moment (it varies from product to product and release to release), the first of 34,771 entries is timestamped 2023-05-08:11:34.

For the curious, the salient .bashrc bits are:

  function _setAndReloadHistory {
      builtin history -a
      builtin history -c
      builtin history -r
  }
  
  # preserve shell history
  set -o history
  # preserve multiline commands...
  shopt -s cmdhist
  # preserve multiline command as literally as possible
  shopt -s lithist
  # reedit failed history substitutions
  shopt -s histreedit
  # enforce careful mode... we'll see how this goes
  shopt -s histverify
  
  # timestamp all history entries
  HISTTIMEFORMAT='%Y-%m-%d:%H:%M '
  
  # not the default, we like to be explicit that we are not using defaults
  HISTFILE=~/.bash_eternal_history
  # preserve all history, forever
  HISTSIZE=-1
  # preserve all history, forever, on disk
  HISTFILESIZE=-1
  # record only one instance of a command repeated after itself
  HISTCONTROL=ignoredups
  
  # preserve history across/between shell sessions...
  # ...when the shell exits...
  shopt -s histappend
  # ...and after every command...
  PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND; } _setAndReloadHistory"

EDIT: Remembered just after submitting that since I am on MacOS, I ran the command

  touch ~/.bash_sessions_disable

back on August 22nd, 2020, to prevent Terminal from saving per-session information. I've never cleaned out ~/.bash_sessions, suppose I should, but it hasn't been updated since that day.

web007• 1 year ago

Curation is probably a good idea, but keeping context is probably a better idea.

The referenced "I don't keep history" philosophy is madness. You won't know what thing would have been useful to keep until you need it $later. Sure, you'll write down some good stuff and maybe alias it.

That's fantastic, do more of that anyway.

Don't pretend you're going to do that for every trick or gotcha you encounter, and don't think you're going to remember that one-off not-gonna-need-it thing you did last week when you find that bug again.

My local history is currently 106022 lines, and that's not even my full synchronized copy, just this machine. It's isolated per-session and organized into ~/.history/YYYY/MM/machine_time_session hierarchy. It has 8325 "git status", 4291 "ll", 2403 "cd .." and 97 "date" entries which don't matter. Literal/complete entries, not including variations like "date +%Y%m%d" which are separate. I can ignore them, either by grepping them out or filtering mentally, but something as benign as "cd .." is INCREDIBLY useful to establish context when I'm spelunking through what I did to debug a thing 2 years ago.

The even better version of both of these variants is to keep everything AND curate out useful stuff. That whole history (4 years locally) is 10MB, and my entire history compressed would be less than a megabyte.

Edit: just realized who posted this, I overlapped with Tod at my first gig in Silicon Valley. Small world!

abathur4• 1 year ago

I think it's a byproduct of how macOS itself saves and restores window state, but the macOS Terminal.app has options to control restoring scrollback on reopen. History is good, but it's really great to be able to scroll back and see more context around the commands I was running. (This can still fail me. In a rarely-touched tab I might have years of scrollback, but I currently have it set to restore 10k lines and I may find there's only a few days or hours in a tab where I ran something noisy with verbose logging...)

mmh0000• 1 year ago

I read this article in horror. Even after he said he kept /only/ the last 9800 commands.

  ⟩ history | wc -l
  170919

I have (deduplicated) shell history with timestamps going back years. A little bit of CTRL+R and `history | grep …` I never have to remember how I did some arcane magic in the past, just that I did.

abathur4• 1 year ago

How long does it seem to take to open a new terminal and reach a working shell?

Your hardware might be fast enough that it's negligible, but several years ago (maybe 2018-19ish?) I noticed my shell startup was getting sluggish and traced a large fraction of it back to the time bash took to load many lines into history.

I still like keeping it all, so I wrote something to dump it into a database as I go but then truncate working histories to something shorter (500, I think).

loeg7• 1 year ago

Instantaneous. bash might just be slow here; I'm using zsh.

  $ time zsh -i -c 'exit'
  zsh -i -c 'exit'  0.03s user 0.02s system 100% cpu 0.055 total

mzs• 1 year ago

Just put them in files, like month per machine/mpount per shell (bash, csh, etc). Then the shell will only load the current month's history. You can use grep of the files and easily narrow down what to search in.

o11c• 1 year ago

The only time I've ever had shell startup time be significant is when loading fancy shell rc stuff (oh-my-zsh is the most infamous but there are many others)

Unfortunately I've hit some obscure bug with exactly when `HISTFILESIZE` is applied, and had some shells truncate all my history undesirably. There are also problems with reading history if the timestamp format changes.

loeg7• 1 year ago

Yeah. On one host:

  wc -l .zhistory
  121104 .zhistory

pama• 1 year ago

Maybe I'm a hoarder. My timestamped histories from four machines in four different locations added up to 898028 lines --- I kept most of my shell commands, including errors and iterative attempts to get a good command, since early 2009 in the servers that matter the most to me. I use named shells with different histories each (roughly one to four shells per project), so the largest single shell history in these logs is only 43424 lines (21712 commands). The average across all shells is a bit over 75 logged commands per day, which doesn't sound as bad as the total line count in these histories. Despite the advent of LLMs I still need a lot of internal esoteric hacks that are relatively easy to find in these histories. Perhaps I should switch to using a centralized db that keeps the outputs as well and use them to finetune LLMs, but the ease of simple unix/Emacs tools operating with standard/simple history files always felt attractive.

koolba2• 1 year ago

Same here. Also included the host, timestamp, user, and current directory. Unifying history across multiple machines is very pleasant when you know you did something, but don’t even remember where you did it.

remram3• 1 year ago

What tool do you use for that?

wswope• 1 year ago

Obligatory plug for Atuin, which is a Sqlite-based shell history tool. It logs shell commands alongside timestamps, the working directory, and the return value.

You can optionally sync your history to a server in encrypted form to keep a shared history across hosts. The server is extremely easy to self host.

As discussed in TFA, it’s easy to filter or scrub junk like `cd ~/Desktop` so it doesn’t pollute your history. You can also fuzzy search and toggle between commands run in your current session, commands run on your current host, commands run in your CWD, and commands run on all hosts.

It’s my single favorite piece of dev tooling, and has made my job + life far smoother and easier. Highly recommend.

https://atuin.sh/

nullwarp• 1 year ago

This is the absolute first thing I ever install now. The amount of time it saves me is immeasurable.

There's no way I'm going to manually curate every command I type, I've got actual work to do.

Too• 1 year ago

What are the most useful features?

acedTrex2• 1 year ago

An absolute requirement for me. I also support it as a sponsor its such a useful project.

calmbonsai• 1 year ago

Ooooh, I'll have to take it for a test drive next week. This looks good for personal use as well as keeping a fleet of production maintenance/monitoring instances sync'd.

mediumsmart• 1 year ago

Atuin sounds great so I asked my llm if I should install it and it said: You already have a solid Bash+JSON+fzf setup that works and is flexible.

considering to ask where my json bash history is located and how that works - I noticed Ctrl-r looked different a while ago.

add-sub-mul-div• 1 year ago

SecureCRT can log everything from all sessions and then you have not only command history but the output, psql query results, etc, all in the order/context of the original session. I'd find it hard to live without.

127dot12• 1 year ago

What author does is nuts. But the advice of curating history is sound.

eitau_1• 1 year ago

I wish there was a (Jupyter) notebook-like interface in shells so only the final set of commands is saved in the history after a trial-and-error/refinement cycle

acedTrex2• 1 year ago

Ellie the inventor of atuin has something similar ish to this?

https://blog.atuin.sh/atuin-desktop-runbooks-that-run/

Im not sure if its entirely what you are asking about though

ellieh• 1 year ago

Sounds similar!

Shoot me an email if you fancy trying it

Ellie at Atuin dot sh

belden2• 1 year ago

I fall into the hoard-and-curate camp.

I use bash within tmux heavily, and got irked that a command I run in one shell session is not immediately available as a history item in other concurrent shell sessions. So I wrote a history plugin based on bash-preexec to track everything to two files: a per-directory history file, and a global history file.

I have a bash function which does history selection for me, by popping an fzf selector to look at the directory-specific history file. A keybinding within fzf allows me to switch to looking at the global history file instead.

Boring commands such as “cd”, “pushd”, and a few others don’t get logged. The log entries are in json format and include basic metadata; directory, timestamp, and pid.

Within the fzf history picker, another keybinding allows me to edit whichever file I’m actively using. So if I fumble a few times to construct a command, then when I get it right I just pop into the selector, edit the relevant file, and remove the lines I don’t want to misfire on again.

I’m sure this is basically what atuin does; now that I’m at the spot where directories are the unit of history relevant history, maybe I should give that tool another look.

One really interesting upside of all of this is that I now tend to make “activity-specific” directories in my repos. For example, I have a “.deploy” directory at the git root of most of my projects. There are no files within that directory; but my tool creates ~/.bash.d/history/home/belden/github/company/project/.deploy.json which contains the history of ~/github/company/project/.deploy/

Empty directories are invisible to git, but for me the directory “has” content: the log of how I need to deploy this service or that service.

It’s a weird way to use my shell, and just sprung out of the initial grief: I shouldn’t have to exit a shell session to have its history become available.

rane• 1 year ago

command I run in one shell session is not immediately available as a history item in other concurrent shell sessions

It's hard to understand how any other way is usable. If you have a dozen concurrent shell sessions, some running a long running process for example, and you had to restart one of them, you wouldn't be able to just Ctrl-C and go to the previous command. For me it's way better shell sessions live their own life but the history is accumulated to a single pool.

tryauuum• 1 year ago

command I run in one shell session is not immediately available as a history item in other concurrent shell sessions

this can be solved in vanilla bash by adding these lines to .bashrc (be sure to remove other HIST* configs options from it)

    shopt -s histappend
    export HISTSIZE=100000
    export HISTFILESIZE=100000
    export HISTTIMEFORMAT="%F %T "
    export HISTCONTROL=ignoredups:erasedups
    export PROMPT_COMMAND="history -a;history -c;history -r;$PROMPT_COMMAND"

additionally, you need to restart or close all the bash sessions to be sure no copy of bash with old settings exists

pabs32• 1 year ago

I recommend a few changes to these:

Infinite history instead of a large limit:

  HISTFILESIZE=-1
  HISTSIZE=-1

Also ignore commands starting with a space:

  HISTCONTROL=ignorespace:ignoredups:erasedups

Enable history substitution to verify commands before running them:

  shopt -s histverify
  shopt -s histreedit

Write commands to history files before running the commands instead of after the commands finish:

  trap 'history -a' DEBUG

Or if you already have a DEBUG trap:

  trap "history -a ; $(trap -p DEBUG | cut -d "'" -f 2)" DEBUG

epistasis2• 1 year ago

I definitely want to keep every single shell command I ever executed, including mistakes. And I want to track across machines, projects, etc.

Lately I have been using atuin for this. I have some gripes about interface defaults, and how the interactive search for history wipes out most of what was on my terminal view and causes a big "wtf was I doing again?" moment when it changes the terminal so much, but the tradeoff of shared history is very very much worth it for me.

wonger_• 1 year ago

Why keep the mistakes? Just curious.

tough3• 1 year ago

So you don't make them again?

AI Agents using tools also benefit from seeing past usage of the tools (what worked, what didnt) to help them inform future usage

epistasis2• 1 year ago

Mistakes often have side effects, and sometimes I don't notice it until later. It's good to come back to a project after a week and try to see where unintentional effects may have come from.

hiAndrewQuinn• 1 year ago

My unusual habit is: turn off the history file completely, by putting the command ‘unset HISTFILE’ in my .bashrc. [...] All the shell history I allow myself is localised and short-term.

I don't do this, but I do disable my browser history on my work laptop so that I'm forced to author things I, and presumably other people, can actually stumble their way to later on.

https://hiandrewquinn.github.io/til-site/posts/disable-your-...

omoikane• 1 year ago

unset HISTFILE

I don't do this because I didn't know "unset HISTFILE" was an option. Instead I symlink .bash_history (and other history things like .python_history, lesshst, etc.) to /dev/null

hamburglar• 1 year ago

Obligatory mention of something I like to call “bashtags”, where you just annotate a command line with a descriptive and unique end-of-line comment, eg “#bounce-web”. This combined with Ctrl-r and you just type the bashtag to find it. This has saved me so much time.

1vuio0pswjnm73• 1 year ago

"If I type a shell command that's valuable - one that did something useful enough that I might want it again in future, and long and complicated enough that I'd be annoyed to have to figure it out a second time from scratch - then I can't rely on it just happening to be in my .bash_history. So instead I put it somewhere else: maybe a shell function in my .bashrc, or maybe a shell script in my directory of random useful scriptlets."

I have a folder marked executable with hundreds of small shell scripts, appended to $PATH. I never write long scripts. I sometimes write scripts that run other scripts but I try to avoid such dependencies. I do not use bash, I use NetBSD Almquist or modified Debian Almquist shell so all the scripts are highly portable. I can move a tarball of this folder from computer to computer, whether the OS is Linux or BSD. These scripts generally run faster than bash scripts.

It would be untenable to save all command line history since I spend every day in the shell in textmode (not x11) and only occasionally switch to a graphical environment. I sometimes use Almquist-based shells that have command line history removed. This forces me to write useful scripts.

A lot of cumulative shell knowledge is contained in that folder of scripts. These scripts have worked for decades unchanged. I think the Almquist shell may be my favourite interpreter. No matter what else I try I always come back to the shell language. There is something appealing about scripts that seem to work forever.

dTal• 1 year ago

It would be untenable to save all command line history since I spend every day in the shell in textmode (not x11) and only occasionally switch to a graphical environment

I don't understand this remark. What has x11 got to do with shell history? My bash appends all commands to .bash_history, immediately, regardless of whether it's run inside a terminal emulator or a VT.

ejoso• 1 year ago

I think they were implying that they generate a lot at the CLI, because they gots no GUI. Their whole day of computing would be pushing into a history file whereas most folk gotta lotta clickin’ happening too. Might be wrong though.

1vuio0pswjnm73• 1 year ago

"I don't understand this remark."

Context switching between textmode and a graphics layer, like x11, is IME time-consuming and relatively janky. Whereas switching from a "terminal window" to some graphical program in a graphical environment like x11 is very quick and relatively smooth. Because I prefer textmode, I stay in textmode and I am not switching to a graphical environment very often, if at all. I am not using graphical programs. Consequently I accumulate an inordinate amount of command line history. Too much to save, IMHO.

As I mentioned, I do not use bash, I use a smaller, faster shell. I have no ".bash_history". And sometimes I use even smaller shells that have no command line history, in order to reduce program size.

Xx_crazy420_xX• 1 year ago

The preference of shell history depends on the use case. In my case it's ever changing projects with different tech stacks that i need to remember set of commands to get SSO to get to container registry etc..

The solution for me was storing bash history separately for each vscode project - https://gist.github.com/Srakai/9f3788cdb07259d65d335ff150c38...

Scripts are cool, but imagine ever evolving script that is controlled by arrows and enter

darkteflon• 1 year ago

TIL that Atuin has a flag that might be useful here: store_failed = false. That looks like a nice low-effort halfway house between active curation and saving everything.

Curate your shell history

Related Stories