Writing simple tab-completions for Bash and Zsh

https://mill-build.org/blog/14-bash-zsh-completion.html

by lihaoyi • 12 months ago

271 96 12 months ago

mill-build.org

lihaoyi• 12 months ago

I wrote this, hope everyone finds it as interesting reading this as I did figuring this out for the first time!

oezi2• 12 months ago

Thanks, for the interesting read.

tetha2• 12 months ago

It's a good first dive into zsh completion. The whole thing is quite the large system to wrap ones head around it and I'm still somewhat struggling.

But at work, I've been slowly adding auto completion to our ansible wrapper scripts, like explanations which playbooks to use when, smart `-l` completion based off a possibly selected playbook (so, if the playbook is postgres.yml, it doesn't suggest mariadb groups), tag autocompletion (with a few, admittedly, hardcoded explanations how these tags should be used) and such.

It's somewhat of a friday-afternoon struggle project, but it's making the big ansible project pretty approachable to use.

imcritic3• 12 months ago

Could you share how you do it? Ansible playbooks are ran via command `ansible-playbook` command and it surely has its own tab auto completion script.

bbkane3• 12 months ago

Thanks for sharing! I hope to incorporate your bash completion ideas into my CLIs (I've already got zsh completions).

Instead of sourcing the zsh completion script on every startup, you can install it into somewhere on $fpath and zsh will "compile" and cache the completions. This can really speed up shell startup time, but of course is harder to set up. Users have to understand $fpath to put the completions there.

I distribute my CLIs via Homebrew which can install completions automatically.

oezi2• 12 months ago

Isn't there a standard flag which programs can implement to avoid writing this bash script?

Ideally this could all be part of a library such as argparse for typical cases, right?

vcdimension2• 12 months ago

In zsh you can use the _gnu_generic function for simple completion of commands with a --help flag. Just put a line like this somewhere in your startup file: compdef _gnu_generic <CMD>

cb3214• 12 months ago

_gnu_generic is fantastic. I use it all the time. If your CLI toolkit emits colorized help, but skips said colorization if NO_COLOR[1] is set then you can prefix _gnu_generic with NO_COLOR=1 as in https://github.com/c-blake/cligen/wiki/Zsh-completion-for-cl... for the cligen Nim CLI toolkit.

A similar thing in Bash is `complete -F _longopt YourCmd`, but these will not work with "multi-commands" that have "sub-commands" as the article of this thread covers. Truth is, as non-standard as GNU long opts already are, how subcommands work is even more non-standard (are there even global options? Or between each subcommand? Is it how Python does it? Or Go? Or some one specific library or ..?)

[^1]: https://no-color.org/

mnahkies• 12 months ago

I've wondered this as well - it would sure be nice if there was a standard --completion or something that common argument parsing libraries could automatically implement for us (much like they often implement automatic help text)

duckerude• 12 months ago

Rust has the clap_complete package for its most popular arg parsing library: https://crates.io/crates/clap_complete

ripgrep exposes its (bespoke) shell completion and man page generation through a --generate option: rg --generate=man, rg --generate=complete-bash, etcetera. In xh (clap-based) we provide the same but AFAIK we're the only one to copy that interface.

Symfony (for PHP) provides some kind of runtime completion generation but I don't know the details.

bravesoul22• 12 months ago

Do many people do bash on osx or zsh on Linux, and would this make much of a difference?

homebrewer2• 12 months ago

I don't know about "use" — luckily, there's no opt-out telemetry — but enough of "enthusiast distribution" users who have also opted in (very biased sample) have explicitly installed zsh (not necessarily run it)

https://pkgstats.archlinux.de/compare/packages#packages=bash...

OTOH, it's only 4-7% on Debian (also opt-in):

https://qa.debian.org/popcon.php?package=zsh

bravesoul22• 12 months ago

I use zsh at work and bash at home. I am such an unsophisticated user that I haven't noticed a real difference! Other than I can install ohmyzsh on zsh.

yonatan8070• 12 months ago

Why does this type of statistic require an opt-in solution? Can't the Arch mirrors just tally requests for each package without identifying information like IP adddress?

mdaniel• 12 months ago

Regrettably due to the licensing stupidity there's actually two bash on macOS: /bin/bash and $HOMEBREW_PREFIX/bin/bash they do differ (as is expected by a two major version spread)

I'm firmly in the never-zsh camp, but I'd guess so long as one doesn't do something silly like use #!/bin/zsh in shared scripts you can continue to use whatever interactive shell you like

I had a college who was hard core into fish but had to be completely self sufficient because he was the only user in the office. I presume it's the same as being on Linux in a Windows shop or being on Windows in a macOS shop: no one has the day to day experience to have the answers handy

e40• 12 months ago

Homebrew bash is my default shell on macOS.

vcdimension2• 12 months ago

Here's another tutorial for creating zsh completers using the built-in functions: https://github.com/vapniks/zsh-completions/blob/master/zsh-c...

wiseowise• 12 months ago

Shell syntax is the exact reason why we've needed LLMs in the first place.

camdroidw5• 12 months ago

You mean Unix shell syntax. Powershell has got this absolutely right ,and only this (which is probably still a 50% of what a shell is)

pastage2• 12 months ago

Can you expand on why it is good I have never read any good evangelisation for it. I can not stand PowerShell the syntax is backward, least important information first. It is also very slow for me but I have understood that it is a pebkac issue. So I am open to be corrected.

anthk• 12 months ago

Perl solved that 20 years ago.

And `rc` under plan9/9front did a Unix shell better than the classic Unix itself.

homebrewer2• 12 months ago

With fish, if the program you're interested in hasn't betrayed the decades-old tradition of shipping man pages, it's often as simple as running `fish_update_completions`.

It parses all man pages on your system and generates completion files for you. By default, they go into ~/.cache/fish/generated_completions/*

If the man page was written poorly/is missing, you can always write your own completion (and hopefully send it upstream). fish uses such a simple format that I don't think there's any need for tutorials save the official doc:

https://fishshell.com/docs/current/completions.html

For example, here's an excerpt from curl

  complete --command curl --short-option 'L' --long-option 'location' --description 'Follow redirects' 
  complete --command curl --short-option 'O' --long-option 'remote-name' --description 'Write output to file named as remote file'

btreecat• 12 months ago

When I screen share, people don't realize I'm not using zsh and dozen plugins. It's just fish and it's beautiful out of the box.

kekebo• 12 months ago

Thank you for the comment. https://github.com/umlx5h/zsh-manpage-completion-generator appears to adapt this to ZSH. Have yet to try through

cb3214• 12 months ago

For those programs that have betrayed shipping man pages, instead say relying only on a --help system, do you happen to know if the fish shell has an analogue to Zsh `_gnu_generic` and Bash `complete -F _longopt`? If not, do you have any insight into why not/what it would take to make that happen?

nikita2206• 12 months ago

The OP mentions them in the last part of their comment, there is `complete …` commands for registering completions

mr_mitm• 12 months ago

At least for a subset of Python CLI programs, I wrote this: https://github.com/AdrianVollmer/pycompgen

Still in an early stage, but it should work.

jnpnj• 12 months ago

oh wow, it's parsing all 9461 man pages on my arch install, for a cute total of 13MB

thanks a lot

sanewombat• 12 months ago

It's surprising that on OpenSUSE `zypper search fish-completion` returns more than 200 packages. Something is fishy here.

OptionOfT• 12 months ago

That is to get live data in your completions.

Say you have a widget that has 3 commands: list, start and stop.

With one of those completion files widget stop <tab> will show you a tab-able list of widgets which are running.

IgorPartola• 12 months ago

man pages are so underrated. I mean every project nowadays has README.md so I don’t see why we can’t just auto generate them with or without an LLM helping. Also I wish programs would use standardized generic arguments for help, config file, version, background the task, PID file, log file, and log level.

yencabulator4• 12 months ago

I'll switch to fish after it stops expanding `car TAB` to `blkdiscard` when I don't have `cargo` in path. Non-prefix completion for commands is plain evil.

dingnuts• 12 months ago

I bet this is configurable but I wanted to say that this is totally personal preference; I have the exact opposite opinion. Prefix only matching requires much more tab slapping in my experience.

wpm4• 12 months ago

I’ll switch to fish when it comes preinstalled on all of the computers I use so I can write scripts in it.

fishywang• 12 months ago

I tried to use fish on some of my debian servers that i only rarely update packages/kernel, so I don't have to carry my bashrc there, but found their completion for apt is pretty naive. for example after updated kernel i would want to clean up the old ones, but `apt purge linux-image-<TAB>` would list all available kernel versions, not just the ones currently installed.

in the end i switched back to bash.

hnlmorg• 12 months ago

Fish isn’t the only shell that does this. Murex has supported man page completions for years too. And it’s fully automatic in Murex.

camdroidw5• 12 months ago

Why doesn't someone (not me) just build a basic DSL and a transpiler that does this?

cb3214• 12 months ago

The answer to your question is that command-lines have a much larger diversity of syntax (even to get help!) than most people realize. Folks have their 30..60 commands they run frequently and don't run into many or conveniently forget/neglect older ones like `gcc` or `tar` or `dd`. Many people (not saying you specifically) do not even realize that double-dash long options are a GNU extension never standardized or that Python toolkits typically allow --my-opt for --my-option abbreviations, just to name a couple of the dozen variations (space or '=', or ':' or '/' or any of the above or etc., etc.). There are probably hundreds if not thousands of syntax possibilities, but people often act like there is only one.

As an example of diversity estimation that you can try at home, a couple of times I have run every single command in my command search PATH with --help </dev/null >/tmp/help.$c 2>&1 . Caution - be careful if you do this! Have backups/checksums of everything important and run as an unprivileged user. I always have to kill off several processes that just hang doing something or otherwise manually intervene. Anyway, this alone suggests data collection of help text is not a trivial problem.

Beyond data collection, many commands did not/do not use CLI toolkits at all. Their commands may have even less regular syntax. Freeform help makes it harder to produce a regular help syntax to convert into the interpreter needed by a completion system. That said, as elsethread commented for some toolkits the Zsh _gnu_generic works great! It essentially IS the "automagic" system you might want, just for a highly restricted circumstance.

Any CLI toolkit itself does have the data, by necessity. So, if the CLI framework supports the 2 or 3 common shells there is no need for a translator exactly. You just need a code generator. There is a stab at an auto-generation framework from said data for the Nim CLI toolkit, cligen, over at:

https://github.com/c-blake/cligen/blob/master/util/complgen....

but it only works for Zsh right now. Anyway, I don't think perfect should be the enemy of the good or anything like that, but you seemed to ask an earnest "why" question and these are some of the complexities.

camdroidw5• 12 months ago

Quickly skimming your code it seems like that's what I was asking for? basically you enter your auto completions without knowing the bash syntax?

As for the 30..60 commands, I use the following "tricks"

1) my ctrl-p is mapped in a way that it searches the prefix into history. (On phone right now)

2) long history which syncs with other machines in a machine-name-suffixed file via syncthing

3) justfile which are really a game changer. I used bare j to tell me what commands are available (as opposed to running the first command) and I basically know everything I can and do do in that folder

bbkane3• 12 months ago

People do!

See: https://pixi.carapace.sh/ or https://github.com/withfig/autocomplete

It's still a hard problem as lots of tools format --help differently. One of the things I'm jealous in Poweshell is their standardized completions

camdroidw5• 12 months ago

Yes! I posed about pwsh here https://news.ycombinator.com/item?id=44860445

derriz• 12 months ago

I feel that the ergonomics of bash completion took a hit as the configurations got “smarter” and “helpfully” started blocking file or directory name completion if it thinks it wouldn’t be appropriate to have a file name at the current cursor position. Instead of blocking, the default should always be to fall back to filename completion.

Sometimes I’m close to disabling/uninstalling all completion scripts out of irritation as decades of muscle memory are frustrated by this behavior.

It’s like that bad/annoying UX with text fields where the UI is constantly fighting against you in order prevent you from producing “illegal” intermediate input - e.g. let me paste the clipboard here goddammit - I know what I’m doing - I’ll correct it.

cryptoz• 12 months ago

I have come to absolutely despise web form inputs with front end email validators that are broken. Input field hints to type your email, so you start typing. As soon as you type the first letter it goes red and says “error!!! Invalid email!”

Unbelievably frustrating.

pastage2• 12 months ago

As a guy who is vicously guarding our company email valdation I can tell you that it is a rite of passage for new frontend hires to mess that up.

compressedgas• 12 months ago

As it works as desired after running: complete -r; there is something broken about the bash-completion script.

IshKebab• 12 months ago

I agree that is annoying. It's waaay less confusing to complete a filename and then get an error from the actual program than it is for just ...nothing to happen so you get confused and have to `ls` to check if the file actually exists and it does and so you think tab completion is broken for some reason and you copy & paste the filename and then finally you get the error that explains what is going on.

It should at least print a message like "file foo.exe exists but it isn't executable".

kevindamm• 12 months ago

If you get into this position what you can do is `ls <tab-completed-path>` or other command to put the filename in the previous command's argument, then you can access it via !$ or !^ (or use !!:1 or your shell's notation for indexing an argument that was already in the previous command).

It's not a fix but it'll save a little time sometimes.

loeg• 12 months ago

There is one particular command I occasionally use that has totally broken completion for files, so I've taken to just using 'ls X Y Z' to get the right completion behavior and then changing 'ls' to the right command as the last step.

jmholla• 12 months ago

Hopefully you saw one of the sibling comments: https://news.ycombinator.com/item?id=44855551

TLDR: M-/ (Alt /) will do file auto-completions regardless of the commands auto-comoletion. It is a different method, but maybe could help.

JNRowe• 12 months ago

There is the complete-filename function that only completes filenames in bash, bound to M-/ by default. You can use that in any place you want a filename where "complete"(the function normally bound to tab) would do something you don't desire.

There are a collection of other non-context aware completion functions that are bound by default too, useful for example when you when you wish to complete hostnames in a for-loop.

zle has what is largely a significant superset of this, the documentation is spread about between the zshzle and zshcomp* manpages.

vbezhenar• 12 months ago

Here's zsh snippet I've came up with for my own simple functions. I'm using it as a base for other completions. In this example, function `set-java-home zulu-21` sets JAVA_HOME to `~/apps/java/zulu-21`. Here's `_set-java-home`:

    #compdef set-java-home
    
    local -a versions=(~/apps/java/*(:t))
    _describe 'version' versions

So basically almost a one-liner (but couldn't do it really one-liner, unfortunately).

medv• 12 months ago

JSON fields autocomplete right in bash/zsh: https://fx.wtf/install#autocomplete

imcritic3• 12 months ago

Thanks for linking this! This is a lightweight solution, compared to ijq (interactive jq), but it still may come in handy.

https://github.com/gpanders/ijq

xenophonf3• 12 months ago

I wish tcsh would get more love.

esafak• 12 months ago

Why, it's a dinosaur? Have you tried nushell, murex, oil shell or xonsh?

xenophonf3• 11 months ago

I don't know why you're being downvoted. You're correct, and as the other commenter mentioned, csh programming is evil in the eyes of the LORD. Starting last year, I've been maintaining a bash profile that implements (mostly) the same setup as my tcsh configuration. I suppose I ought to make the final leap into something more modern. It would be nice not to jump through hoops to redirect stderr somewhere different than stdout or [insert your favorite of the myriad examples of csh/tcsh weirdness here]! But I feel irrationally sad about it. I cut my teeth on Unix systems where csh was the default shell—namely, NeXTstep and sunOS 4—and it feels a little like saying the final goodbyes to a dead friend. Silly of me, I realize, but there it is.

chasil3• 12 months ago

There is a famous paper on the perils of scripting in the csh. It is unfortunate that Bill Joy was not able to write a formal grammar or parser for his language. It was certainly a missed opportunity, and tcsh cannot fix the design.

That being said, csh advocates definitely influenced everything in the Bourne/POSIX family.

xenophonf3• 11 months ago

It's a classic Usenet post that deserves linking at every opportunity!

https://harmful.cat-v.org/software/csh

sebtron2• 12 months ago

Basic completion in ksh is as easy as defining an array. From https://man.openbsd.org/ksh :

Custom completions may be configured by creating an array named ‘complete_command’, optionally suffixed with an argument number to complete only for a single argument. So defining an array named ‘complete_kill’ provides possible completions for any argument to the kill(1) command, but ‘complete_kill_1’ only completes the first argument. For example, the following command makes ksh offer a selection of signal names for the first argument to kill(1):

    set -A complete_kill_1 -- -9 -HUP -INFO -KILL -TERM

chasil3• 12 months ago

Is this in the Korn & Bolsy ksh88 book?

Or is this ksh93 syntax that oksh back ported?

sebtron2• 12 months ago

I don't know actually. Are there currently used version of ksh that are not derived from ksh93?

gosub100• 12 months ago

I did something similar to this for tab-completing server names for use with ssh. I went a step further and allowed pattern matching based on the server being qa/prod and on locale. so for instance you could type `ssh co prod <tab>` and it would tab-complete / suggest any servers that were production and located in the Denver datacenter (co is the state abbrev for Colorado, for non-US readers).

Unfortunately my work doesn't allow me to share code, but essentially I remapped ssh to a bash script that maintains an environment variable containing the args (you must do this because each <tab> press is an independent invocation of the script. Then you run into persistence problems, so I added a call to compute elapsed seconds so that it flushes the state variable after a 10s timeout).

The bash script then forwards the args to a python script that reads a JSON file and figures out which params (such as 'co' or 'qa') map to which hostnames. It also matches against partial hostnames, so when you see this after tab

qa-server-co1 qa-server-co2 pr-server-co3

you only need to add '3' to the list of args to narrow it down to 1 match, then hit <enter> to ssh to that host.

paradox4602• 12 months ago

I've started using jdx's usage[1] for my clis. It integrates neatly into clap, and can be used stand alone in scripts. It can generate completions, argparse, manpages, and more

I'm still on the fence if replacing the argparse blocks in my fish scripts is worth the hassle, but against things like old school optparse, it's far better

[1] https://usage.jdx.dev/

FujiApple• 12 months ago

I’ve recently been building a similar tool [1] which defines a specification for CLIs, though the goals are slightly different to the tool you mention I think. I just added support for fish as it happens.

[1] https://github.com/fujiapple852/claptrap

paradox4602• 12 months ago

Nice. I'll have to add that to my tool belt, although I think usage will still sit front and center haha

vismit2000• 12 months ago

Since auto text completion is the primary task of an LLM, I have seen GitHub copilot do that very well on both bash and zsh within vscode terminal!

VagabundoP• 12 months ago

zsh standard tab completion drives me batty. But I come from bash where 99% of the time I just want filename completion.

Writing simple tab-completions for Bash and Zsh

Related Stories