The case of the mysterious --help directory (2020)

https://leahneukirchen.org/blog/archive/2020/05/the-case-of-the-mysterious-help-directory.html

by panic • 4 years ago

171 58 4 years ago

leahneukirchen.org

moonbug• 4 years ago

another datum in the "why you should never customise your environment" argument.

ktpsns• 4 years ago

another datum in the "you can fix everything if you can control your environment"

ironmagma3• 4 years ago

another datum in the "never support GNU coreutils"

I jest, with full knowledge of the trouble I'm causing.

mst2• 4 years ago

Get New Utilities is a double edged sword, much like adding layers of abstraction.

Severian• 4 years ago

Actually it should be: "Customize your environment, but take some damn notes FFS."

anyfoo2• 4 years ago

Nah, shell environments are really begging to be customized. Just be mindful of it. In a sense, just installing command line tools is customization already.

userbinator2• 4 years ago

More like "why you should be aware of all the hidden side-effects if you do".

ironmagma3• 4 years ago

Didn't push a fix upstream?

tazjin• 4 years ago

My understanding is that this function was in her dotfiles, so by virtue of fixing it there it was fixed upstream.

ironmagma3• 4 years ago

That isn't what it sounds like to me.

Jannis Harder told me that the zsh completion for mkdir would run mkdir --help (to see if it’s the GNU variant), which I could verify.

Completions are typically installed as an add-on or as part of a core library of a shell.

I found this line, maybe it's the culprit? https://github.com/zsh-users/zsh/blob/master/Completion/Unix...

frazbin• 4 years ago

Yeah I figured it was shell customization gone awry early on, with 'mkdir --' in the mix. But if they'd figured that out immediately it wouldn't have been a fun post; busting out some custom unix plumbing tools and showing how cool they are is really what the article is about. Also gotta admit they have a pretty darn fun github.

wink• 4 years ago

showing how cool they are

That would've been writing a custom kernel extension to track this down, but fortunately tracing exists these days.

Seriously, I don't see anything wrong here, it's about the same route I would've gone down. Except the other way round because writing a custom ~/bin/mkdir would've saved me reading up.

don-code2• 4 years ago

For reasons like this, I almost wish software came with a "list of unexpected side-effects of using this software".

Take bash, for instance - each time you run a command (this list is nonexhaustive; I'd love to know others):

1. If you have a `PROMPT_COMMAND`, it'll be run.

2. The command will be echoed to your history file.

3. If you hit <TAB>, arbitrary functions might run (for completion).

I don't know what zsh may do on top of this, but my guess is that it's significantly more than bash. That's not a demerit to zsh - that's exactly what it should do - but visibility would be nice.

eiginn• 4 years ago

In Zsh setting '-x' on an interactive shell will also show you what is executed for prompts, completion, and any hooks (probably other things too)

gryn• 4 years ago

same for bash

eyelidlessness• 4 years ago

For that matter, it would be nice if software with side effects of this sort disclosed them at runtime by default. For all its faults, this is something npm gets (partly) right. Partly, in this case, because it does print the commands it runs and their output, but it doesn’t tell you why.

I’m sure this would be overly verbose for most users, who would likely turn it off, but it would still be educational in terms of the huge set of implicit side effects normal every day software produces.

0xcde4c3db• 4 years ago

2. The command will be echoed to your history file.

Important caveat: this goes into an in-memory history buffer; the history file is written when bash exits. This means that if you have multiple shells open, you will lose the history from all but the last shell to exit.

theli0nheart• 4 years ago

This is the default, but you can get around this behavior by writing to history immediately.

jefftk3• 4 years ago

Yes! If you want to maintain a full history (and you should!), you can make bash consistently log it:

    promptFunc() {
        # right before prompting for the next command, save the previous
        # command in a file.
        echo "$(date +%Y-%m-%d--%H-%M-%S) $(hostname) $PWD $(history 1)" \
          >> ~/.full_history
    }
    PROMPT_COMMAND=promptFunc

gurjeet• 4 years ago

The history will not be written if either the shell/OS crashes, or if you `unset HIST_FILE` before exiting the shell.

LinAGKar• 4 years ago

No, it will append the newly run commands. Or at least on SUSE it does, maybe other distros configure it differently.

1vuio0pswjnm72• 4 years ago

The .bash_history truncation feature of bash I find quite annoying

I am used to NetBSD's ash where I set HISTSIZE very high in .profile and save history with

   fc -l 0 > .history

With NetBSD I never "lose the history from all but the last shell to exit". I write history files when and where I want them written. (The irony is I am more likely to lose history if using bash, which auto-saves it, than I am when using ash, which does not. As GP suggests, bash is needlessly complicated.)

A large number of GNU/Linux use a fork of NetBSD's ash called dash. It is the installed default scripting shell on Debian, Ubuntu, etc. and the default shell in Busybox, OpenWRT, etc.

https://en.wikipedia.org//wiki/Dash_(shell)

Dash can be compiled with history support using libedit.^1 The static dash binary I use is 448k compiled with musl.

Dash is not as nice as NetBSD's ash for interactive use. For example, there's no tab completion in dash^2, hash -v does not work despite being listed in the manpage; there are various other small differences. In any event, I am using it as an interactive shell.

The Debian project decided dash is faster than bash as a scripting shell. For me, it is also faster as an interactive shell, because I am used to libedit and most of the interactive use I do is editing and running scripts/command line history

1. Instructions

   # install libedit - may already be installed for other programs such as openssh or sqlite
   wget https://git.kernel.org/pub/scm/utils/dash/dash.git/snapshot/dash-0.5.11.5.tar.gz
   tar xzf dash-0.5.11.5.tar.gz
   cd dash-0.5.11.5
   ./configure --with-libedit
   make
   cd src
   gcc -static -Wall -g -O2 -o dash alias.o arith_yacc.o arith_yylex.o cd.o error.o eval.o exec.o expand.o histedit.o input.o jobs.o mail.o main.o memalloc.o miscbltin.o mystring.o options.o parser.o redir.o show.o trap.o output.o bltin/printf.o system.o bltin/test.o bltin/times.o var.o builtins.o init.o nodes.o signames.o syntax.o -ledit -lcurses

2. Mostly I just use globbing instead. If I absolutely must have tab completion, I use tcsh. If I am not mistaken, tcsh was the first shell to offer that feature. I find it is the fastest of any shell when it comes to filename completion

broses• 4 years ago

That reminds me of a function I have in my environment called `dg` that does `cd` then `ls`. The name `dg` is short for dig, since the name `dig` was taken.

I also sometimes write `2>1` instead of `2>&1` and get a mysterious file named `1` in my current directory. Luckily it I figured that one out pretty quick.

tedunangst2• 4 years ago

Why does zsh want to know where mkdir came from?

LukeShu2• 4 years ago

The tab completion sniffs the `--help` output to determine whether it should tab-complete `--long` flags and if so whether it should include `--context` (SELinux) among them.

https://github.com/zsh-users/zsh/blob/master/Completion/Unix...

trashburger2• 4 years ago

Presumably to figure out which options the completion engine allows you to select. Being able to select GNU extensions on a BSD mkdir would be silly.

adhesive_wombat2• 4 years ago

Semi related: if you write software that takes only one of -h or --help, but not both, you are bad and you should feel bad.

scubbo• 4 years ago

Even more so, the worst coworker I ever had once wrote a script that accepted neither, and (when run) would default to trying to process data (and deleting it whether or not it was fully handled) _in production_. The first time we ever ran it (unsuspectingly) was after he'd left for Facebook. Good riddance.

adhesive_wombat2• 4 years ago

Oooo, not failing on unknown arguments is evil. It's almost guaranteed at that point that what you're about to do is not what the user expects to happen.

codeflo• 4 years ago

One place I worked had very explicit rules about that that would be drilled into new recruits. Besides --help, running without argument was also required to print the help.

I was told that this rule came about when they "discovered", presumably at nontrivial cost, a disk imaging tool that would default to unconditionally mirroring the first hard disk onto the second.

tedunangst2• 4 years ago

Not as bad as blindly assuming -h prints help.

trashburger2• 4 years ago

To be honest, it should, or if you feel like your program is very simple then you could do what feh(1) does and simply print "see man $PROGRAM". Running -h or --help on a program which spits out an "unknown argument" error is one of my pet peeves.

MaxBarraclough• 4 years ago

GNU cat is on the naughty list, supporting --help but not -h. less is the same (and is not part of GNU).

edit Looks like the GNU echo, sort, and uniq programs are the same. GNU has an official policy endorsing --help but not -h. [0]

Also related: --version should print the version numbers (and perhaps build details) of the software, and immediately terminate.

-v lacks the (at least somewhat) universal meaning of -h, though. Some commands, such as curl, use -v as shorthand for --verbose rather than --version, but all commands should support --version.

I believe Java HotSpot used to use -version, supporting neither -v nor --version. Now it supports both --version and -version, but not -v.

[0] https://www.gnu.org/prep/standards/html_node/_002d_002dhelp....

benibela• 4 years ago

And do not forget the most important flag:

/?

icambron• 4 years ago

Just today, I deleted a directory called --process in my home. Most likely, I just forgot a -- or otherwise messed up a command. We'll see if it comes back!

userbinator2• 4 years ago

"Extended" tab completion seems like one of those things which sounds like a good idea on the surface, but actually involves considerable hidden complexity, and when something doesn't work right, can instead cause plenty of confusion and lost productivity.

As someone who routinely uses tab completion to look into the contents of directories before entering them to confirm that they are the correct ones, I was very confused when a recent system I had to use briefly (I don't remember if it was zsh) decided to not show files if the command started with 'cd'. It wasted more time trying to be "helpful" because it required an extra 'ls' which I could essentially get for free via tab completion.

enriquto• 4 years ago

Why would you want "cd folder/<TAB>" instead of "ls folder" ?

gmueckl• 4 years ago

If you are navigating a complex folder structure, it's easy to start typing

cd path/to/

and then realizing that you don't remember whether you need subdirectory1 or subdirectory2 next. With tab completion, you can just continue typing

cd path/to/subdirectory1<tab><tab>

to check whether what's in there is what you're looking for. Without tab completion, you need to move to the start of the line, change "cd" to "ls", run that, go back up in the history, change "ls" to "cd" and continue from there. It's much more inconvenient.

avian• 4 years ago

Smart completion is the first thing I remove or disable on any system I work on.

"Complete with a path on <tab>" is simple to understand and reason with and becomes second nature after working a lot in the shell prompt.

"Run a complex set of possibly incomplete/broken rules to come up with a string" is the exact opposite. Commonly bumping at "why isn't this command-line completing/is completing with something unexpected" just negates all benefits of <tab> sometimes working for stuff like git branches, long command-line flags, etc.

bentcorner• 4 years ago

I ran into a similar problem on windows, using process monitor with the appropriate filter made it pretty easy to find the problem. (I had a batch file containing a forgotten copy command getting called as a side-effect of some other tooling on my PC).

joshuaissac• 4 years ago

Indirectly related story from my workplace: the application that I maintain changes some terminal line settings, such that after it is run, we have to use Ctrl-H instead of backspace (unless we also change PuTTY settings to send Ctrl-H for backspace).

This means that sometimes people accidentally create files whose name is a backspace.

These cannot be seen directly as '\127' via ls (or at least the default Solaris ls; not sure what GNU does), but instead as the output missing a character, usually padding, so people do not realise it is a problem.

Then it causes issues when the file gets caught in globs that process all files in the directory, but the offending file is almost invisible, leaving the user confused.

And we cannot change the program not to change terminal settings because who knows what that will break!

account42• 4 years ago

Indirectly related story from my workplace: the application that I maintain changes some terminal line settings, such that after it is run, we have to use Ctrl-H instead of backspace (unless we also change PuTTY settings to send Ctrl-H for backspace).

Things like that is why I use PROMPT_COMMAND="stty sane"

mnw21cam• 4 years ago

The one good thing about having a file/directory called "--help" lying around is that it protects you against running "rm -rf *".

The case of the mysterious --help directory (2020)

Related Stories