test, [, and [[ (2020)

https://jmmv.dev/2020/03/test-bracket.html

by mattrighetti • 3 years ago

548 225 3 years ago

neverrroot• 3 years ago

[[ is bash only. If you know you’ll only use bash, use it. For details, the article is nice.

nerdponx• 3 years ago

That is, test and [ are specified by POSIX and usually are physical binaries (but might also be masked by shell builtins). Whereas [[ is not specified by POSIX and usually only exists as a shell builtin.

shmerl7• 3 years ago

Why would you not use Bash (unless you are using completely different shell like Fish explicitly?).

It feels like some completely archaic concern to target the minimal common shell denominator.

LambdaComplex• 3 years ago

Not installed by default on BSDs. Sometimes not installed in minimal environments, e.g. where BusyBox is all you have available. Maybe not installed by default on Unixes? Not sure

mikem170• 3 years ago

Bash seems to be about as archaic as sh.

I write my shell scripts in sh, because it comes with every unix-like os by default.

I know that bash has more features, but I've never really missed them. I switch from sh to perl for more complicated tasks.

To each their own, right?

yjftsjthsd-h5• 3 years ago

IIRC Debian uses dash as /bin/sh because it's faster (execution speed) than bash.

driggs• 3 years ago

Because Apple switched the default macOS shell from the GNU-licensed `bash` to the BSD-licensed `zsh`?

aidenn02• 3 years ago

IIRC, /bin/sh on Ubuntu defaults to dash

JohnFen4• 3 years ago

I've worked on several modern projects that needed to be able to run on a wide variety of Unix platforms, several of which didn't have bash. Writing for the common shell denominator was important, not archaic.

csydas• 3 years ago

it’s contextual as most things. need to actually use and work on the machine? use whatever shell you want to make your life easier.

need a script to run on many different systems and/or need to write a script to be managed automatically by a service account? probably you want a shell with syntax that is guaranteed to be the same on all your systems.

dredmorbius2• 3 years ago

Writing Bourne-compliant scripts ensures maximum portability.

As many here have noted, bash isn't universally available, with another possible issue being OpenWRT devices. Stock/base images tend to use a Bourne-compatible shell, not full Bash. Though the latter's installable through opkg, for sufficiently small devices (typical of consumer kit), you simply won't have the space to install them.

There's also the slight PITA that Apple's OSX ships with a very old, pre-GPLv2 Bash, out of licensing concerns. (Apple is phenomenally averse to GPL-based code, much as some *BSDs are, such as OpenBSD.)

And if you're dealing with legacy systems (which tend to be extraordinarily and stubbornly persistently legacy), you'll often find that either bash isn't present or is quite dated.

I freely confess that I tend to write fairly recent-feature bash scripts myself by default, and appreciate many of the newer features. But if and when I am writing portable code, I'll go back to Bourne-compatibility.

But when writing system level code, an appreciation for standards and the very long tail of legacy standards and the limitations they impose is in fact a large component of professional maturity.

bregma• 3 years ago

It's archaic if you're a dabbler, a hobbyist, an academic, or a junior dev who lacks the experience of shipping software into the wild.

For the rest of us, we have learned the hard way, sometime repeatedly, portability and adherence to published standards matters.

re• 3 years ago

Well, zsh too :)

https://zsh.sourceforge.io/Doc/Release/Conditional-Expressio...

bgm19752• 3 years ago

And ksh (ksh88 & later)

dredmorbius2• 3 years ago

"bash only" typically refers to "bashisms", that is, bash features not present in the plain Bourne shell (or Bourne-compatible interpreters such as dash). The fact that other shells (such as zsh) may include those features ... is beside the point of writing universally compatible shell scripts.

Confirming my facts for this comment, #TIL that "dash" is the "Debian Alquist Shell", that is, Debian's "ash" shell:

<https://en.wikibooks.org/wiki/Guide_to_Unix/Explanations/Cho...>

emmelaich5• 3 years ago

Always use [[

zsh and ksh have it; in fact I'm pretty sure it originated with ksh in 1988 or earlier.

OmarAssadi2• 3 years ago

Always use [[

While zsh, bash, mksh, ksh93, probably others have it, sure. But many don't -- and not totally irrelevant ones either. Debian's default, dash, for example, does not support `[[`.

IMO, unless you're writing something like shell-specific dotfiles, avoid non-POSIX features.

It's usually pretty trivial to avoid them, especially if you're willing to call other mandated commands like awk, etc. But often, with a bit of creative thinking, most non-standard features can be replicated with some combination of `set`, separate functions and/or subshells.

Shell scripts, in general, have dozens of footguns, are pretty much impossible to statically analyze, difficult to make truly robust, and many of the shells themselves -- e.g., bash -- have huge, borderline inauditable codebases.

I can think of a dozen reasons not to write shell scripts. Yet still, there is incredible value in the fact that some form of POSIX-compliant/compliant-enough shell can usually be found on most systems.

All of that value goes out the window, though, the moment you start relying on non-standard features.

metadat• 3 years ago

I've learned to only use [[ when I want to do a regex match, e.g.

  if [[ "${foo}" =~ ^bar$ ]]; then echo Yes; fi

Otherwise, just stick with "test" or "[".

85,000 lines of bash and counting... Not saying bash is great, but it's still meeting my needs for a lot of stuff. Shrug.

cvccvroomvroom• 3 years ago

That's a pointless example.

    [ "$foo" = bar ] && echo Yes

For substring matches, [ and * globs are generally good enough.

    [ "$bar" = extra* ] && echo '$bar began with extra'

Bash's regex dialect is primitive and rarely worth fussing over. For anything complicated, use another tool be it grep, awk, perl, or such. There are diminishing returns of obsessing over doing everything in bash when a complicated task demands more capabilities suitable to another tool with greater reusability, modularity, and intrinsic types.

aidenn02• 3 years ago

I thought regex inside [[ was the same as egrep?

stephenr2• 3 years ago

If you want to do (relatively) simple pattern matching in a posix-compliant way, `expr`[1] can match BREs, and even return a capture group.

1: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/e...

somat3• 3 years ago

As someone who enjoys shell scripting I don't think I have ever used the "expr" command. this astonishes and baffles me because I used to read the man pages for fun, I even had a little script to pick a random page. So thanks a bunch for bringing it to my attention.

My only guess as to why I am unfamiliar with the command is that perhaps younger me failed to figure out "expression" means regular expression. It only clicked this time because of your comment and the see also: re_format link at the bottom.

http://man.openbsd.org/expr

penguin_booze• 3 years ago

The fact that the regex is written bare and unquoted, got me a while ago. I'm someone who religiously quotes everything; it took me a while before I figured out why my stupidly simple regex doesn't match.

shmerl7• 3 years ago

And now for the final lolz. I’ve said above that these are the commands you use to evaluate expressions… but the shell also has expressions of its own via the !, &&, and || operators—all of which work on command exit statuses.

It works for simulating very basic boolean expressions, but it gets ugly quickly if you need some more complex combos of NOT, OR and AND. I wish Bash had proper boolean expressions support.

remram• 3 years ago

What do you mean? Bash does have (), {}, !, &&, ||

What else do you need for "proper Boolean expressions support"?

shmerl7• 3 years ago

I don't think you can assign boolean variables to boolean expressions, becasue there is no boolean type.

Something like this won't work to become false:

    foo=true
    bar=false

    baz=$foo && $bar
    echo $baz

Or even simply:

    foo=! $foo

With numeric expressions, you can at least use $((...))

candiddevmike• 3 years ago

It works fine, you can satisfy your boolean logic needs with bash tests and exit codes.

You'd be amazed at how much of the world runs (just fine!) on bash conditionals.

hyperhopper• 3 years ago

You'd be surprised how much of the world runs on cobol and Fortran.

Doesn't mean those are good or even okay or that people should choose to use them.

jmmv4• 3 years ago

Hey, original author here. Thanks for sharing this and making it rise to the front page! :) By the way, the title probably deserves a (2020) and it would be nice if "test" wasn't capitalized, because it actually refers to the command.

Here is something related from 2021 that also touches on bash's [[ operator and that I think you might enjoy in this context: https://jmmv.dev/2021/08/useless-use-of-gnu.html

o11c• 3 years ago

[[ is not really a builtin, it's fundamentally syntactical (but presumably uses a mostly-inaccessible builtin internally). Fun fact, `]]` is also a reserved word despite never being allowed in a context where reserved words matter.

In some non-bash shells, the `function` keyword is needed to declare certain types of function.

For make `$(shell)`, if you're building a lot of targets the performance difference can be measurable. Still, it loses in the nop case, so you should actually usually do `include` to trigger re-making.

GNU is completely right to ignore POSIX, since POSIX is not useful for solving most real problems.

account425• 3 years ago

In some non-bash shells, the `function` keyword is needed to declare certain types of function.

In shells that aren't POSIX-compliant [0], maybe. In which case: yes, there are many wildly different scripting languages with REPLs.

[0] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...

pmarreck2• 3 years ago

since POSIX is not useful for solving most real problems

POSIX, being a standard that can be tested, is at least a specification or agreement that can be met by multiple products to enable them to interact

dang• 3 years ago

Ok, I've lowercased the leading 't'. We never do that but for this, ok :)

edanm• 3 years ago

As per the Zen of Python:

Special cases aren't special enough to break the rules.
Although practicality beats purity.

gnfargbl• 3 years ago

This article complains that using the extended GNU features (--ignore-case, set -o pipefail etc) makes scripts less portable. Fair enough.

What it doesn't explain is why a Linux user should much care about portability. OpenBSD and FreeBSD are alive and well, but the number of users seems so small that they aren't a particular concern. Maybe you could argue that we "should" consider these OSes out of a sense of fairness, but where does that stop? Do I also need to consider something obscure like vxWorks?

BusyBox (Alpine) is more interesting, but the changes there are so significant that a port will almost always be needed anyway.

Are there other compelling reasons to care about the non-GNU ecosystem?

kqr7• 3 years ago

Is your comment best read in Internet Explorer 6?

----

What I'm trying to say is that open standards (and the portability that comes with them) is not something that just happens on its own. It takes active maintenance, and part of that maintenance is opting to adhere to the standard even when it would be more convenient to use extensions available in the most popular systems.

Will you personally suffer from liberally using Bashisms? Not in the first order. But if we encourage that sort of thinking as a rule, the standards become meaningless. I believe that would be a net negative change for the world, but there are many intelligent people who would disagree.

chasil• 3 years ago

You will notice that the parent article mentions "dash."

The dash shell is small and fast, but it does not allow any bash/korn language extensions beyond what was recorded in the POSIX.2 standard in the early 1990s.

Linux users should care because the Debian/Ubuntu family use dash as the system shell, so this problem is very real as many have learned.

pjungwir• 3 years ago

I want to agree, but macOS is a big one. For example `sed -i` has bitten me before.

sundarurfriend• 3 years ago

I find the obsession with shell-script portability in contexts where it doesn't matter to be bizarre, but this particular argument is amusing:

the number of users seems so small that they aren't a particular concern. Maybe you could argue that we "should" consider these OSes out of a sense of fairness, but where does that stop?

This is the same argument that many companies and products have made over the years (and still do at times) for ignoring Linux. To have a Linux user use that same argument against OSes with even smaller userbases is kind of amusing to see.

jonhohle• 3 years ago

`-o pipefail` isn’t a GNU extension or even a difference in user space programs, but specifically a bash option. The portability is between shells, not operating systems. It will work on bash, regardless of the underlying OS, but may or may not work on csh, ksh, fish, etc.

jmopp• 3 years ago

Mac's command-line tools are the BSD versions. I always get confused about the command-line arguments for `date` because they differ so much betwen platforms.

account425• 3 years ago

A lot of the GNUisms described in https://jmmv.dev/2021/08/useless-use-of-gnu.html are very useful in interactive use. Search the current directory without an explcit . - useful. Append an option to a command you just typed - heck yeah, always annoyed when commands don't support that.

For scripts though, sticking to POSIX sh often makes sense, yeah. You should at least be aware if you use of Bash-isms.

a13692099933• 3 years ago

https://jmmv.dev/2021/08/useless-use-of-gnu.html

Tangential, but your blog software seems to have mangled the headings; it's serving eg:

  make $(shell &amp;mldr;) expansion

instead of

  make $(shell ...) expansion

(note that (as is correctly written in the body) that's three pediods, not one elipsis, so mldr wouldn't be correct even on it's own, meaning there's two probably-unrelated bugs affecting it).

jmmv4• 3 years ago

Hmm… I’ll have to check later but right now I’m seeing the right thing in iOS. The blog is built by Hugo, so it’s all static files. But maybe something changed with the latest update. Thanks.

hwc• 3 years ago

    if [ a = b ] || grep -q ^hello$ /usr/share/dict/words; then
      echo "test failed and grep succeeded"
    fi

> “You pick whether to be amused or horrified. I don’t know how exactly my coworker reacted when I hinted at this during a recent code review I did for them.”

Isn't that normal everyday shell use?

OJFord• 3 years ago

The only thing that might 'horrify' me in code review for that is the assumption that it only evaluates true when test failed.

(And I might quote '^hello$' personally just on principle for having special chars, especially dollar, and to help syntax highlighters.)

yjftsjthsd-h5• 3 years ago

I actually thought [ and test were symlinks, but that might just be from Alpine Linux where they're both symlinks to busybox.

somat3• 3 years ago

Hardlinks actually, but who's counting? plus in real world usage you will hit the shell builtin anyway.

  ls -li /bin/\[ /bin/test
  26016 -r-xr-xr-x  2 root  bin  133256 Mar 25  2023 /bin/[
  26016 -r-xr-xr-x  2 root  bin  133256 Mar 25  2023 /bin/test

I have to admit I avoid "[" in my scripts, it is a weird hack trying to make a command look like syntax and this really bothers me for some reason.

yjftsjthsd-h5• 3 years ago

No, Alpine uses symlinks:

    $ docker run --rm -ti alpine
    / # ls -l /usr/bin/test /usr/bin/[
    lrwxrwxrwx    1 root     root            12 Sep 28 11:18 /usr/bin/[ -> /bin/busybox
    lrwxrwxrwx    1 root     root            12 Sep 28 11:18 /usr/bin/test -> /bin/busybox
    / # ls -li /usr/bin/test /usr/bin/[
     495254 lrwxrwxrwx    1 root     root            12 Sep 28 11:18 /usr/bin/[ -> /bin/busybox
     495360 lrwxrwxrwx    1 root     root            12 Sep 28 11:18 /usr/bin/test -> /bin/busybox
    / #

chatmasta3• 3 years ago

What bothers me is that you can write equivalent code with `test` and `[`, but when using `[` you need to terminate your conditional expression with `]`. Why? Isn't it the same binary? Why is the shell adding this extraneous requirement, just to surface "syntax errors" rather than simply treating `]` as a no-op?

mmphosis• 3 years ago

They are different binaries on Linux Mint!

  # file /bin/test /bin/[
  /bin/test: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=6fe552c80ab0b3d3e60de2ab09167329e222eb67, for GNU/Linux 3.2.0, stripped
  /bin/[:    ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=99cfd563b4850f124ca01f64a15ec24fd8277732, for GNU/Linux 3.2.0, stripped

  #/bin/test --version
  #/bin/[ --version
  [ (GNU coreutils) 8.30
  Copyright (C) 2018 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.

  Written by Kevin Braunsdorf and Matthew Bradburn.

Built with LBRACKET or not ...

https://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;...

ridiculous_fish• 3 years ago

The biggest footgun in `[` and `test` is the single argument behavior. For example, you might attempt to check if a variable is nonempty like so:

     [ -n $FOO ]

but if FOO is unset, it expands to nothing (as opposed to the empty string), so this is equvalent to:

     [ -n ]

and POSIX requires that the one-argument form of `[` succeed if that argument (here, "-n") is non-empty. So this will falsely report that $FOO is non-empty.

Remember to quote your variables!

whateveracct• 3 years ago

ShellCheck your scripts!

mr_toad• 3 years ago

Or use set -u to fail early.

Or use [ -n ${FOO-} ] which will replace the unset variable with an empty string.

dixie_land• 3 years ago

[ x"$FOO" != x"" ]

andrewshadura2• 3 years ago

No. This hasn't been necessary for decades.

overtomanu• 3 years ago

whoa, lot of history in this trick

https://news.ycombinator.com/item?id=26776956

stouset• 3 years ago

Please stop with tricks like this and just quote your variables. Everywhere.

mattrighetti• 3 years ago

In this case I would use [ -n "${FOO?}" ] so that the script will immediately stop if $FOO is null or unset

8n4vidtmkvmk• 3 years ago

Set -xufo pipefail

Or whatever the magic string is. Enable all the errors at the start of the script

jstimpfle• 3 years ago

I think your last sentence needs to go first. Quote your variables! There's no actual footgun in the specification of the test builtin -- the footgun is shell itself. The behaviour that you mention makes sense because

   [ "$FOO" ]

is always the non-empty check regardless what it contains (could be "-n").

gpvos• 3 years ago

Also, if $FOO contains a space, it will expand into multiple arguments. Just quote your variables, always.

JNRowe• 3 years ago

chubot has written an interesting document¹ exploring more of the nuance with test/[/[[, and many of the other entries in that blog have intriguing explanations of the oddities of our shells(a random example²).

¹ https://www.oilshell.org/blog/2017/08/31.html

² https://www.oilshell.org/blog/2016/11/18.html

cellularmitosis• 3 years ago

I stopped using [ a few years ago because ‘test’ reinforces the idea that this is just a command like any other, not syntax. Also, “man test” is much more pleasant that sifting through “man bash”.

kazinator9• 3 years ago

Your comment makes no sense. GNU Coreutils has a "man [" as well as "man test" man page.

Bash has "help test" for a quick cheatsheet.

The [ command is very old; it was already present in Version 7 Unix in 1979.

c0l0• 3 years ago

Parts of the comment make a LOT of sense actually, when you look at shell scripts written by the uninitated. Often times, I see constructs like

    while [ 1 ]; do ...; done

which are a pretty clear indication of the author's misconception about the perceived nature of [ ], I think.

Karellen• 3 years ago

I think GP's point was that `[` feels like syntax, but - importantly - isn't.

Yes, `[` is a command, and has a man page and everything, but in a script it doesn't look like a command. It looks like something "special" is going on.

Whereas using `test` emphasises the point that you're just running another program. That all `if` does is check the output status of whatever program is being run, whether that's `test`/`[`, or `grep`, or anything else at all.

(Personally, I don't think that emphasis is necessary. But I've been using shell scripts long enough that I understand these nuances fairly well without having to think about them much any more. So I think that GP's point is a reasonable perspective to have, and worth considering.)

rascul• 3 years ago

Note that bash implements [ as a builtin so the coreutils man page might not match exactly.

tuyiown4• 3 years ago

I'm with you, the [ (and [[ bashism) introduces lots of confusions about what is really happening, I could never manage any real confidence.

That said, [[ being guaranteed to be built-in certainly had its purpose at ages where shell script performance had any kind of relevance, and that was no so long ago.

kqr7• 3 years ago

[[ has more to do with trying to build an intuitive shell scripting environment than performance. [[ makes conditionals behave much more like you'd expect from other programming languages. I think it's a great idea, but then again, if I don't have to care for POSIX portability, I'd rather use something that's not a shell language for scripting.

paiute• 3 years ago

I’ll probably just use test after reading this, I write bash scripts infrequently enough that if statements always get me (whitespace). Now that i see the why it’s super obvious, and using test helps communicate that it’s just args

cduzz3• 3 years ago

I have strong opinions about shell and they don't actually line up with the rest of the world...

I believe that [ should never be used, only "test" because [ gives the illusion that the mechanism is some part of the language syntax when it is just another "program" (I'm including built-ins and functions in "program").

(if / || / && look at exit status; a program can't see the exit status of other things other than looking at the magic variable $? which is just another string once expanded; case looks at strings but doesn't operate based on exit status and doesn't set an exit status as part of the case ... esac operation; "programs" set an exit status)

I also believe that [ / test should only ever be used for evaluating filesystem constructs -- test -f /dev/null and if you've got string evaluations use case.

Unsurprisingly, most scripts make me itchy, and scripts that I write people find weird.

[edited to add the explanation of "program" vs "syntax" opinion]

cduzz3• 3 years ago

Joke's on me; that's the point of the article.

I prefer, when writing in shell, to do this:

    if 
      program
    then
      something
    else
      otherthing
    fi

To emphasize that the thing after the if is just "look at the exit status of the last command before the then."

    if
      ls /tmp/goober
      test -d /tmp/goober
      echo "I'll always execute the then clause because echo will always return a 0 return code"
    then
      echo "this always gets run"
    else
      echo "this never gets run"
    fi

because the "if" is just looking at the exit status of "echo".

[edited to do code blocks]

tpoacher• 3 years ago

largely agree, but stylistically speaking I prefer the following:

  if   ls /tmp/goober
       test -d /tmp/goober
       echo "I'll always love you or whatever"
  then echo "the grasshopper always jumps higher"
  else "never gonna let you down"
  fi

i.e. all the commands line up at the 6th column, leaving the if/then/elif/else/fi stuff appearing on the 'margin', as it were.

This works particularly well, visually, when nested ifs are involved.

chatmasta3• 3 years ago

Hello fellow traveler. I've got 8 years of scripts with `test` in them to prove I agree with you. It's a lonely road out there... I blame the Google shell style guide.

I picked up the habit of preferring `test` when I was writing scripts that needed to run in both `sh` and `bash`, but I kept it because it makes more semantic sense to me than treating a character like `[` as a command. It's also weird that `]` is an argument to `[` rather than also being a binary. I mean, I understand the technical reasoning... but it feels like a hack.

js23• 3 years ago

Hello fellow traveler. I've got 20 years of scripts with `if test` in them. I only use `[[` when I need functionality it provides that `test` does not (pattern or regex matching, typically, and for pattern matching I'll generally use a case statement instead).

quicklime• 3 years ago

The Google style guide doesn’t disagree with you, it says to code exclusively for bash where possible (ie mostly everywhere inside Google) and to prefer [[ over [ and test:

https://google.github.io/styleguide/shellguide.html#s6.3-tes...

In a scenario where you also need to support shells other than bash, Google’s shell style guide doesn’t say to use [ over test.

tuyiown4• 3 years ago

Reading the comments here it looks like there are somehow dozens of us using test only. Dozens !

stevage2• 3 years ago

But if you know your script is going to be Bash-specific anyway, you are probably better served by using [[ unconditionally and consistently

Honestly, I think writing a Bash script is just a terrible idea. Use a real language, and all of these nightmares go away.

For me, lately, that's been Zx (which uses Node), but there are other fine choices too.

LegibleCrimson2• 3 years ago

I agree. If a POSIX shell script serves your needs, use it. If it doesn't, you shouldn't be reaching for a more powerful shell, but an actual language. I'd reach for Perl before Bash, and I can't stand Perl.

jmmv4• 3 years ago

Bash is "real language" if you treat it as such. Yes, there are plenty of spaghetti scripts out there, riddled with global variables and full of side-effects, but if you write shell in a principled manner, you can solve some pretty big problems with ease, and you can write maintainable code.

I wouldn't start new projects in shell, of course, but one area in which I think the shell shines is in writing integration tests for tools. More on this in a recent post I wrote: https://jmmv.dev/2023/10/unit-testing-with-shtk.html

Joker_vD2• 3 years ago

but if you write shell in a principled manner

The same can be said about QBasic as well, honestly. And here's the catch: people who are inclined to write anything in a principled manner are likely the ones who'd write things in a principled language instead of e.g. shell. Or to put it in another way, if someone chosen to write in shell (or failed to consider an alternative, which also happens quite often), they're quite likely exactly the person to not write anything in a principled manner.

lmm• 3 years ago

Bash is "real language" if you treat it as such. Yes, there are plenty of spaghetti scripts out there, riddled with global variables and full of side-effects, but if you write shell in a principled manner, you can solve some pretty big problems with ease, and you can write maintainable code.

Even carefully written bash code tends to be much less maintainable than code in better languages. It's just badly designed on multiple levels, like the famous post about PHP. Yes, if you're really careful you can write decent code in Malbolge - but why would you?

I wouldn't start new projects in shell, of course, but one area in which I think the shell shines is in writing integration tests for tools. More on this in a recent post I wrote: https://jmmv.dev/2023/10/unit-testing-with-shtk.html

The fact that you've written something that compiles to shell scripts rather than writing shell scripts rather undermines your claim that shell is a decent language. Integration tests of tools are important, but I'd still find e.g. TCL a much better way to write them.

stevage2• 3 years ago

but if you write shell in a principled manner, you can solve some pretty big problems with ease, and you can write maintainable code.

So if you are extremely good at writing shell, and you write it extremely carefully, then you can achieve basically what you can do in other languages without those caveats?

It sounds like we both agree that shell is not a sensible choice for scripting, for most people, then.

tuyiown4• 3 years ago

People would had such a take and tell to use perl twenty years ago. Looks where we are now. I'd rather have improved my shell scripting abilities right away.

It's a three (optional) steps thing really:

- I want something that can use on multiple systems right away (shell)

- Ok things getting a bit serious, I'll allow myself to add some pre-requirement tooling on the system (others script)

- pre-requirement sucks, I'll invest in native code tooling and produce binary (eg rust & go for the choices of the moment).

bigstrat2003• 3 years ago

Agreed. If your shell script has more than the most basic of functionality, you should probably write it in literally any other language. Like another poster said: I don't like perl, but I'll take it any day of the week over bash.

l0b0• 3 years ago

See https://mywiki.wooledge.org/BashFAQ/031

user261• 3 years ago

I really didn't understand why the last if statement is confusing. Is it because when starting out with shell scripting one would usually assume that the [ is a part of the bash scripting language not just another program? If it's then I think I get it now. Otherwise please mention why it's surprising. Also, @author thanks for a nice article. was a good reed.

computerfriend• 3 years ago

Even if you don't know about [ as a binary, I still don't see how it's confusing. Seems like very normal bash to me.

glandium• 3 years ago

One place where you learn not to use [ or [[, or be very careful about them is autoconf shell scripts (and that applies to more than `if`, like globs, sed, awk, etc.). Because square brackets are quote characters in autoconf. https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/a...

stn_za• 3 years ago

js frontend dev masses ripping on bash here. lulz

pavlov• 3 years ago

Writing bash scripts seems like the ideal use case for Chat-GPT style programming.

The programs are fairly short, examples are common in the corpus, and the syntax and execution model are so inscrutable that only a machine can pretend to understand what’s going on.

croo2• 3 years ago

I already tried to customize my command line prompt with it. All the arcane colouring character decodings and random character escaping missions became a "put git branch names in parenthesis and make it cyan".

db48x2• 3 years ago

What’s so arcane about `echo "$(tput setaf 6)cyan$(tput sgr0)"`?

spullara• 3 years ago

For sure:

https://gist.github.com/spullara/0fc3e88150f66179017b9aa1758...

rjblackman• 3 years ago

yes, this is the best use case I have found for chat gpt. I've been automating away all of my little annoyances with powershell.

saagarjha• 3 years ago

I tried doing this and it made a bunch of mistakes with handing spaces and whatnot that makes shell scripts often brittle. Of course, a human would probably make the same mistakes, because who actually knows how to do that correctly? But it doesn’t really make me comfortable using it for anything but the smallest of automation tasks.

account425• 3 years ago

who actually knows how to do that correctly?

Many people. It's not exactly rocket science to quote your arguments which already gets you most of the way there.

wodenokoto• 3 years ago

I had no idea [ was a program and the fact that it checks if the last argument is a closing bracket is kinda funny to me.

But at least it explains why you need spaces on both sides of the brackets.

kqr7• 3 years ago

Taking the last point one step further, we can also dispense with the if block entirely:

    if [ a = b ]; then
        echo "Oops!"
    else
        echo "Expected; phew!"
    fi

becomes

    [ a = b ] && echo "Oops!" || echo "Expected; phew!"

I'm not sure how often you should do this but sometimes it comes in handy for things like

    [ "$debug" ] && echo "what's going on" >&2

to conditionally print debug output to stderr.

----

And the fact that the if block tests a regular command means we can also do things like

    if grep -q 'debug' /var/log/nginx/access.log; then
        echo "Debug request found!"
    fi

----

Something I have not yet bothered to figure out is whether I should write

    [ $(expr 1 + 1) -eq 2 ] && [ $(expr 2 + 2) -eq 3 ]

or use the built in logical and of test:

    [ $(expr 1 + 1) -eq 2 -a $(expr 2 + 2) -eq 4 ]

As long as performance is not a concern, I can see roughly equal reasons in favour of either.

illo• 3 years ago

[ a = b ] && echo "Oops!" || echo "Expected; phew!"

Not to be taken as a general rule though. I might be mistaken but I think that bash would parse the line as:

  ([ a = b ] && echo "Oops!") || echo "Expected; phew!"

so if the command sequence after `&&` fails, then the code sequence after `||` is executed anyway:

  illo@joe:~ $ [ "a" == "a" ] && >/dev/full echo "strings match" || echo "strings don't match"
  -bash: echo: write error: No space left on device
  strings don't match
  illo@joe:~ $

This is different from the semantics of the `if` block:

  illo@joe:~ $ if [ "a" == "a" ]; then >/dev/full echo "strings match"; else echo "strings don't match"; fi
  -bash: echo: write error: No space left on device
  illo@joe:~ $

nwellnhof• 3 years ago

Something I have not yet bothered to figure out

According to POSIX, the -a and -o binary primaries and the '(' and ')' operators have been marked obsolescent. See https://pubs.opengroup.org/onlinepubs/9699919799/utilities/t... under "Application Usage".

darrenf3• 3 years ago

Regardless of `-a` vs two tests and `&&`, there's no need to shell out to `expr` if bash's arithmetic evaluation is available:

    [ $((1+1)) -eq 2 ]

kqr7• 3 years ago

I'm going to assume that if one is using [ rather than [[ then one will also want to use expr rather than $(()).

neuromanser3• 3 years ago

arithmetic evaluation is ((…)), $((…)) is arithmetic expansion.

There’s no need for test(1) / [(1) or conditional expressions ([[…]]) if you’re doing arithmetic:

if ((1+1 == 2)); then …; fi

cesnja• 3 years ago

I'd advise against that kind of shortening. If you use set -e, which you should, then

    if [ a = b ]; then
      echo "Oops!"
    fi

will do exactly what you imagined, but

    [ a = b ] && echo "Oops!"

will quit with an error if expression a does not equal expression b.

Calzifer• 3 years ago

No it won't. set -e is implicit disabled for the first command with && and ||. Same for a command after if/while/until and after !. It should only matter if you implicit return immediately after.

  $ bash -ec 'if [ 1 = 2 ]; then echo true; fi; echo $?'
  0
  $ bash -ec '[ 1 = 2 ] && echo true; echo $?'
  1

In both cases it does not quite and execute the last echo.

skrebbel2• 3 years ago

I'll never understand how there's a whole class of developers who absolutely despise JavaScript for acting weird when adding arrays to objects, but at the same time gladly write bash scripts and send each other articles about how [ is totally a program but ] isn't and that's all fine and dandy and in no way objectionable.

EDIT: this comment is a bog standard HN middlebrow-dismissal and the blog post doesn't deserve this for to be the top comment. It was morning and I was grumpy. I can't delete it anymore so would appreciate some downvotes.

hun33• 3 years ago

You can replace bash with JS. Scripting is their common application domain. Can you replace JS with bash, though?

If the answer is "no," that means JS has applications (e.g., servers and web clients) that bash can't be used for. And they're saying JavaScript is bad at them. Bash is arguably worse, but it's usually not an option in the first place so you don't get to complain.

mananaysiempre• 3 years ago

You can replace bash with JS

Noo-ot really except in a Turing-tarpit sort of way. Like it or not, once learned (!) Bourne shell together with the traditional tools is a well-designed user interface and works even better for that than, say, Tcl[1]. Not an automation or scripting language, a user interface for all daily interaction. I would absolutely hate to manually sort through my files in JS, while in shell I can often do it faster than in a GUI file manager.

And, of course, it’s also a fairly strong contender as a programming language in the paradigm of many pipelined imperative processes—probably because that paradigm remains largely unexplored. I can only maybe name Icon as a viable competitor, and Icon’s also very nice. (Python, no matter how “inspired” it is by Icon, has traded its command of streams for more mainstream ease of use.) By comparison, the old complement of parallellized JS build tools (Gulp? Grunt? I forgot, it’s been a while) always surprised me with how awkwardly it accomplished shell-script-equivalent tasks.

To be clear, it’s not that Bourne shell is good and JS is bad. JS is a passable Fortran[2], while shell is at best a marginally functional one—there’s a reason Awk exists. But shell competes in categories that most other languages don’t even try to qualify for.

[Yes, I know about rc. I happen to think rc’s focus on one-level lists of strings (incidentally shared by Jam, an attempt at a better make) is a mistake, and more consistent arbitrarily-nested quoting, giving a “stringy Lisp” in the vein of Tcl, would be the way to go.]

[1] http://yosefk.com/blog/i-cant-believe-im-praising-tcl.html

[2] http://conal.net/blog/posts/can-functional-programming-be-li...

rnmmrnm• 3 years ago

one language being the only option available on browsers and the other being a (replaceable) glue layer between _other_ programs? I also see minimalistic beauty in the fact that `[` is not even part of the shell. "do one thing and do it well" at its finest.

prerok2• 3 years ago

Well, shell scripts are small helper programs, where you don't have to implement all sorts of business rules and complex object interactions. I am pretty sure most developers (I definitely) would be horrified at the prospect of having to to implement in shell what we are now doing in JS. Even if that were possible.

The likes this gets is the design simplicity and uniformity (across processes), not how it looks and what it actually does.

DonHopkins4• 3 years ago

Oh come on. Who are you to say developers don't have to implement business rules and complex interactions? You're just making an after-the-fact rationalization for a terrible, foolish, thoughtless, pointlessly complex design. And then trying to dictate how people should use their tools and what kinds of problems they should limit themselves to solving.

It's not like the shell script designers sat down at a meeting and said:

"OK, it's very important that we don't want people using this language to implement all sorts of business rules and complex object interactions, because decades from now there will be invented an Ousterhoutian dichotomy and government regulations enforcing that developers should use other kinds of languages for that, because of the essential definition of what it means to be a shell scripting language, so we've got to come up with some way of punishing people who attempt to do that, and introduce obscure hard to spot bugs in their programs as a consequence if they have the audacity to do that, or even if their initially simple scripts later get more requirements and have to become more complex. Now let's brainstorm about how we can do that, and make sure the syntactic syrup of ipecac we come up to solve this problem fits in well with the rest of the language design by being totally off-the-wall and unlike every other piece of syntax in any other programming language including itself."

Then again, maybe you have a point, and they did do it on purpose, judging by how terrible the rest of the language is!

"Language Design Is Not Just Solving Puzzles" -Guido van Rossum

https://news.ycombinator.com/item?id=20672739

https://www.artima.com/weblogs/viewpost.jsp?thread=147358

Summary: An incident on python-dev today made me appreciate (again) that there's more to language design than puzzle-solving. A ramble on the nature of Pythonicity, culminating in a comparison of language design to user interface design.
Some people seem to think that language design is just like solving a puzzle. Given a set of requirements they systematically search the solution space for a match, and when they find one, they claim to have the perfect language feature, as if they've solved a Sudoku puzzle. For example, today someone claimed to have solved the problem of the multi-statement lambda.
But such solutions often lack "Pythonicity" -- that elusive trait of a good Python feature. It's impossible to express Pythonicity as a hard constraint. Even the Zen of Python doesn't translate into a simple test of Pythonicity. [...]

http://lambda-the-ultimate.org/node/1298

Guido: Language Design Is Not Just Solving Puzzles
And there's the rub: there's no way to make a Rube Goldberg language feature appear simple. Features of a programming language, whether syntactic or semantic, are all part of the language's user interface. And a user interface can handle only so much complexity or it becomes unusable.
The discussion is about multi-statement lambdas, but I don't want to discuss this specific issue. What's more interesting is the discussion of language as a user interface (an interface to what, you might ask), the underlying assumption that languages have character (e.g., Pythonicity), and the integrated view of semantics and syntax of language constructs when thinking about language usability. [...]

Ousterhout's Dichotomy is a contrived descriptive not prescriptive fiction, to rationalize the design of TCL after the fact. And despite its flaws and limitations, TCL is orders of magnitude better and more thoughtfully designed and purposefully thought out and internally consistent than any Unix shell scripting language.

https://news.ycombinator.com/item?id=9970505

https://en.wikipedia.org/wiki/Ousterhout%27s_dichotomy

Ousterhout's dichotomy is computer scientist John Ousterhout's categorization[1] that high-level programming languages tend to fall into two groups, each with distinct properties and uses: system programming languages and scripting languages – compare programming in the large and programming in the small. This distinction underlies the design of his language Tcl. [...]
Criticism: Critics believe that the dichotomy is highly arbitrary, and refer to it as Ousterhout's fallacy or Ousterhout's false dichotomy.[4] While static-versus-dynamic typing, data structure complexity, and dependent versus stand-alone might be said to be unrelated features, the usual critique of Ousterhout's dichotomy is of its distinction of compiling versus interpreting. Neither semantics nor syntax depend significantly on whether a language implementation compiles into machine language, interprets, tokenizes, or byte-compiles at the start of each run, or any mix of these. In addition, basically no languages in widespread use are purely interpreted without a compiler; this makes compiling versus interpreting a dubious parameter in a taxonomy of programming languages.

simias• 3 years ago

I dislike JavaScript and shell scripts equally, but sometimes I have to add a feature to a web page and I have to use JS, so I do, and sometimes I need to automate some un*x system task in a portable way and without heavy deps and shell scripts are the obvious solution.

What annoys me is using JavaScript and shell scripts when there are clearly superior alternatives and no clear advantage for it besides the familiarity (which, admittedly, can be a strong argument).

Shell scripts being an arcane mess is no excuse for Javascript being as clunky as it is, and vice-versa.

skrebbel2• 3 years ago

One clear advantage that both JS and Bash have over nearly every other language is stability. Code you write today is very likely to still work in 20 years.

chlorion• 3 years ago

(I am going to use the word "shell" to refer to posix-sh like shells specifically)

I think a lot of this is a (li)nix culture thing.

Not liking javascript is "cool" in some sense. It's this new fandangled web language, not a "real" programming language in this culture. Shell scripting is not seen in the same light because its older and associated with unix I guess.

Not liking newer technologies in general is a thing that I have noticed, like with Rust or even C++ which isn't that new!

I do think shells have a better excuse for being bad programming languages than javascript though. Shells are primarily an interface to your computer and not a programming language. I use a shell in a terminal emulator to do most things on my system, like managing files and updating the system, you couldn't easily use javascript for this without writing a shell in javascript.

Shells are also much older than javascript, and "newer" shells like bash or zsh need to maintain some level of backwards compatibility.

Mostly shell scripts are used for very small tasks and not for writing large programs which is another factor.

You do bring up a very interesting point though. I think this idea applies to perl as well. It's interesting to see the difference in how these things are viewed.

xiaodai• 3 years ago

R programmers be like: I still don't know when to use [ or [[. I just put a browser() then and test out the options

DonHopkins4• 3 years ago

Syntactic Syrup of Ipecac.

hibbelig2• 3 years ago

I have recently written

    if some_command 1>/dev/null 2>/dev/null; then
        : # A-OK
    else
        here_is_the_code_i_actually_needed
    fi

And it leaves me wondering if there is a way to “negate” the exit status of a command...

(The command was “docker volume inspect VOLUME”, and in one place in my script I had to do things when the volume existed, and in other places I had to do things when the volume did not exist...)

jeramey• 3 years ago

Yes, there is. To negate, just use ! as with many other languages.

    if ! command …; then
        do_needed_things
    fi

ramshorns• 3 years ago

How does that work? It doesn't look like ! is a command or shell builtin, so I guess it's an argument to if.

hibbelig2• 3 years ago

Amazing! I've been writing shell scripts since the 90s and now I still learn something new. Thank you!

xyzzy_plugh• 3 years ago

Some people insist that Linux distributions be called GNU/Linux, not just Linux. The reason is that, strictly speaking, Linux is just a kernel and, when you are using a Linux system, you are primarily interacting with the GNU userland. While I sympathize with that idea, I’m not the one to adopt the GNU/Linux name, in part because most Linux distributions today ship with software developed by many more vendors than just GNU. In this post, however, I use the GNU/Linux term because saying Linux alone would be unfair to Linux.

Hear, hear!

jmclnx• 3 years ago

Interesting, on Linux (Slackware, Debian), /bin/[ and /bin/test are not the same. But on NetBSD (and I assume others too, they are the same.

1vuio0pswjnm7• 3 years ago

Bourne has said there were proposals from the other users at Bell Labs to use braces or brackets with the "test" form, but these suggestions were not adopted. Bourne has said he did not like having to worry about missing braces in C, and there were no IDEs with automatic syntax-checking like today, and that is why he used macros to allow him to write in C using Algol-like syntax. That's what he said. Maybe that's why he decided not to braces or brackets with test.

Personally, I prefer the "test" form to the "[" form. It's easier for me to read. Personal preference. To me, "[" looks uglier.

gkfasdfasdf• 3 years ago

Instead of trying to make a shell script portable across all different shells, I prefer to just use #!/bin/bash and not worry about it. (Of course bash version compatibility is a concern of it's own)

test, [, and [[ (2020)

Related Stories