What was the point of [ “x$var” = “xval” ]? (2021)

https://www.vidarholen.net/contents/blog/?p=1035

by pabs3 • 3 years ago

257 161 3 years ago

lolc• 3 years ago

This abomination reminds you to use a proper scripting language.

junon• 3 years ago

... or perhaps, as the article suggests, that parsing was hard when we first started doing it decades ago and not everyone got it right.

If it wasn't abundantly clear, this is not an issue with any of the shells today.

mschuster91• 3 years ago

If it wasn't abundantly clear, this is not an issue with any of the shells today.

The least-common-denominator aka /bin/sh and its syntax will stick around for decades to come. You can't rely on any of the "modern" shells to be around, particularly if you write scripts that target a multitude of unix-like systems (macOS, xBSD, Linux).

kmill• 3 years ago

It's amusing that parsing theory was already at least 1-2 decades old by the time they made this mistake (consider that ALGOL 60 had a complicated grammar, introducing BNF too). I'd chalk it up to this being a shell language that was evolved rather than designed.

mananaysiempre• 3 years ago

I wouldn’t say that parsing is hard (precedence parsing is especially not hard), but instead that logical operators and grouping inside test aka [ were both a bad idea (given a sufficiently complex expression, I expect you can still confuse a human as to what’s an operator and what’s a string even if POSIX is precise enough to dictate a single interpretation) and unnecessary (the shell itself has perfectly serviceable !, &&, ||, and grouping, though the ! is a bit awkward and seems to be a later addition).

If you omit those, a fairly intuitive unambiguous parse is very easy: if we’re [, check last argument is ] and throw it away; if one argument, return whether it’s empty; otherwise, if two, the first is a unary operator, evaluate and return that; otherwise, if three, the second is a binary operator, evaluate and return that; otherwise fail. That’s it.

hgsgm4• 3 years ago

The OP gives two examples where it is not fixed, because bash is so poorly designed that the fixes are partial hacks.

marcosdumay2• 3 years ago

If it wasn't abundantly clear, this is not an issue with any of the shells today.

I really doubt it. If the comparison parameters are undistinguishable from the operators, it's only a matter of enough creativity and people will to find new ways to break it.

1vuio0pswjnm75• 3 years ago

LOL. And what how would one run that proper scripting language. On top of a shell? With a userland and kernel built with shell scripts? Or maybe Microsoft Windows? Perhaps with Windows one can avoid the Almquist shell. Otherwise, it's still there whether one chooses to use it or not. Scripts using this convention are common, at least on NetBSD.

cxr2• 3 years ago

This is such a bizarre response. A Python* script, to give one example, does not—at least in any meaningful sense—run on "top of a shell". The shell will accept the command string and fork the process, but that has nothing to do with the design of the shell language being discussed here, and the whole thing happens to be incidental, anyway—you could completely remove the Unix shell from the equation by just making the Python interpreter your shell if you wanted, which would sidestep every element of the already-tenuous line of reasoning that you're using here to try to make an argument...

* which I abhor, by the way—just like GNU abhors man pages—but that doesn't make your argument any less absurd

1vuio0pswjnm75• 3 years ago

If don't like UNIX, stop using it. Let us know how that goes. It's all but impossible. Generally, one cannot even download or install a "proper scripting language" without using a computer that runs shell scripts.

papruapap• 3 years ago

It's funny how this community trash on JS at every chance they have but for some reason bash is off the hook. I am tired on seeing CI pipelines failing and Live environment crashing because typos in .sh scripts. I propose to switch to powershell every time I can but that still sounds as heresy for the devops guys.

crabbone5• 3 years ago

Lol... You are going to switch to PowerShell in initramfs?

PowerShell is just another Perl / Python / Ruby. Except made by Microsoft, so it's got a ton of useless features, lacking some essential stuff and has bombastic syntax. It's not suited for the role of UNIX Shell. Not by a long shot.

----

On a larger note, the whole idea of needing a shell is bad. It comes from a defective (or rather overly simplistic and un-insightful) design of UNIX. A better system wouldn't need a distinction between a system programming language and a shell. You would just use one and the same thing for both purposes.

If you are on Linux, Guile is supposed to be that, but... maybe it was supposed to be that?.. I'd still want it to take the role of Shell, even though it doesn't tick all my boxes.

aeyes• 3 years ago

You aren't going to write 100k lines of shell scripts but you might reach this amount of code in JS projects so we should rightfully hold the language to a higher standard. Shell scripts are just glue and unless they get huge they are easy to understand, modify and maintain.

I have a fair amount of shell scripts in CI pipelines and things usually break when we hit unhandled edge cases which is exactly what I would expect.

That said, if your script is for a single platform you won't need any of these hacks which only exist for compatibility with different/older shells. My scripts won't even run on bash 3 and I accept that.

Pxtl2• 3 years ago

PowerShell will happily run scripts with misspelled variables and not fail until it hits the invalid line, if it ever does. It's not like it's statically typed or something.

nirvdrum• 3 years ago

Most people don't like shell scripting, it's just that it's way easier to maintain the environment. A shell script from 20 years ago will likely still run without dealing with EOL runtimes or ensuring you're running an LTS version of a shell. The shell is quite stable from an API perspective, so you're not dealing with removal of deprecated features. Shell scripts largely run the same across different systems. With the notable exception of Apple shipping an ancient version of Bash, rarely do I have to care what version of a shell a system ships with. And that only matters if I want to use new Bash-specific features.

While shells can and do have security issues, much of that is historical and given the smaller surface area they just have fewer security issues than say, Node. I don't need a shell version manager à la NVM, virtualenv, or rbenv. I don't need to worry about a massive dependency graph. A shell of some sort comes pre-installed with every *nix system I've ever used so I don't need to worry about figuring out how to install a new runtime in an unprivileged environment.

I'm sure some people love Bash and love shell scripting. For many of us it's just a matter of practicality.

bmacho• 3 years ago

I am still waiting for a language that has variables, strings, loops and if-branches, and can pipe programs together.

OIL and ABS are close but who uses those.

ajross• 3 years ago

The weird thing is we were already there in the early 90's with perl, and as a community uniformly decided to walk away from it. Now we're "still waiting" for stuff people were running on SPARCstations?

ufo2• 3 years ago

Check out Tcl! It's built around strings, with a command syntax similar to shell. The standard library "exec" command allows launching programs including pipes and redirects. Although it's a bit clunky (e.g. there's no way to quote a '|' or '>' argument)

https://www.tcl-lang.org/man/tcl/TclCmd/exec.htm

nailer8• 3 years ago

pwsh. Also hashmaps (and structured data generally) so you can do

`ls | where createtime < now - hour`

That's pseudocode, I don't use pwsh every day.

pwsh never really took off as it was good tech during bad leadership (Steve Ballmer), but I hope someone (maybe Rust coreutils people) steal the approach.

marcosdumay2• 3 years ago

AFAIK, perl still comes by default with all Unixes that are actually used.

But well, most languages support piping program together. You will need a 5 to 10 lines wrapper on the most used ones, but it's always well supported.

And as a bonus, you get to explicitly say how the pipe should behave, instead of searching the docs to learn how to handle failure or early stream termination.

Pxtl2• 3 years ago

Every major admin scripting language I've ever used has so many foibles and edge-cases it makes me pine for full programming languages.

What admin scripting languages bring to the table:

1) Easy access to executable binaries - no creating a Process object to describe it, just run it

2) Easy piping and filtering

3) A standard library designed for terse and easy-to-use access to administrative tasks.

4) Pleasant in REPL form.

That's the good side. The bad side is that they're all such a collection of hideous hacks and edge-cases for basic concepts like variable scope and typing that even the simplest operations that would be hilariously trivial in a real language are utterly agonizing.

JohnFen9• 3 years ago

There are many situations where you don't get to choose what scripting language you're going to use.

citrin_ru• 3 years ago

A posix shell script written in 1990s would still work today and likely will work 10 years from now (given that external commands it uses had not breaking changes). Not many proper scripting languages have such track record - most of them either had many small backward incompatible changes or an update which breaks many things at once (like Python 2 -> 3). Though I have some 20+ years old Perl scripts which still work today but it is hard to find a language which attracts more haters than Perl.

chrismorgan• 3 years ago

Discussed extensively last year: https://news.ycombinator.com/item?id=26776956

One key point there is that it’s still easy to trigger cases where you need this hack, because the cleverness is only stable for up to four arguments; so and, or, and parentheses all exceed the point of safety.

topspin• 3 years ago

So keep doing it despite whomever it annoys. Also, throw some superfluous cats in to annoy them more.

amluto• 3 years ago

All the cleverness and precedence rules in the article seem to ignore the fundamental problem: the test and [ binaries are regular programs, and they cannot distinguish between an argument that looks like “(“ and (. In both cases, they get the string “(“ in argv.

So x serves as a form of quoting. If I were to potentially invoke a binary to evaluate an expression, I would quote any string-like variable (using the x hack) and I would do my best to avoid numeric comparisons.

(Or use built-in shell features where the shell knows which values are quoted.)

rascul3• 3 years ago

All the cleverness and precedence rules in the article seem to ignore the fundamental problem: the test and [ binaries are regular programs, and they cannot distinguish between an argument that looks like “(“ and (. In both cases, they get the string “(“ in argv.

Regular programs can make that distinction with no problem, the mistake is thinking that the outer quotes are part of the argument. In the context of the POSIX style shell, they're not, they signify that what is enclosed is one argument. So you'll have to use one of the various mechanisms the shell provides to indicate that you want to pass the quotes as an argument.

dwheeler4• 3 years ago

No need for that.

In shell the preferred form for "and" and "or" is outside the test, and you always quote anything with a variable reference, like this:

    if [ "$x" = "foo" ] && [ -z "$empty" ] ; then
      echo "Worked"
    fi

I think the "&&" is easier to read, as well as completely eliminating the problem.

dataflow• 3 years ago

That was also discussed in the thread: https://news.ycombinator.com/item?id=26778153

dezgeg• 3 years ago

Not to mention this syntax short-circuits, which is usually desirable.

cduzz2• 3 years ago

Oh, shell pet peev #301;

shell has "case" as part of the language, and case is way better at string evaluation that the "test" built in

case "$x" in (foo) echo Worked ;; esac

tragomaskhalos• 3 years ago

Guilty here, but have been doing so for so long that it's largely muscle memory at this point and the original rationale has been lost. In mitigation, it certainly used to be the case that you were likely to come across a broad variety of Unixy flavours and hence using highly conservative scripting constructs made sense; then after a while these become second nature, and unless they cause actual problems they just stay in the mental toolkit.

mkovach• 3 years ago

Same here!

When I started off in the UNIX world, the shop had a collection of AIX, SunOS, Solaris, HP/UX and Digital UNIX. the x-hack was standard form and I spent a few years fixing errors with the x-hack (or converting the scripts to Perl).

It is just second nature to use the x-hack in if's and case statements on this point. I have to try really hard to NOT use it. Of course, the folks with less gray in their hair and often confused by it, referring to it a "Olde Unix."

I have approved plenty of PRs with the comment, "Updating mek's Olde Unix syntax."

pixelbeat__• 3 years ago

Oh I will reference this from https://www.pixelbeat.org/programming/shell_script_mistakes.... where I also advise against this common archaic idiom

pyentropy2• 3 years ago

Wait... zsh which is quite a powerful shell that's now by default on macOS too, would say "(" is equal to ")" until 2016?

Was it just single [ (test) or [[ as well?

adql• 3 years ago

Wouldn't be surprised if it was just backward compatibility so bash/sh scripts work

yrro3• 3 years ago

Using [ is a mistake. Use [[ and relax!

rurban6• 3 years ago

Unportable

orra2• 3 years ago

Portability isn't binary. The question is, is it portable to the systems you want to support?

Arch-TK• 3 years ago

Only as unportable as bash. Which is probably the most portable shell.

rurban6• 3 years ago

You'd never do [ "x$left" = "x$right" ], but [ x$left = x$right ]. The whole reason for the x-hack was to omit the double-quotes. With the dquotes you can of course leave the x out. [ "$left" = "$right" ]

pyentropy2• 3 years ago

No you can't. Read the post.

Karellen2• 3 years ago

What happens if you use the x hack but $left has a space in it?

rurban6• 3 years ago

Then you get a nice syntax error, that you didn't properly split your string. Good point

Karellen2• 3 years ago

Remember though, just because the shell on your rolling-release dev machine fixed this issue years ago, does not mean the shells on all your users/clients/legacy servers/hobby backports have fixed the issue too.

MontyCarloHall• 3 years ago

However, the value was mostly gone by the mid-to-late 1990s, and the few remaining issues were cleaned up before 2010 — shockingly late, but still over a decade ago.

Anyone running a shell that still requires the x-hack is deliberately maintaining a legacy system, at which point the onus is on them to get modern tooling working on their machine.

pfortuny2• 3 years ago

Being forgiving towards the user is one of the important principles of programming. Because the user may not be able to “upgrade” his system.

Windows 95 is still very much alive.

Embedded-like systems are a thing.

Asooka• 3 years ago

I get what you're saying, but the fix to support that old shell is literally two characters. If you don't want to support it, it would be good courtesy at least to print an error and abort, lest the buggy behaviour ends up damaging the user's system.

JohnFen9• 3 years ago

I think telling your customers who rely on older systems to go pound sand is a good way to lose customers. Not to mention a bit callous.

aliher1911• 3 years ago

Most recent MacOS is still shipped with bash 3.2.57 FWIF. Since default was changed to zsh they would likely never bump bash. So you never know.

niccl• 3 years ago

up until last year I was supporting a system for a client that would only run on Solaris 2.5.2 from 1996 or so. (finding working versions of the hardware was becoming _really_ hard). The system was retired last year, replaced by something that costs several million dollars, but until then I had no choice.

There are other things like that out there. I think I recall seeing something on HN fairly recently abut a controller for some CNC machine that was running an early version of MSDOS. Again, replacing with something more modern was very expensive. These things exist and the users aren't necessarily being wilfully malign in not upgrading

skrebbel2• 3 years ago

You really want to cater to people who chose to install Dash, and then not update it for 8 years?

johnbellone• 3 years ago

Revenue has no bounds.

naniwaduni5• 3 years ago

I'm not sure any dash has ever been broken in a way that would be worked around by the x-hack. This is an issue for, basically, pre-standard shells.

quotemstr• 3 years ago

Yes it does. Sorry, I'm not supporting some ancient buggy AIX thing. People tying their shell scripts into knots using hacks like this are just doing performative compatibility nonsense. It hasn't mattered in the real world in a decade.

naniwaduni5• 3 years ago

Also, this is hardly going to be the deal-breaker for running your software on such systems unless you're expressly writing for them. Pre-standard systems are exotic in ways that are going to break whatever you're trying to do in way more exciting ways!

JohnFen9• 3 years ago

It hasn't mattered in the real world in a decade.

This is not true at all. There are plenty of such systems, run by major companies, in daily use. It isn't the most common case, but it certainly matters in important parts of the real world.

kazinator2• 3 years ago

That is false. The only systems which have /bin/sh shells that have that sort of bug are proprietary Unixes, which also ship improved shells that better conform with POSIX, installed in a different location.

A much smarter thing is for the script to detect, at the top, that it's running on a crappy /bin/sh, find the better shell, and re-execute itself with that shell (setting some environment variable to prevent recursion).

Then the rest of the script doesn't have to use unusually stupefying coding techniques. Just the regular stupefying techniques.

SAI_Peregrinus2• 3 years ago

IMO either write pure POSIX shell (no BASHisms or assuming external programs not guaranteed by POSIX exist, no assuming FHS paths are in use, etc) or require a specific shell and set of dependencies. Check for that at the start, exit with error if things aren't found.

Scripts and programs will always get run in unsupported (broken) environments. If your program can't exit with a clear error in such a situation, it's not very portable.

Personally I tend to use the multi-line nix-shell shebang:

#! /usr/bin/env nix-shell #! nix-shell -i SHELL -p DEPENDENCIES #! nix-shell -I nixpkgs=NIXPKGS_TARBALL_URL

If you want `bash` as your shell, then `bash` goes in the SHELL location and as one of the DEPENDENCIES. If you want GNU coreutils that goes in DEPENDENCIES. Etc. The last line is optional, it pins the versions of all the dependencies you want. This also works for Python scripts.

mauvehaus• 3 years ago

If you aren't quoting the left hand side, you still need it if the left side is an empty string:

  $ foo=""
  $ [ $foo = "" ]
  -sh: [: =: unary operator expected
  $ [ x$foo = "x" ]
  $ echo $?
  0

I'm not arguing that this is a good idea, merely observing that this case remains.

Empty strings had always been the case that I'd been taught required the x-hack.

misnome• 3 years ago

Isn't this one of the things that bash [[ ]] explicitly fixes?

hgsgm4• 3 years ago

I hope not, because $empty is necessary when you actually want to elide a missing value, like Python's `filter`

naniwaduni5• 3 years ago

Well, on the left side. On the right side, [[ still requires you to quote, otherwise:

    % bash -c 'a=xy; b="x*"; [[ $a == $b ]] && echo "$a == $b"'
    xy == x*

My hot take is that you should avoid using [[, preferring to use [, which at least pretends to work like a normal program that can be invoked using normal quoting rules—it just interprets a funky mini-language on its command-line arguments. [[ is some exotic thing that seems reasonable at first glance but follows no rules but its own and varies its behavior depending on syntactic nits that literally cannot possibly matter and aren't even distinguishable for normal programs.

And then just quote consistently, which is something you need know how to do with almost every other line of a shell script anyway!

hgsgm4• 3 years ago

Why would someone want to intentionally write broken code and then hack around it, instead of the obvious correct thing?

There are lots of ways to hack around lots of ways to break code.

digitalsushi3• 3 years ago

Because when we try something that works, we have a bias that it must be the right way to do it.

It's probably the biggest drawback to self-learning.

dwheeler4• 3 years ago

You don't need the x-hack for strings. In shell, a widely-recommended form for ANY expansion of a variable uses quoting unless you have special requirements. E.g., normally don't use $foo, use "$foo". Tools like shellcheck enforce this. Not ideal, but not a big deal either.

firstlink• 3 years ago

in shell

This advice is highly dependent on specific choice of shell. This kind of quoting is unnecessary and discouraged in zsh, for example.

hgsgm4• 3 years ago

You can still see this on macOS bash today:

  $ str="-e"
  $ [ \( ! "$str" \) ]  
  bash: [: `)' expected, found ]   # bash

> POSIX fixes all these ambiguities for up to 4 parameters

So... It's not fixed, and the hack is still needed?

zimpenfish• 3 years ago

To be fair, that's bash 3.2.57 which was released 2014-11-07.

Oldest bash after that I have available is cygwin bash 4.4.12 (2017-10-13) which doesn't have the issue.

Just install a newer bash using ports or homebrew and bob is your shell uncle.

aflag• 3 years ago

If your script is going to be used by people on macs it’s probably easier to use the hack than ask them to update. Given all she’ll weirdness, I don’t even think this is the most offensive

mannyv• 3 years ago

Linux's not unix!

msla• 3 years ago

Legally, FreeBSD isn't Unix.

There is, however, Inspur K-UX, a Linux distro which is a Unix.

jmmv• 3 years ago

Here is something I wrote a while ago which may help provide some context and that explains the differences between test, [ and [[ : https://jmmv.dev/2020/03/test-bracket.html

russfink• 3 years ago

Helpful, thanks!

cabalamat• 3 years ago

It is for this sort of reason that I use Python instead of shell scripts.

scbrg• 3 years ago

Yes. No gotchas there.

    >>> foo = b'foo'
    >>> foo[0] == b'f'
    False

Don't get me wrong. I love Python, it has been my daily driver for 20+ years (and it pays my salary), but it does have its own set of cognitive burdens.

Also, shell scripts I wrote in the 90:s work today. Python scripts - not so much.

bagels• 3 years ago

Bash is the hardest language to write comparisons in if you're not doing it every day. The syntax is so unlike anything else.

freedomben5• 3 years ago

Bash is the hardest language to write comparisons in if you're not doing it every day. The syntax is so unlike anything else.

worse than brainfuck? I find that hard to believe...

on a serious note though, I don't agree. the syntax is not that different from the awk -> perl -> ruby heritage. Just a few constraints/conventions due to whitespace significance that other langs don't have.

JohnFen9• 3 years ago

I agree. There are genuinely weird things about bash that increases the difficulty of writing for it, but I never really counted comparisons as one of them.

klyrs• 3 years ago

worse than brainfuck? I find that hard to believe...

Malbolge perhaps. Brainfuck is tedious, but it's a language of 8 symbols; there's nothing to forget. But doing comparisons in bash calls for a whole quickref of its own.

pengaru• 3 years ago

I only ever encounter this pattern in autotools-related shell code, stuff that ultimately ends up in configure scripts.

There it arguably makes sense to be so robust against old/broken shells, since the whole point is to successfully configure the build on a broad spectrum of environments.

kazinator2• 3 years ago

Here is a question: what is the point of

   if test x$var = val ; ...

instead of using the [ command?

Some years ago, I checked: the [ alias for test was already in the Bourne shell in 1979.

Assume the year is 1999. What real portability problem do you solve by avoiding [ in favor of test?

Real meaning that the rest of the script, and the software stack the script is related to, runs on a system where [ wouldn't work.

dundarious• 3 years ago

I always use test:

- it looks like every other thing in sh (`[` is the outlier)

- it prevents confusion like putting `&&` inside the `[ ... ]` brackets (understandable that `[` works that way if you think about it, but it’s another way it’s an outlier)

- mildly discourages use of one particular bash-ism, `[[`, as it works slightly differently, and I prefer POSIX sh unless there is a good justification

- it doesn’t require a closing `]`

I don’t use it for what I think you mean by portability, that is, because `test` exists whereas `[` does not, or anything like that.

ezekiel68• 3 years ago

For all of the arcanities, things like this really make me admire the pragmatic attitude of the grey beards. Kind of like how the limited register count in early CPUs led to "of course" reusing one of your operand registers to store an instruction result.

What was the point of [ “x$var” = “xval” ]? (2021)

Related Stories