The roots of an obscure Bourne shell error message

https://utcc.utoronto.ca/~cks/space/blog/unix/BourneShellObscureErrorRoots

by r4um • 3 years ago

95 35 3 years ago

utcc.utoronto.ca

bdamm• 3 years ago

And today I learned about shellcheck.

Thank you for that fun and concise trip into shell evaluation :)

pinusc2• 3 years ago

Excellent explanation! However, I would disagree both that there isn't an easy fix and that each of the decisions are sensible.

If the syntax to temporarily set variables was different, say using : instead of =, or requiring some delimiter between assignments and operations, or maybe enclosing the statement in braces (something like: {foo=bar; echo $foo}...) to delimit temporary assignments would have been reasonable alternatives, perhaps even more readable than this. Or having the sntax disallow `foo=`, requiring, say, `foo=null` or `foo=''`, which are both more explicit, or even `unset foo`. These are more sensible individual choices, and that's partly because they're more explicit and partly because they avoid a (now) common error. I'd argue good language design needs to think slightly bigger picture than "individually not catastrophic" choices. Sh is littered with a myriad of small bad choices, which add up to a language that's hard to code or script without bugs. Nowadays with shellcheck it's easier, but it shouldn't be necessary. The consensus is: it's a scripting language, it shouldn't be used for complex stuff. But if it was a slightly better scripting language, it _could_ be used more safely for larger or more complex scripts.

The truth is that it's an ancient language, its (bad) syntax is set in stone, and there's nothing we can do about it except slowly move to alternatives. Sad that there's nothing established to take its place (Perl is read-only, python is not good enough as unix glue, everything else is too obscure).

I know how and love to write in bash. But oh god was it painful to learn

__float• 3 years ago

I know how and love to write in bash. But oh god was it painful to learn

This is very well said :)

Sad that there's nothing established to take its place (Perl is read-only, python is not good enough as unix glue, everything else is too obscure).

Is there anything notable? Either particularly well designed, or just popular? I think I've only ever heard of Oil (https://oilshell.org)

gavinray• 3 years ago

Ruby has very ergonomic and pleasant UX for doing the sorts of things you typically want to do in bash scripts IMO.

Izkata3• 3 years ago

or maybe enclosing the statement in braces (something like: {foo=bar; echo $foo}...) to delimit temporary assignments

As long as you don't need to get anything out of it, it's a bit heavy but dropping to a subshell works like that:

  $ (foo=bar; echo $foo)
  bar
  $ echo $foo

Galanwe• 3 years ago

I don't think your comment fairly represents the situation.

Bash is not _just_ a scripting language, it's also (and primarily) a shell.

That means it needs to be good at both interactive REPL-style usage _and_ at simple scripting.

Being a shell implies that it has to be exceptionally succinct and flexible about executing programs and manipulating environment variables.

In that light, your comparison with Python and Perl don't make much sense.

Would you enjoy replacing:

    $ cd /tmp
    $ ls -l

with:

    $ os.chdir('/tmp')
    $ subprocess.run(['ls', '-l'])

I guess not.

Shells will always be a different breed of languages. Bash is maybe not the best at what it could be, but it's not the end of the world either.

The example in the article is a bit silly to be honest. I think the result is kind of obvious for anyone half bash litterate.

jhoechtl• 3 years ago

That means it needs to be good at both interactive REPL-style usage _and_ at simple scripting.

The creators of powershell obviously were of a different opinion

pinusc2• 3 years ago

You are right to point out that in my comment I talk about bash in terms of a language rather than just a shell; but I did have its shell-nature in mind while writing it, and in fact your example is exactly what I meant when I said that python is not good enough unix glue... there's a certain level of script complexity where python starts making more sense than bash, but because the ergonomics of calling other programs (unix glue!) or building pipelines are pretty poor, that threshold is pretty high.

Some of bash syntax makes perfect sense in the context of a shell-but-also-scripting-language. Some of it makes perfect sense in the context of "everything is a string", which I think is an absolutely great abstraction---for example, unset variables being equivalent to empty strings. Other parts are just jargony, and might make the life of a few experienced shell-slingers slightly easier, while complicating many users' interactions with the shell.

I suppose part of my point is that succintness here comes at the cost of readability. A shell could very well be _slightly_ more verbose than bash (say, requiring assignment operators to have two sides, so `foo=''` instead of `foo=`), still be plenty succint as a shell, but being slightly more explicit and readable in a scripting context. `foo=''` is only _two_ characters longer than `foo=`, and is already-implemented behavior. But the `foo=` syntax has certainly confused generations of unix users... that's just not a good tradeoff imo. I feel pretty comfortable calling this lack of foresight, or suboptimal design; that's not bashing (!) the language or its creators, rather, merely noting that we can do better.

And if you don't think so... check out fish shell! It's a beloved and successful shell, and they forgo the use of = entirely (instead, you have to `set foo bar`, which is more verbose and a lot more explicit!)

Edit: fish does allow `foo=bar echo $foo` to override variables, so they actually manage to keep bash ergonomics while avoiding mistakes while scripting.

paddim8• 3 years ago

I'm working on a shell language that combines these things.

You can write things like: let column = du -h | str::column 0 echo(column[1..3])

kpcyrd• 3 years ago

For programming, I would gladly go with the second option in return for proper flow-control mechanics and data structures.

1vuio0pswjnm74• 3 years ago

This does not seem obscure to me at all. Having learned about C from reading djb's programs, I write programs that read their arguments from environmental variables. (See djbdns for example.) When I run one of these programs it's common to set variables on the command line, e.g.,

   echo https://example.com|Connection=keep-alive yy025|nc -vvn 127.127 80

yy025 sets the HTTP Connection header according to the value of the corresponding environmental variable. If I want to send a POST request I set the variable httpMethod to POST. The address 127.0.0.127 is a TLS forward proxy. Original netcat allows one to type 127.127 without the zeros.

Using djb's envdir utility, one can put a selection of different HTTP headers into a directory so that there's no need to type them on the command line.

    cd /path/to/dir
    echo close > connection
    echo aplication/x-www-form-urlencoded > content-type
    echo id=123\;key=xyz > cookie
    echo 19 > content-length
    echo POST > httpMethod
    echo /api/whatever > path
    echo "{ \"key\": \"value\" }" > post-data
    cd
    echo https://example.com \
    |if cd /path/to/dir;envdir . yy025;then 
    cat post-data;fi \
    |nc -vvn 127.127 80

It's strange to me to see this author use printenv instead of set (a built-in) to display Bourne shell environmental variables. As someone who uses Bourne shell every day and on a variety of computers, I never use printenv. On the smaller systems I make, I do not include printenv in the userland.

The use of fgrep to match a single word in all caps is also strange.

Maybe I am just missing some UNIX wisdom here. If so, please disregard this comment.

From the article

  $ export FRED=value
  $ printenv | fgrep FRED
  FRED=value
  $ FRED=
  $ printenv | fgrep FRED
  FRED=
  $ unset FRED
  $ printenv | fgrep FRED
  $ # ie, no output from printenv

Without using printenv or grep -F (in shell script /bin/fgrep)

  $ export FRED=value
  $ set|grep FRED
  FRED='value'
  FRED=
  $ set|grep FRED
  FRED=''
  $ unset FRED
  $ set|grep FRED
  $ # ie, no output from set

zerocrates• 3 years ago

A lot of people are probably just not familiar with the single-command envvar syntax, and of people that do know it some of them probably don't know the syntax for unsetting.

So it's just not immediately obvious to many why a simple extra space is a problem at all, and why it's this specific error. It can also just generally be hard to see past what you intended to write to what you actually did and how that will be parsed.

Izkata3• 3 years ago

"printenv" (or just "env", the one I habitually use) shows only environment variables, not all available functions and variables like "set" does.

But otherwise yeah, I also don't see this as obscure at all. Was expecting something confusing when, say, fiddling with arrays or something.

1vuio0pswjnm74• 3 years ago

Good point about functions. Normally it's not something I notice because I'm not using Bash. But every once a while I use set in Bash and I'm blasted with a slew of preloaded functions.

siebenmann• 3 years ago

In this case, one reason to use printenv is that it is an external command, and so it is clearly and unambiguously seeing (and reporting) whatever the Bourne shell would export into the environment for a real command. In this specific case, it appears that set does not report such single-command variables (whether or not they have a value, eg 'FRED=barney set | grep FRED').

(I am the author of the linked-to entry.)

1vuio0pswjnm74• 3 years ago

Missing semicolon.

   FRED=barney; set | grep FRED

Also there is no need for grep, nevermind grep -F (fgrep), if using printenv.

   FRED=barney printenv FRED

Without semicolon

   sh -c 'FRED=barney set|grep FRED'

   echo 'FRED=barney set|grep ^FRED'|sh

1vuio0pswjnm74• 3 years ago

   Without using printenv or grep -F (in shell script /bin/fgrep)

   $ export FRED=value
   $ set|grep FRED
   FRED='value'
   $ FRED=
   $ set|grep FRED
   FRED=''
   $ unset FRED
   $ set|grep FRED
   $ # ie, no output from set

(Missed a dollar sign.)

Another thing is that printenv does not make it easy to see spaces or other nonprinting characters in the values of environmental variables. For example PS="# " displays as PS=#. TAB=$(printf '\11') displays as TAB=. For me, that's not really helpful. Whereas the builtin set command displays the values in single quotes so one can easily the presence of spaces and other non-printing characters.

A workaround is to use printenv with sed, something like

    printenv|sed -n l

If I was going to investigate "single command environmental variables" I might just use getenv and printf. That way I can see the non-printable characters. But not being an IT person or a programmer, maybe I am missing the reason why printenv, written in 1979, would be preferable to getenv. (That, IMO, would be an interesting blog post.) For example,

   int printf(const char *__restrict, ...);
   char *getenv(const char *);
   int main()
   {
   if(getenv("FRED"))
   printf("%s=%s\n","FRED",getenv("FRED"));
   return 0;
   }

usage: FRED= a.out

da39a3ee4• 3 years ago

For what it’s worth, the explanation of the error was obvious to me after seeing the code for a few seconds. So maybe my point is… don’t worry if stuff like that in shell programming seems obscure to you; it becomes much less obscure with practice (and along the way you do learn genuine lessons about unix-like operating systems, not just shell programming details).

rob742• 3 years ago

The problem is that most people (me included) don't want to become proficient in writing shell scripts, but they need to write a script (or, God forbid, extend and/or debug a script someone else wrote) from time to time. At least I do it so rarely that I forget most things from one time to the next. I understand that Bash is the way it is because of historical reasons, but maybe some less obscure error messages would be nice? ShellCheck is all nice and well, but Bash itself being more helpful would be better. Or, no, please don't tell me that other tools rely on these error messages and therefore they can't be changed?

da39a3ee4• 3 years ago

maybe some less obscure error messages would be nice?

Bear with me here! The error message is exactly what we’d expect in this situation. The author literally tried to use the string “107” as an executable name, and bash says ‘I can’t find an executable named “107”’.

It’s impossible to be a good software engineer in 2023 without understanding the basics of shell programming. And the absolute number one thing to understand about bash and related shell languages is that “everything is text”. I know one hears people saying that a lot, but you’ve got to really internalize what it means. bash is just constructing strings of text and doing stuff with it. It barely has any non-string data structures.

jlv2• 3 years ago

Sorry, that's not obscure!

rob742• 3 years ago

...not if you live and breathe shell scripts, no. But most people coming from other programming languages first need to understand the concept of whitespace being significant, and even if they do, it's easy to insert a space accidentally. In this case, if you get this error message, you would probably first look at the part in the "$(...)" (which, you think, is the only thing resembling a "command" on that line) and wonder why on earth it doesn't work, overlooking the problem slightly to the left of it.

naniwaduni• 3 years ago

I would expect to look at the part inside the $(), probably attempt to run it to confirm that it's working at all, and be a bit curious as to why the error message implies I was trying to run the output of that command!

blueflow• 3 years ago

I remember yaml and python also having syntactically relevant whitespaces.

cmeacham98• 3 years ago

Can I suggest a simple fix that doesn't involve changing behavior at all but probably improves the lives of the many users who hit this?

Just add an additional line to the error message for lines that look like this pattern:

    sh: 107: command not found
    Did you mean "AVAR=$(... | wc -l)"?

hnben• 3 years ago

I feel like that approach would create much effort to do properly. I.e. to be consistent, you would add a lot of such messages, and that would add a lot of complexity to the overall code base.

The roots of an obscure Bourne shell error message

Related Stories