A peculiarity of the GNU Coreutils version of 'test' and '['

https://utcc.utoronto.ca/~cks/space/blog/linux/CoreutilsTestPeculiarity

by ingve • 3 years ago

78 66 3 years ago

utcc.utoronto.ca

PPS: In theory GNU Coreutils is portable and you might find it on any Unix. In practice I believe it's only really used on Linux.

Oh how times have changed:) But even with the demise of most commercial unixen, I was under the impression that Darwin preserved the ancient tradition of installing the system and then immediately replacing its coreutils with GNU?

monocasa• 3 years ago

Darwin's coreutils are mainly updated from FreeBSD I think. They've got some gnu stuff, but it hasn't been updated since gnu switched to GPLv3.

yjftsjthsd-h8• 3 years ago

Right; the coreutils built in to Darwin are BSD-derived, I was referring to a user installing the GNU versions over top of the OS proper.

xcdzvyn2• 3 years ago

Unfortunately not. macOS comes with violently outdated FreeBSD coreutils, for the GPLv3 situation. Though the default shell was recently changed to zsh from Bash 3.2 from 2007.

yjftsjthsd-h8• 3 years ago

Right, which is why I thought it was common for people to install the operating system, then immediately use macports or homebrew to go get the GNU coreutils and a modern version of bash because the BSD versions are less friendly.

(Thus continuing and extremely long tradition of layering GNU over the vendor tools; ex. Sun had their own tools, but everyone liked the GNU versions better.)

pram• 3 years ago

The BSD licensed stuff isn't outdated, it's actually directly from FreeBSD 14.0 afaict. ZSH is current (5.9) in Sonoma for example.

zappb• 3 years ago

That was the case for a while, but when they were updated to use GPLv3, Apple stopped updating them, probably to avoid licensing problems on iOS. Nowadays you can install them from Homebrew, MacPorts, straight from source, or other methods.

o11c• 3 years ago

and in case anyone wants to bring up GNU/kFreeBSD - that's officially dead as of July.

yjftsjthsd-h8• 3 years ago

Tragically true:( But you can still install https://www.freshports.org/sysutils/coreutils/ on FreeBSD if desired.

chungy• 3 years ago

I tend to install coreutils on FreeBSD. Besides some minor annoyance with every command being prefixed with "g", some of the programs work a bit nicer than the FreeBSD-shipped versions (or some Linux-centric programs just want the coreutils versions...).

1500100900• 3 years ago

I'd be interested in learning which commands you have in mind and what specifically is a bit nicer about their coreutils implementation.

dveeden2• 3 years ago

Installing GNU coreutils etc was also common on Solaris. However as it usually wasn't in your path before SysV utils it was only used with its full path.

pixelbeat__2• 3 years ago

Yes, macos takes its utils from FreeBSD.

An interesting factoid is that FreeBSD/macos sort(1) was using GNU code until recently, since this is quite tricky to implement. Eventually it was reimplemented for GPL avoidance reasons.

We do consider macos though, and ensure all tests pass on macos for each release

freedomben2• 3 years ago

Does Apple compensate or sponsor that work at all?

hallman76• 3 years ago

Thanks hn for doing what you do! I'm so glad to see this here.

A few months back I noticed '[' under /bin on my mac. I tried googling to understand what it was, but my google-fu came up short. It felt like one of those ungoogleable things. This link is an excellent starting point for me.

dylan6043• 3 years ago

it is funny that [ is the name of binary but when using it, you can't just type [ without the closing ]

by the name, you'd think you could just use [ 0 -gt 1 <enter>

after all, we don't have to type grep query perg

yjftsjthsd-h8• 3 years ago

That does work if you call it as `test`; just a quirk of faking shell syntax in an external binary I think.

tedunangst• 3 years ago

The really funny part is running it as /bin/[ probably requires omitting the ].

mike_hock2• 3 years ago

What's not so funny is that /bin/[ has no way of enforcing this syntax rule, or any other. Just like sed can't enforce any syntax rules on its regular expressions. This is why the shell is still a crock of shit for scripting even with "set -e" on.

The invoked binary has no way of aborting script execution. All it can do is barf out on stderr and return an error code, which the shell interprets as false and `if [ "x" = "x"` (without ]) goes into the else branch.

bobbyi• 3 years ago

Searching for /bin/[ gets reasonable results for me

https://www.google.com/search?q=%2Fbin%2F%5B

unnah• 3 years ago

Doesn't mac have manual pages for shell utilities? In other words, this should work: man [

DonHopkins5• 3 years ago

If I can go "man [", then why doesn't "man (" work? And why aren't there man entries for the rest of all the punctuation marks that bash uses?

dang• 3 years ago

Recent and related:

test, [, and [[ (2020) - https://news.ycombinator.com/item?id=38387464 - Nov 2023 (225 comments)

1vuio0pswjnm7• 3 years ago

There must be reasons to use the standalone programs instead of the shell builtins. But the test builtin will be the same in, e.g., NetBSD sh as it is in Linux or Android dash. That is what I use 100% of the time. It is much faster to use builtins.

usr11063• 3 years ago

Much faster in theory, unnoticeable in practice. I assume nobody does number crunching using shell scripts.

arp2425• 3 years ago

You don't need to do "number crunching"; the performance differences are very real even for simple scripts on modern systems. Launching a process is comparatively expensive (read binary from disk, run linker, allocate new process space in kernel, etc.) whereas a builtin is just "if progname == "[" { .. }". And when you do something like "loop over a few thousand files" the difference really adds up.

Consider this example:

  for i in $(seq 1 10); do
      for f in /*; do
          if [ $f = /etc ]; then
              : # Do nothing.
          fi
      done
  done

I have 23 entries in /, so this will run [ 230 times – not a crazy number.

  % time sh test
  sh test  0.00s user 0.01s system 91% cpu 0.009 total

bash, dash, zsh: they all have roughly the same performance: "fast enough to be practically instantaneous".

But if I replace [ with /bin/[:

  % time sh test
  sh test  0.11s user 0.38s system 96% cpu 0.509 total

Half a second! That's tons slower! And you can keep "fast enough to be practically instantaneous" for thousands of files with the built-in [, whereas it will take many seconds with /bin/[.

(Aside: if I statically link [ it's about 0.4 seconds).

hibbelig2• 3 years ago

I vaguely remember that the program and built-in could not always do the same, eg the program might not have supported -e. But it’s possible that ksh was used to get -e. Does anyone remember better than I?

A peculiarity of the GNU Coreutils version of 'test' and '['

Related Stories