Why “process substitution” is a late feature in Unix shells

https://utcc.utoronto.ca/~cks/space/blog/unix/ProcessSubstitutionWhyLate

by r4um • 5 years ago

168 82 5 years ago

utcc.utoronto.ca

veltas• 5 years ago

Didn't even know about process substitution, had been using fifo's to achieve this!

ogogmad3• 5 years ago

Fish tries to use FIFOs to emulate process substitution, and it leads to deadlock. Not sure why.

By default, Fish actually runs the processes in a strict sequence. But this is to avoid the above deadlock situation. And it therefore isn't process substitution.

ridiculous_fish• 5 years ago

To be sure, fish runs external processes in parallel. It's only internal functions which are serialized against each other. I'm hoping to lift this restriction eventually.

This SO answers a bit about the issues with FIFOs and fish shell. Basically it comes down to evaluating redirections before fork instead of after, which is because fish uses threads. https://stackoverflow.com/questions/61946995/ls-fifo-blocks-...

zokier3• 5 years ago

The usability of unixy shells generally falls down a cliff when you need to deal with more than just one input one output. The awkwardness in trying to shoehorn process substitution is just one of the examples of that.

rscnt• 5 years ago

do you know alternatives to that? I assume PowerShell but don't know if there's anything beyond that.

zokier3• 5 years ago

I remember seeing some academic work on extending shell sematics to more complicated pipe networks, but nothing particularly promising. In industry, I think that is generally the point where people pick up "real" programming language instead of trying to work in shell; on top of my head I imagine golang with its channels and goroutines to be particularly well suited for these sort of problems. I can't say if there is something in golang that shells could adapt somehow.

lnanek2• 5 years ago

You can always setup some named pipes. E.g.:

mkfifo named_pipe

echo "Hi" > named_pipe &

cat named_pipe

I used to do this in bash scripts to keep them cleaner and the lines simpler.

matheusmoreira2• 5 years ago

Yeah. It's caused by this single standard input/output model. Even using standard error is unergonomic and leads to magic like 2>&1.

What if programs could have any number of input and output file descriptors and the numbers/names along with their contents and data types were documented in the manual? Could eliminate the need to parse data altogether. I remember Common Lisp had something similar to this.

nicoburns2• 5 years ago

My understanding is that they can. stdin, stdout and stderr are just pipes assigned to file descriptors 0,1 and 2 by convention. There's nothing stopping a program from having more file descriptors passed in, and some programs do. There's no just no standard convention on how to do it.

mathgorges• 5 years ago

What if programs could have any number of input and output file descriptors and the numbers/names along with their contents and data types were documented in the manual?

What you’re describing is similar to one of the more common ways that programs are run on mainframes by using Job Control Language (JCL).

TorKlingberg• 5 years ago

Great read, but worth noting from the end of the article that "late feature" here means it was added in the early '90s. The late addition to Unix that surprises me is ssh. It was only invented in the late '90s. Before that everyone used unencrypted remote shells like telnet.

Encryption in general was in a pretty bad state in the '90s: original http was unencrypted and early mobile phone standards that are still in use have very weak encryption.

teddyh• 5 years ago

Telnet has encryption options, it just wasn't widely implemented.

zokier3• 5 years ago

Everything was unencrypted until late 90s (and in many cases until late 00s). Email (both smtp and pop3/imap), irc, web, gopher, telnet, ftp, local disks, removable storage, network storage (smb/nfs etc), everything. Computing and the internet was much nicer place, there wasn't such an adversial attitude where everything would be broken just because its out there like today.

mturmon• 5 years ago

Yeah, on the time sharing Unix systems I would use in the 80s and 90s, everyone’s home directory (and most everything under it) was world readable by default. You could change the permissions, but most people didn’t.

I feel like those old folks who tell of a time when people didn’t bother to lock their doors at night.

errcorrectcode4• 5 years ago

I started before CompuServe, Internet, or the internet were nouns.

It wasn't nicer back then, it was lazy and naïve.

3DES was widespread in the payment card industry, but the attitude towards protecting any/all parts of networks corresponding to the 7-layer OSI model was generally lax.

IPv4 public address ranges (mostly registered Class B's and C's) were wasted frivolously for internal corporate networks where they weren't suited or even necessary.

Unless they didn't know what they were doing, bank logins weren't unencrypted. Ever.

I and some lab peeps played with ARP and IP spoofing to steal each other's telnet sessions in the late 90's. It was obvious telnet, rcp, rsh, echo, char, finger, and nfs needed major reworking and/or abandonment.

Later, the Equifax hack broke SSN's as universal American private "UUIDs" (primary keys).

Things still broken as of 2022:

0. Without deploying 802.11x, DHCP by itself is still terrible because anyone spoof being a server and disrupt many communications on a LAN. Properly managed campus ELANs/WLANs should authenticate all WiFi and Ethernet connections equally and disconnect any misbehaviors at the port or AP-association level.

1. PII should be held by a secure, independent, nongovernmental nonprofit where it can be updated in one place and set access policies by the individual. Companies then can request access to it. That way, PII is treated more like medical records (PHI) and payment card info. For the most part, corporate customer data should be anonymized as much as possible by law.

2. There is no global universal standard identity / proximity card / secret keys HSM. Similarly, it should not be held or managed by any country, only issued by their organizations.

3. There is simultaneously too much anonymity for launching cyberattacks while not enough for protecting dissidents. Social media app operators should understand how much anonymity and identity-revealing/-proving is appropriate to ensure people invest-in and maintain a minimum amount of decency and empathy vs. cyberdisinhibitionism.

usr11063• 5 years ago

Before Eternal September the Internet was trustworthy place. And criminals had not found it yet.

mprovost2• 5 years ago

Encryption is CPU-heavy and CPUs weren't nearly as fast then as they are now. Unix was developed on systems like a VAX which could do 1 MIPS (millions of instructions per second). For comparison an M1 chip can do about 10 trillion instructions per second. It just wasn't possible to encrypt data in real time like it is now.

vimax2• 5 years ago

A fun trick is using the `paste` command with process substitution to combine lines of output from different commands.

  ./cmd1

outputs:

  a
  b
  c

  ./cmd2

outputs:

  1
  2
  3

  paste <(./cmd1) <(./cmd2)

outputs:

  a 1
  b 2
  c 3

You can then use xargs to use each line as arguments for another command:

  paste <(./cmd1) <(./cmd2) | xargs -L1 ./cmd3

calls:

  ./cmd3 a 1
  ./cmd3 b 2
  ./cmd3 c 3

sillysaurusx9• 5 years ago

Should’ve been named zip. Is there an unzip?

But of course it can’t be named zip because that’s the compressor, so I’m really asking whether there’s an unpaste.

I guess the closest is awk {print $1;}, but it makes me wince every time I write it. (I already know that would give syntax errors as written here.)

vimax2• 5 years ago

Depending on the data, the right options for xargs would work, and there's the pr command I haven't used much.

zamfi• 5 years ago

if you know the separator, you can use `cut` -- not an unzip per se (doesn't give you the full separated lists, without calling it multiple times anyway) but approximately equal to the awk line

teraflop• 5 years ago

Now you've got me curious which came first, "paste" or "zip".

Python's zip function was added in PEP 201 [1], which credits the Haskell 98 report with the name. The same function appears in all prior versions of the Haskell language that I could find, dating back to Haskell 1.0 in 1990.

On the other hand, the GNU coreutils version of paste[2] has copyright dates going all the way back to 1984, which presumably means it was derived from UNIX System V or an even earlier version.

[1] https://www.python.org/dev/peps/pep-0201/

[2] https://github.com/coreutils/coreutils/blob/master/src/paste...

fragmede• 5 years ago

    alias awk1="awk {print $1;}"

Counting to 9, inclusive, is left as an exercise for the reader.

cerved3• 5 years ago

Is there an unzip?

xargs printf

throwawayboise2• 5 years ago

How do you know that the outputs a, b, and c correspond to outputs 1, 2, and 3 and whether they will always occur in the same orders. This technique seems like it's full of possibly invalid assumptions.

sillysaurusx9• 5 years ago

I love zsh’s take on process substitution:

  cat =(echo foo)

It’s almost identical to <(echo foo), but crucially it drains the output before performing the substitution. It’s like <(... | sponge), but somehow much more reliable.

I’ve used it in a few situations where process substation failed, though being old and feeble I can’t remember why it failed. But for example you can copy the file safely, knowing that it will contain the full output of the shell pipeline, unlike process substitution.

I don’t even know the name of =(...). Hmm. Sponge substitution?

sillysaurusx9• 5 years ago

Thought of an example.

  cp =( ( echo foo; sleep 1; echo bar) | sponge) ) foobar

The first will complete instantly and result in a file named foobar but whose contents is either zero length or foo, depending on how unlucky you are.

The second takes one second to complete and results in a file named foobar containing foobar.

It’s strange that vanilla process substitution can’t seem to solve this at all.

LukeShu3• 5 years ago

I'm not sure what you're trying to say here.

With GNU cp on Linux, the commands

  cp <( ( echo foo; sleep 1; echo bar) | sponge ) foobar
  cp =( ( echo foo; sleep 1; echo bar) | sponge ) foobar

both have the same result: the file "foobar" containing the string "foo\nbar\n".

renewiltord• 5 years ago

In practice I end up caching the output often. I have used process substitution but the iteration process feels more useful to me if I've slowly built up data and I can inspect the internal pieces each time and reuse them in different ways.

But I can see if it's relatively fast. I like it. I just don't end up using it often.

unixhero• 5 years ago

Which is the best she'll, fish or zsh?//

yjftsjthsd-h• 5 years ago

There is no possible way to usefully answer that question as given; every shell has its own advantages and disadvantages. Plain bourne is universal but bare-bones, bash is mostly ubiquitous on Linux, zsh is powerful but just different enough to occasionally bite you, fish is very user friendly but doesn't even try to be compatible with anything else, ksh is a nice option and is built-in on BSDs, dash sucks for interactive work but is great for running scripts...

nmz• 5 years ago

tcl

errcorrectcode4• 5 years ago

Nawh, it's all about tk. ;)

errcorrectcode4• 5 years ago

When everything is both nails and screws, a hammer isn't the best tool for everything (unless it has a screwdriver on the end and the hammer part is big enough to be actually useful as a hammer).

shuffel• 5 years ago

For whatever reason I can never remember the syntax of <(command) and end up "rediscovering" it every year. It's seldom used but when it's needed it's rather elegant.

Another somewhat related useful bash feature is using this form:

    wc <<< 'a b c'

instead of:

    echo 'a b c' | wc

JNRowe• 5 years ago

With the caveat that the here string causes a tempfile to be written¹, so they're not quite equivalent. How much that matters for your use cases though is a different question, but it is worth thinking about if you're doing lots of repeated calls.

¹ With Bash v5 it may use a pipe if the data is small enough, but you can't guarantee people will have that because of GPLv3 phobia. I believe it is always a tempfile with zsh.

iamevn• 5 years ago

I quite like how fish does process substitution.

  foo (bar | psub)

Instead of syntax it's just a plain function.

docs: https://fishshell.com/docs/current/cmds/psub.html source of psub.fish: https://github.com/fish-shell/fish-shell/blob/master/share/f...

limoce• 5 years ago

But fish's psub only supports file or named pipe, not the one mentioned in the post, which is described as the "the best way".

zanchey• 5 years ago

It doesn't even support named pipes - the function's been broken for years. Interestingly it doesn't actually seem to matter a great deal, at least now that temporary files are usually on a fast disk.

ogogmad3• 5 years ago

While the syntax might be nice, strictly speaking, process substitution isn't currently possible in Fish: https://news.ycombinator.com/item?id=29845845

jakub_g• 5 years ago

An unfortunate thing is that process substitution does not work in git bash on Windows. (at least it was the case last time I tested; googling around I found a random comment in random github repo saying it's been fixed in 2.25 but I don't have a laptop handy to test it now).

cerved3• 5 years ago

it does

chubot• 5 years ago

Fun fact, in bash you lose the exit codes of process subs. It doesn't even wait() on those processes in many cases.

So there is no way to abort a bash script if something like <(sort nonexistent) fails.

OSH lets you opt into stricter behavior:

    $ osh -c 
    set -e
    shopt -s oil:basic

    diff <(sort left) <(sort nonexistent)
    echo "should not get here"'
    '
    sort: cannot read: nonexistent: No such file or directory

    diff <(sort left) <(sort nonexistent)
                      ^~
    [ -c flag ]:1: fatal: Exiting with status 2 (command in PID 29359)

In contrast, bash will just keep going and ignore failure.

You can also get all the exit codes with @_process_sub_status, which is analogous to PIPESTATUS.

(I should probably write a blog post about this: http://www.oilshell.org/blog/)

andreyv• 5 years ago

So there is no way to abort a bash script if something like <(sort nonexistent) fails.

The process ID of the last executed background command in Bash is available as $!.

  cat <(sort nonexistent)
  wait $! || echo fail

gives

  sort: cannot read: nonexistent: No such file or directory
  fail

mkdirp2• 5 years ago

The process ID of the last executed background command in Bash is available as $!.

After a quick search it doesn't seem like this is an option for anything but the last executed background process. So if you have the typical `diff <(left) <(right)`, there is no easy way to get the pid of `<(left)`.

I suppose you could output the pid to a file, and use it from there, though I haven't tested if that would work.

errcorrectcode4• 5 years ago

0. Process substitution is a potential DoS vector as it could take up all of RAM and/or disk space.

1. Also, not all commands are compatible with it, especially if they need rewinding or reopening. diff has issues with using it for both arguments often. It's likely the use of memory mapped files, but I could be wrong.

2. Shells ought to implement a flag for process substitution to allow temporary files to reside on disk for the lifetime of the command line. This way, it can operate on extremely large files.

FedericoRazzoli• 5 years ago

This is a great explanation! I wonder many times why we had to play with obscure xargs instead of being able to pipe a command output to an argument.

kazinator• 5 years ago

The claim is false; process substitution can be cobbed together with named fifos,* and those are "ancient".

Only problem that those are temporary objects that have to be created in the file system, and cleaned up.

However, temporary objects (files, no fifos) are also used in here doc implementations.

Process substitution is a late feature simply because the creativity juice in Unix (tm) dried up some time before the middle 1990's, leaving the FOSS reimplementations of Unix to carry the development torch.

Those projects had to balance among other goals like quality/robustness and compatibility.

(If we look at the quality of the FOSS tools compared to the Unix originals, we could also remark that "quality and robustness was late in coming to Unix". But we equivocate on Unix, because GNU stands for GNU is Not Unix!)

Features appearing in FOSS utilities like GNU Bash take time to make into Unix (tm).

Process substitution is not yet in the standard, therefore it is not in in fact in Unix (tm).

Shell scripting is a conservative activity. The language isn't very good and so improving it is like kicking a dead horse in some ways; the most important matter in any new shell release is that old scripts keep working. (Like configuration scripts for the build systems of nicer languages).

---

* See GNU Bash manual: https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash....: " Process substitution is supported on systems that support named pipes (FIFOs) or the /dev/fd method of naming open files. "

xeromal• 5 years ago

Yeah, but Saget could still have had a damaged heart from it regardless of Clapton's experience.

Why “process substitution” is a late feature in Unix shells

Related Stories