Redirecting Stdio is Hard

http://blog.httrack.com/blog/2014/12/20/redirecting-stdio-is-hard/

by mmastrac • 11 years ago

40 15 11 years ago

blog.httrack.com

evmar2• 11 years ago

One FD_CLOEXEC-related bug that got us in Chrome: on Linux, while we were careful to close any open fds when spawning a subprocess, the GTK IME ("input method editor", modules to facilitate typing Japanese etc.) could itself spawn subprocesses in response to innocuous calls (like user keystrokes). Those subprocesses could then inherit (and keep open) fds we wanted to be closed.

semenko• 11 years ago

Oh hey, any chance you could explain this bit of the /usr/bin/google-chrome script I've always wondered about? (Sadly, the bug is RVG.)

   # Sanitize std{in,out,err} because they'll be shared with untrusted child
   # processes (http://crbug.com/376567).
   exec < /dev/null
   exec > >(exec cat)
   exec 2> >(exec cat >&2)

Somehow child processes can abuse stdin/stderr/stdout in ... creative ways?

hobarrera• 11 years ago

Funny. /usr/bin/chromium shows nothing like that.

I'm wondering why these launchers differ so much.

rwmj• 11 years ago

I had a really "fun" redirection bug. My library was appening a log file to stderr, but for various convenience reasons, it opened /dev/stderr (instead of just writing to fd #2).

We got a bug report that if you ran a program that used the library by doing:

    prog >& /tmp/log

then stderr from the program before my library started logging would disappear. If you did this intead, it would work fine:

    prog |& tee /tmp/log

It turned out (obvious in hindsight, not at the time) that we opened /dev/stderr with O_TRUNC, causing /tmp/log to be truncated and earlier logs to be lost.

TLDR: /dev/stderr can be truncated, which was unexpected.

pyre• 11 years ago

/dev/stderr wasn't being truncated. The redirect caused the options (e.g. O_TRUNC) to be inherited by the file it was redirected to.

BorisMelnik• 11 years ago

Great post, and can I also plug the fact that the parent website / app (HTTRACK) is an excellent app and something I use or have used on a daily basis for a few years now.

thwarted• 11 years ago

The thing is that spawned processed do not give a damn about already opened file descriptors (except for stdio). But the standard is broken for historical reasons, very unfortunately.

It would be nice to have more detail here, at least why this is "historical" and "unfortunate". A whole slew of programs wouldn't be possible and capabilities wouldn't exist without this, both older and modern. Two examples are inetd's wait functionality (letting the spawned program accept(2) further connections on the inherited listening socket) and systemd's, ahem, socket activation.

main program sens[sic] commands to remote location and closes the socket

If you intend the socket connection to be ended, you should be calling shutdown(2) on it, not close(2). close(2) only asserts that the calling process is done with it. The child process would end up with a dead socket fd (although, there are bugs related to improperly handling this, here's one that bit me: https://github.com/brianmario/mysql2/issues/516 ). This is why the failure cases on syscalls should always be examined and there should be a sane, default failure case. In an ideal world, the leaked file descriptor would merely be just that, and a properly written program would ignore (or close) any file descriptors it wasn't explicitly designed/meant to use.

gdwatson• 11 years ago

It would be nice to have more detail here, at least why this is "historical" and "unfortunate".

I think a lot of people feel that way about Unix features that interact awkwardly with threads. I tend to feel that threads are the unfortunate part, but that seems to be minority opinion.

Someone• 11 years ago

The thing is that spawned processed do not give a damn about already opened file descriptors (except for stdio). But the standard is broken for historical reasons, very unfortunately.

I read that as "it would be better if the default for spawned processes were to _not_ share file descriptors", not as "it would be better if it were impossible to inherit file descriptors"

frou_dh3• 11 years ago

Say I have a command that I want to read both a piped in stdin _and_ interactive input from the user.

Is there a saner way to facilitate this than what I have done? Which is to run it like this from the shell:

    something | mycmd 10<&1

and in mycmd read stdin as normal and interactive input from /dev/fd/10

chousuke• 11 years ago

I'm not sure if this is POSIX, but if you program accepts positional arguments, in bash you can do eg.: diff <(somecommand) <(someothercommand) and the shell will replace these with /dev/fd/<n>, allowing you to also accept standard input without relying on hardcoded file descriptors.

frou_dh3• 11 years ago

Cool tip. You're right though, bash in /bin/sh mode doesn't have that feature.

teddyh• 11 years ago

Read from stdin as normal, and read interactive input from /dev/tty.

frou_dh3• 11 years ago

That's the ticket!

Redirecting Stdio is Hard

Related Stories