Shelling Out Sucks

http://julialang.org/blog/2012/03/shelling-out-sucks/

by vgnet • 14 years ago

77 36 14 years ago

nessus427• 14 years ago

I remember ages ago getting in an argument with the kernel hackers on unix-wizards. My position was that filenames shouldn't be allowed to have meta-characters in them because it made shell programming difficult and dangerous. Their argument was that the spirit of Unix was that the kernel should be as simple as it possibly could be, and that you should "Live free or die". Therefore it was not the kernel's business to dictate what characters should be able to be in a filename.

They were wrong of course. The true spirit of Unix was to enable programmer productivity by being able to easily piece together new programs out of reusable components. E.g., via shell scripting, and the kernel hackers' notions about filenames and living free was directly at odds with this true spirit.

The OP claims that they've solved this dilemma in Julia. I await with bated breath.

rcthompson2• 14 years ago

How can the kernel possibly dictate what filenames are acceptable if it wants to interoperate with filesystems created by other operating systems?

nessus427• 14 years ago

if it wants to interoperate with filesystems created by other operating systems?

That, of course, was not an issue back in the day, as Unix did not in fact support anything such thing.

As for today, please do tell me how Unix can interoperate with filesystems that allow nulls and/or forward slashes in filenames. I'm sure that you might come up with an existing or potential solution, and well, there's your answer!

perlgeek• 14 years ago

Note that different shells have different meta characters. Which one would you forbid? All of them?

Then I'm sure somebody comes up with a shell where nearly all characters are meta characters, leaving only one or two to make up file names.

nessus427• 14 years ago

Back in the day, there were only two shells: The Bourne Shell and the C Shell. And before that there was only whatever the Bourne Shell was based on.

In any case, the answer to this is that the kernel developers and the shell developers get together and decide on something, and then future shells have to live within those constraints. At the point that they should have done this, they should have made sure that this was flexible enough for future needs.

Personally, I would have been fine if filenames were restricted to more or less to the same degree that variables names are restricted in, let's say, C. Why can't I name my variables whatever I want to? I thought it was "Live free or die!"?!?!?!?!

tesseract• 14 years ago

> The OP claims that they've solved this dilemma in Julia. I await with bated breath.

The obvious solution would seem to be to provide an easy way to directly create a pipeline from within Julia. I.e. you use syntax similar to shell syntax, but rather than spawning a shell Julia spawns the desired processes directly. That way it can directly pass arguments to execv (avoiding escaping issues altogether), wait on all of the child processes individually (allowing clean and detailed error detection), and avoid the overhead of actually starting a shell.

mappu• 14 years ago

> Although shell escaping does work, realistically, no one actually bothers — it’s too much trouble.

I bother : ( Next you're going to tell me that nobody bothers to sanitise input in their SQL queries.

Well, i suppose that's true, in a sense. Maybe the solution is to follow SQL and implement a prepared-query system for shell commands?

hesselink2• 14 years ago

> Maybe the solution is to follow SQL and implement a prepared-query system for shell commands?

You mean shell scripts?

angelbob• 14 years ago

No. Shell scripts have the same problem where you're not separating options from filenames and you're often not preventing a single user-supplied argument from expanding into multiple final arguments.

The list-form shell commands referenced earlier are closer, but still don't separate filenames (input) from arguments unless you remember to add "--".

But they do prevent things like pipes, which is a big deal.

sigil3• 14 years ago

1. Shell-escaping all values used to construct commands

Avoid shell-escaping and the shell altogether! Use list form program execution wherever possible.

In Perl:

    open(my $fh, "-|", "find", $dir, "-type", "f", "-print0") or die;

In Python:

    f = os.popen2(["find", dir, "-type", "f", "-print0"])

If you're executing a pipeline, you do need to get the shell involved. But I'd probably write the author's examples like so (once again avoiding the need to shell escape arguments):

    f = os.popen2(["sh", "-c", \
       'find "$0" -type f -print0 | xargs -0 grep foo | wc -l', dir])

2. Prefixing each multi-command pipeline with “set -o pipefail;

Alas, pipefail is not in POSIX /bin/sh, which many of us prefer for shell scripting. It's in bash and ksh though.

3. Explicitly checking for failure after each shelled out command

Valid. You should always do this. In shell scripts, "set -e" is a good thing, although pipelines without "set -o pipefail" are still a problem.

zwp• 14 years ago

The article does not address another significant point: filenames should also be classified as untrusted input. Using -- to separate flags from arguments can help:

    /tmp$ mkdir x
    /tmp$ cd x
    /tmp/x$ echo hello > a 
    /tmp/x$ grep hello *
    hello
    /tmp/x$ touch -- -q 
    /tmp/x$ grep hello *
    /tmp/x$ grep hello -- *
    a:hello
    /tmp/x$

It's rare to see protection against this problem either when shelling out or in native shell scripts.

joeyh2• 14 years ago

Your quoting of "$0" does not protect it from the shell. An attacker need only provide an input such as: /tmp"; rm -rf /; "

There are pipeline libraries that avoid the shell for several languages. Some I can recommend:

* libpipeline http://libpipeline.nongnu.org/ easy, if somewhat verbose, pipeline construction in C.

* hsh https://github.com/jgoerzen/hsh/wiki makes pipelines in haskell, using operators so they really look like pipelines, but without involving the shell. Example: "ls -l" -|- "wc -l"

sigil3• 14 years ago

Your quoting of "$0" does not protect it from the shell. An attacker need only provide an input such as: /tmp"; rm -rf /; "

Not true.

  $ sh -c 'ls "$0"' '/tmp"; rm -rf /; "'
  ls: cannot access /tmp"; rm -rf /; ": No such file or directory

If you were right, there would be no reasonable way to write shell scripts.

emmelaich• 14 years ago

I think the silent failures are the worse of the three by far. Also, related, very few unix commands have a consistent easy to parse format.

I have a dream -- a unix-like OS with a strong convention (or even enforcement!) of file typing and format. Optional perhaps.

The efficiency aspect is often exaggerated. As long as you do a fair bit of work in the spawned command the fork/exec cost disappears to nothing. Also, it should be rare in any decent language.

raggi• 14 years ago

The efficiency aspect can creep up on people easily though, regardless of language. It's the standard concern: you don't know how your code will be used in future.

One example: http://tenderlovemaking.com/2011/12/05/profiling-rails-start...

omaranto2• 14 years ago

Have you used Microsoft Powershell? It (indirectly) does a useful portion of what you're dreaming about. It's a UNIX like shell where commands pipe .NET objects instead of plain text. Just like UNIX commands take and produce arbitrary text but most think of it as a sequence of lines, Powershell commands can take and produce arbitrary .NET objects but most use an object which is a collection of records.

Powershell comes with builtin commands for most of the simple UNIX utilities, and you build new commands in any .NET language. Additionally, you can interop with UNIX style programs that produce plain text by using a parsing filter (there are a few builtin, you can make more).

It's actually pretty good and a lot more regular and modern-feeling than any UNIX shell I've tried. I still prefer to use GNU/Linux, but wish there was a good port of Powershell...

rwmj• 14 years ago

This article is nonsense. In libguestfs[1] we invoke shell commands for many filesystem utilities ... from C which (if you believe the article) would be the worst place to do this.

All of the problems are avoided by having a smart way to run external commands and capture their errors. See the functions [2] and examples of use [3] [4].

[1] http://libguestfs.org

[2] https://github.com/libguestfs/libguestfs/blob/d0cf52b9118ab1...

[3] https://github.com/libguestfs/libguestfs/blob/d0cf52b9118ab1...

[4] https://github.com/libguestfs/libguestfs/blob/d0cf52b9118ab1...

hesselink2• 14 years ago

Why is it nonsense? It's arguing exactly for your solution: a smart way to run external commands and capture their errors.

luriel• 14 years ago

The rc shell by Tom Duff (original for Unix 10th Edition, later more known as the Plan 9 shell) got escaping right:

http://rc.cat-v.org

After using rc for scripting for years, I dread writing more than a couple of lines of sh (and csh is best forgotten).

omaranto2• 14 years ago

If you like rc, check out es [^1]. It's inspired by rc but has higher order functions and a few other things to support functional programming.

[^1]: http://192.220.96.201/es/es-usenix-winter93.html

ggchappell• 14 years ago

The article is quite correct, of course. But this kind of bug gets tiresome. We saw the same kinds of troubles with SQL for years. And we'll see it again the next time we get a popular technique that involves code that generates code using outside input, and passes it as plain text.

So we need a new rule: If you're going to provide an interface that allows programs to generate code and pass it to something, then you need to deal with these problems proactively. You can provide sanitizing/escaping functionality. Or you can avoid passing code as plain text. Or you can do something else just as good. And proper error handling needs to be the default.

But if you don't address these issues, then your interface is lousy, and it deserves to be treated as such.

mark_l_watson• 14 years ago

I can't really agree with the article. I use Ruby a lot for small one-off utility programs that might be used for a day to get something done. Using the back tick quotes to run external processes and capture stdout, I can solve real problems in a few lines of code.

I don't think that I have every been bit by the "silent failure" problem mentioned in the article.

A little off topic, but I don't throw away small one-off utility programs when I am done with them. Instead I rename them to long and meaningful file names, toss them in a special directory, and save even more time in the future by hacking on them for similar purposes.

I may be typical: although I usually design and implement robust systems with a long life, I also work a lot with data and this requires a lot of sloppy little bits of code to change data formats, get statistics, etc.

I do agree that shelling out in production code would suck.

StefanKarpinski• 14 years ago

The trouble is that so much production code starts as a one-off. Worse still, in a startup environment there's often not much distinction between production code and one-off code.

cheatercheater• 14 years ago

Check out ShQQ, a Haskell module that does something like that:

http://mainisusuallyafunction.blogspot.com/2011/10/shqq-embe...

> λ> let x = "/proc/uptime" in [sh| sha1sum $x |]

> "337ec3fb998fb3a4650a18e0785f0992762b3cda /proc/uptime\n"

rcthompson2• 14 years ago

Yeah, I kind of figured Haskell would have solved this problem in a way that makes the typical syntax do the right thing (shell-quote the interpolated variables) and then provides an override for those few cases when you're doing something weird.

Shelling Out Sucks

Related Stories