Why "ls *.c" is at least dumb, and possibly dangerous

http://groups.google.com/group/comp.unix.shell/msg/5d19dadaf9329f87

by r11t • 17 years ago

28 41 17 years ago

groups.google.com

nuggien• 17 years ago

this is one of those things no one cares about until it affects them.

nailer5• 17 years ago

> Suppose you have a file with a name of "aaa\nbbb.c". (from the article)

To be honest, I wouldn't care about my shell not handling newlines in files, I'd care about my users creating files with newlines in them. Space is natural language, newline is not.

GeoffWozniak• 17 years ago

Dumb on whose behalf? If you need to know the inner workings of the shell in order to know that is "dumb" then I submit that in and of itself is dumb.

weaksauce• 17 years ago

Apparently you have not been bitten by a filename expansion bug when shell scripting. This kind of thing is fairly common with bash and some of the other ones when dealing with files that have spaces/newlines/whatever. It boils down to learning and knowing your tools and how to work around their limitations.

rmaccloy2• 17 years ago

In this case, if you're not an expert with bash, a good rule of thumb would be to pipe your glob-expanded list of junk into perl -nle 'print' (or echo, or... etc) before running something destructive.

Every time. Until you're an expert.

DanHulton• 17 years ago

Filenames can contain newlines?

You learn something new (and vaguley horrifying) every day.

rmaccloy2• 17 years ago

Filenames can contain plenty of happy stuff.

$> touch -- --

$> rm --

usage: rm [-f | -i] [-dPRrvW] file ... unlink file

$> rm -- --

$> touch \

$> touch \\ \ \ . \-

favorites of malware authors everywhere.

mncaudill• 17 years ago

I instinctively cringe whenever I see an odd looking 'rm' command.

aw3c2• 17 years ago

On a sidenote, creating a -i file in important directories can help against accidental "rm -rf"s. The -i will be given to rm as switch and rm will now prompt for each file. Of course backups are better but this might just safe your ass that certain time.

mustpax• 17 years ago

Traditionally Unix filenames can contain everything except for / and null (aka \0). This is why the find utility provides a facility for printing null terminated filenames (-print0), when used in conjunction with xargs -0 this is a fool proof way of iterating over files in the shell.

For example, to delete all .o files in the current directory and its subdirectories:

   find . -name "*.o" -print0 | xargs -0 rm -f

BrandonM• 17 years ago

Why not just

  find . -name "*.o" -delete

?

__david__• 17 years ago

Yep. Though I don't think I've ever seen an actual legitimate use of this capability. I mean really, what's the point of an embedded newline...

Actually the Mac uses (or at least "used") a hidden files named "Icon\r" (yes, an embedded carriage return) for its custom folder icons. I'm still not sure that counts as "legitimate".

blasdel2• 17 years ago

I had to write a one-liner to deal with that on Mac OS 10.4 -- had to detect if a user had placed anything on their Desktop, which had a custom folder icon because it was actually a cross-filesystem symlink:

  desktop=$(test -z $(ls -A1 /Volumes/Documents/Desktop | \ 
            sed -e '/^.DS_Store$/d' -e '/^Icon^M$/d'))

The ^M was a literal CR, which would get munged by all my usual editors except vim. I'm sure there's some sturdier way to escape it...

I remember choosing to use ls intentionally, but not what sucked about the alternatives.

duskwuff2• 17 years ago

As noted above, I've seen some concepts thrown around about a (Linux) kernel patch to disallow filenames which contain unprintable characters, and I approve of them.

drp• 17 years ago

"ls *.c" is not dangerous. The pipe is dangerous.

nailer5• 17 years ago

The user making files with newlines is dangerous.

omouse2• 17 years ago

The user doing something that the system allows him to do is dangerous??

You guys are insane.

catzaa• 17 years ago

Am I the only one who thinks that scripting is a "Dirty Job"?

I have not written nor seen any bash script that is neat, clean and fairly clear. It all seems like a very shady hack that someone think is smart.

nailer5• 17 years ago

Not sure why you're being downmodded. On the simplest of engineering levels, shells don't adequately separate data from execution. They also lack data structures beyond strings and arrays.

ipython / iruby can and do make much nicer shells out of the box.

yan3• 17 years ago

Wait. What?

1) Is he implying that launching a perl interpreter is somehow a more lightweight operation than launching ls and having it stat() half a dozen files? Having ls check if a file exists 99% of the time won't even touch the disk. If you're working in that directory, those inodes will probably be cached. Running stat() on a cached inode is hardly more expensive than a system call. Read: cheap. Now perl launching will probably load a few libs and check a few files before even getting to his one-liner.

2) Am I the only one, who in my ~10 years of running unix-like operating systems have never created a filename with a newline char in it? Like even when opening obscure filesystems, never.

3) Does he think unlink() skips the check for the file's existence?

4) If he wanted to overcomplicate things and make sure everything's kosher, why not:

  find . -maxdepth 1 -type f -name '*.c' -delete

Whenever I see anything piped to a perl one liner, 4 times out of 5 it's some sort of hack that can be done with standard utils.

It also grinds my gears when people take longer to think of making something more efficient than the amount of CPU time they'll save in their lives of using that 'more efficient' solution. So he spent a minute coming up with this and ten minutes posting about it. Now, will he save roughly 11 minutes of execution time in his life while running this "superior" command (also count that it takes longer to type)? If not, he wasted his time.

From his sig: "Smalltalk/Perl/Unix consulting". When you have a large-enough hammer....

geocar• 17 years ago

1) stat() is a system call, so one system call is bound to be faster than two. 2) Sneaky users do this; they create files like /dev/shm/...\n-rf\n.. and you'd better know how to delete it correctly. 3) ??? 4) -delete is a GNU extension.

yan3• 17 years ago

1) Just the system call overhead is not much compared to what the system call actually does, so it depends on what you're calling. Two system calls versus one with identically-cached information, sure it's slower. I'm just saying perl will also probably stat() and if not perl, the unlink() system call would perform the same function. When you're modifying files and flushing it to a drive, especially while using the shell interactively, the sys call overhead is completely negligible. All I'm trying to say is claiming the first command as inefficient is groundless; you execute it as a human and once -- it won't feel any slower.

2) If cleaning up after malicious users, sure why not. Doing this in your source tree or during normal usage? I don't know.

3) He mentioned ls stat()ing files as an inefficiency since the shell does it and ls does as well.

  FIND(1)                   BSD General Commands Manual                  FIND(1)

  NAME
       find -- walk a file hierarchy

...

       -delete
             Delete found files and/or directories.  Always returns true.
             This executes from the current working directory as find recurses

basugasubaku2• 17 years ago

This is not the point of what he is saying.

Read the context of the thread. It is a reply to this:

http://groups.google.com/group/comp.unix.shell/msg/30e16d21d...

Randal is pointing out that piping the output of ls to the perl interpreter is dangerous and shows the safer way to do what the original poster wanted.

tptacek• 17 years ago

You aren't going to create a file with a newline in the name. The person who wants to totally screw you over is going to create the file with the newline in the name.

yan3• 17 years ago

I realize that. The example he used was a collection of c files, leading me to believe he was showing a personal collection of source as an example, wrongly or rightly. And I realize you aren't going to be creating files with newlines in them (or leading dashes, or many dots, etc), but once you get into a territory of someone being able to do that, and you're not competing in a CTF, I'd say there are larger issues to deal with.

slice2• 17 years ago

I wonder how many files with a name containing \n are in the wild. And if you encounter such a file, does it really matter if you accidentally delete it?

Waisting brain cycles to save CPU cycles (in the wrong places) is shameful, insulting and boring.

It would be amazing if there weren't tons of other, much more significant, things that deserved attention in the relevant script, system, or the developer's life. (sign)

duskwuff2• 17 years ago

I've seen some concepts thrown around about a (Linux) kernel patch to disallow filenames which contain unprintable characters. I approve of them.

With regard to why this matters, it's not uncommon for badly written scripts to interpolate filenames directly into commands. Consider this Perl fragment:

  $sessionFiles = `ls /tmp/session_*`;
  system("rm -f $sessionFiles");

and what it would do with a file called:

  "/tmp/session_blah; chmod +s"

omouse2• 17 years ago

And if you encounter such a file, does it really matter if you accidentally delete it?

It matters. Leaving a bug around is disgusting and wasteful especially when you can fix it.

* would be amazing if there weren't tons of other, much more significant, things that deserved attention in the relevant script, system, or the developer's life. (sign)*

Fixing bugs and writing programs that have as few bugs as possible is very important.

slice2• 17 years ago

When you find a bug, by all means, go ahead and fix it. (Well, not always. You need to make sure your fix is not likely to introduce a more severe bug. Not all bugs made equal and some guts-feeling required).

If your goal is to make your system as robust as engineerically possible, you should most likely pay more attention in some directions and less in other. It's not only about quantity - you better have no critical bugs then have only-a-very-few critical ones.

prodigal_erik• 17 years ago

The problem is the file you accidentally delete (or whatever) is probably not the one with the weird name. Most bogus filenames I've seen were created by shell scripts that had a set of commands that was accidentally quoted instead of executed, so they tend to contain the names of files which were important to the author.

wvenable2• 17 years ago

This article does point out one thing that I've always found really strange/dumb about Unix. Why does the shell expand the wildcards? That's not how it works in Windows, for example.

I periodically have to cleanup abandoned and overflowing unix mailboxes where every message is it's own file. You'd think a simple "rm *" would work, but it doesn't, because you get an error message saying that there are "too many arguments". From my perspective there's one argument, the asterisk. Very confusing. I end up having to use xargs to get it work. Seems like a poor design.

bobbyi• 17 years ago

It's so that every script in the world doesn't have to duplicate the logic for how to interpret globs. They inevitably won't end up doing it exactly the same way.

snprbob86• 17 years ago

As a Windows developer, I can tell you with certainty that practically no one does it even close to the same.

wvenable2• 17 years ago

Shouldn't that be part of the OS API? Why call me with 1,000 arguments, when I could call an API with the wildcard and get the 1,000 files back? My poor argv can't handle it the other way.

Why "ls *.c" is at least dumb, and possibly dangerous

Related Stories