Why not parse `ls` and what to do instead

https://unix.stackexchange.com/questions/128985/why-not-parse-ls-and-what-to-do-instead

by nomilk • 2 years ago

170 189 2 years ago

unix.stackexchange.com

fellerts• 2 years ago

The title omits the final '?' which is important, because the rant and its replies didn't settle the matter.

Shellcheck's page on parsing ls links to the article the author is nitpicking on, but it also links to the answer to "what to do instead": use find(1), unless you really can't. https://mywiki.wooledge.org/BashFAQ/020

noobermin• 2 years ago

Posts like these are like the main character threads on twitter where someone says, "men don't do x" or "women aren't like y." It just feels like people outside of you who have no understanding of your context seem intent on making up rules for how you should code things.

Perhaps it would help to translate this into something more like, "what pitfalls do you run into if you parse `ls`" but it's hard to get past the initial language.

qsort• 2 years ago

When we say "don't do X" we mean "the obvious way is wrong". If you have enough knowledge to ignore the advice, you likely are already aware of the problems with the obvious solution.

I'm pretty sure you can come up with scenarios where parsing the output of "ls" is indeed the simplest solution, but that kind of article is supposed to discourage people who don't know better from going "oh, I know, I'll just parse the output of ls". As a general advice, people should indeed be pointed towards "man find" or "man opendir 3".

g15jv2dp5• 2 years ago

What to do instead: use pwsh to completely obviate all these issues.

Qwertious• 2 years ago

Or, once it's API-stable, use nushell.

DonHopkins6• 2 years ago

Python has had an API-stable module for listing directories for decades, you know.

Aerbil3133• 2 years ago

Do you recommend it? I feel like I'd get RSI from pressing shift when using it. https://learn.microsoft.com/en-us/powershell/module/microsof...

gfv• 2 years ago

Powershell is mostly case-insensitive, and most of the core cmdlets have short aliases. Try `Get-Alias` (or `gal`) to learn more.

shiandow• 2 years ago

It's case insensitive for what it's worth. My main problem is trying to figure out which utilities they've bundled into which command.

g15jv2dp5• 2 years ago

Yes, I absolutely recommend it. I use it every day.

Commands and flags are case-insensitive.

DonHopkins6• 2 years ago

Isn't it ironic that Powershell from Microsoft is so much vastly superior than bash, not because it's great or even better than Python, but because bash is such a terribly low bar to beat, that it totally undermines the "Unix Philosophy"?

Who would have thought that little old Microsoft, purveyors of MSDOS CMD.EXE, would have leapfrogged Unix and come out with something so important and fundamental as a shell that was superior to all of Unix's "standard" sh/csh/bash/whatever shells in so many ways, all of which historically used to be and ridiculously still are touted by Unix Supremacists as one of its greatest strengths?

You see, Microsoft is willing to look at the flaws in their own software, and the virtues of their competitors' software, then admit that they made mistakes, and their competitors did something right, and finally fix their own shit, unlike so many fanatical monolinguistic Unix evangelists.

They did the exact same thing to Java and JavaScript, leaving Visual Basic and CMD.EXE behind in the dustbin of history -- just like Unix should leave bash behind -- resulting in great cross platform languages like C# and TypeScript.

Edit: that reinforces my point that taking so long to get there is a hell of a lot better than taking MUCH LONGER to NOT get there.

Maybe bash's legacy inertia is a problem, not a virtue. Is certainly isn't getting a JSON parser in the foreseeable future. The ironic point is that even Microsoft's power shell has much less legacy inertia, and therefore is so much better, in such a shorter amount of time.

crispyambulance• 2 years ago

  > Isn't it ironic that Powershell from Microsoft is so much vastly superior than bash

I agree that powershell is now better than bash. But it took SO LONG to get there. Moreover, bash has had a 12 year head-start (ok, 30 if you count earlier unix shells). Bash has legacy inertia. Even though you can now supposedly run powershell in linux, I don't know anyone who does. Does anybody?

That said, I think powershell is great for utility-knife uses on windows machines.

kbolino2• 2 years ago

At this point, all PowerShell has accomplished is creating a separate ecosystem. The designers set out to make a "better" shell and yet refused to ever learn the things they were allegedly "improving".

Basic features are still lacking from PowerShell that have been in UNIX shells since the very beginning: https://github.com/PowerShell/PowerShell/issues/3316

But hey, that's a fixable problem, right? No, because PowerShell is so suffused with arrogance about its superiority that anything, no matter how simple it was to do in a UNIX shell, has to be cross-examined, re-imagined, and bent over the wheel of PowerShell's superiority, before ultimately getting ignored or rejected anyway.

PowerShell is a language unto itself. It is not a replacement for bash/zsh/etc because nobody who knows the latter well can easily migrate to the former, and that's by design.

alganet2• 2 years ago

Alternative shells or higher languages don't solve _all_ the issues.

I won't install a new shell to generate a file list on my CI server. I won't install a new shell on remote machines. Ever.

These structured shells also require commands to be aware of them, either via some plugin that structures their raw I/O output or some convention. They solve _some_ command output structuring but not _all_ the general problem.

So, the answer is good. It promotes the idea that one should be careful when machine parsing output meant for humans.

g15jv2dp5• 2 years ago

I won't install a new shell to generate a file list on my CI server. I won't install a new shell on remote machines. Ever.

Uh... that's on you? Why do you intentionally hinder yourself?

These structured shells also require commands to be aware of them, either via some plugin that structures their raw I/O output or some convention. They solve _some_ command output structuring but not _all_ the general problem.

Okay. It doesn't solve literally every single problem, that is true. It's still miles ahead. And when interfacing with non-pwsh commands, you just fall back to text parsing/output.

eikenberry• 2 years ago

If you're going to skip using the standard shell that is installed everywhere by default, then you should go ahead and use a full language with easily distributed binaries.

hawski2• 2 years ago

I think that when someone uses ls instead of a glob it means they most probably don't understand shell. I don't see any advantage of parsing ls output when glob is available. Shell is finicky enough to not invite more trouble. Same with word splitting, one of the reasons to use shell functions, because then you have "$@" which makes sense and any other way to do it is something I can't comprehend.

Maybe I also don't understand shell, but as it was said before: when in doubt switch to a better defined language. Thank heavens for awk.

DonHopkins6• 2 years ago

When you don't want to waste your time and sanity and happiness being in doubt and then throwing away all you've done and switching to a new language in mid stream, just don't even start using a terribly crippled shell scripting language in the first place, also and especially including awk.

The tired old "stick to bash because it's already installed everywhere" argument is just as weak and misleading and pernicious as the "stick to Internet Explorer because it's already installed everywhere" argument.

It's not like it isn't trivial to install Python on any system you'll encounter, unless you're programming an Analytical Engine or Jacquard Loom with punched cards.

cassianoleal2• 2 years ago

In most places where I run shell scripts, there is no Python. There could be if I really wanted it but it's generally unnecessary waste.

On top of it, shell is better than Python for many things, not to mention faster.

It's also, as you mentioned, ubiquitous.

In the end, choose the tool that makes more sense. For me, a lot of the time, that's a shell script. Other times it may be Python, or Go, or Ruby, or any of the other tools in the box.

Pesthuf• 2 years ago

People new to *nix make the mistake of thinking this stuff is well designed, makes sense and that things work well together.

They learn. We all do.

larodi• 2 years ago

People new to the internet think alike. Still, not a day passes and we are once again reminded how fragile yet amazing this all information theory stuff is.

lukan• 2 years ago

Coincidently I discovered the unix haters handbook today:

https://web.mit.edu/~simsong/www/ugh.pdf

boricj• 2 years ago

People new to *nix don't realize that it's a 55 year old design that keeps accumulating cruft.

com2kid• 2 years ago

One of the original demos showing off PowerShell was well structured output from its version of ls.

That was 17 years ago!

CJefferson2• 2 years ago

Sometimes I want all filenames from a subdirectory, without the subdirectory name.

I can do (ignoring parsing issues):

    for name in $(cd subdir; ls); do echo "$name"; done

This isn't easy to do with globbing (as far as I know)

rascul2• 2 years ago

One alternative:

  for name in subdir/*; do basename "$name"; done

silvestrov2• 2 years ago

I'd really like if the "find" command supported this much easier, so if I write

    find some/dir/here -name '*.gz'

then I could get the filenames without the "some/dir/here" prefix.

It would also be nice if "find" (and "stat") could output the full info for a file in JSON format so I could use "jq" to filter and extract the needed info safely instead of having to split whitespace seperated columns.

hawski2• 2 years ago

If you are gonna do a subshell (cd subdir; ls) you can wrap the whole loop:

  (cd subdir
  for name in *; do
    echo "$name"
  done)

But I prefer:

  for name in subdir/*; do
    name="${name#*/}"
    echo "$name"
  done

chasil2• 2 years ago

This is really easy to do with a shell pattern.

  $ x=/some/really/long/path/to/my/file.txt
  $ echo "${x##*/}"
  file.txt

englishspot• 2 years ago

I just use find. it's a little longer but gives me the full paths and is more consistent. also works well if you need to recurse. something like `find . -type f | while read -r filepath; do whatever "${filepath}"; done`

gnuvince• 2 years ago

Is there a reason to prefer `while read; ...;done` over find's -exec or piping into xargs?

ykonstant8• 2 years ago

This will fail for files with newlines.

chasil2• 2 years ago

Wow, you people are really young.

http://www.etalabs.net/sh_tricks.html

stouset2• 2 years ago

I love this example, because it highlights how absolutely cursed shell is if you ever want to do anything correctly or robustly.

In your example, newlines and spaces in your filenames will ruin things. Better is

    find … -print0 | while read -r -d $'\0'; do …; done

This works in most cases, but it can still run into problems. Let's say you want to modify a variable inside the loop (this is a toy example, please don't nit that there are easier ways of doing this specific task).

    declare -a list=()

    find … -print0 | while read -r -d $'\0' filename; do
        list+=("${filename}")
    done

The variable `list` isn't updated at the end of the loop, because the loop is done in a subshell and the subshell doesn't propagate its environment changes back into the outer shell. So we have to avoid the subshell by reading in from process substitution instead.

    declare -a list=()

    while read -r -d $'\0' filename; do
        list+=("${filename}")
    done < <(find … -print0)

Even this isn't perfect. If the command inside the process substitution exits with an error, that error will be swallowed and your script won't exit even with `set -o errexit` or `shopt -s inherit_errexit` (both of which you should always use). The script will continue on as if the command inside the subshell suceeded, just with no output. What you have to do is read it into a variable first, and then use that variable as standard input.

    files="$(find … -print0)"
    declare -a list=()

    while read -r -d $'\0' filename; do
        list+=("${filename}")
    done <<< "${files}"

I think there's an alternative to this that lets you keep the original pipe version when `shopt -s lastpipe` is set, but I couldn't get it to work with a little experimentation.

Also be aware that in all of these, standard input inside the loop is redirected. So if you want to prompt a user for input, you need to explicitly read from `/dev/tty`.

My point with all this isn't that you should use the above example every single time, but that all of the (mis)features of shell compose extremely badly. Even piping to a loop causes weird changes in the environment that you now have to work around with other approaches. I wouldn't be surprised if there's something still terribly broken about that last example.

dotancohen• 2 years ago

  > I think that when someone uses ls instead of a glob it means they most probably don't understand shell.

In 25 years of using Bash, I've picked up the knowledge that I shouldn't parse the output of ls. I suppose that it has something to do with spaces, newlines, and non-printing characters in file names. I really don't know.

But I do know that when I'm scripting, I'm generally wrapping what I do by hand, in a file. I'm codifying my decisions with ifs and such, but I'm using the same tools that I use by hand. And ls is the only tool that I use to list files by hand - so I find it natural that people would (naively) pick ls as the tool to do that in scripts.

notnmeyer• 2 years ago

exactly—well said

AtlasBarfed2• 2 years ago

What I don't understand is why in the two most popular Unix flavors we have not got something like a json list output or something else that is parsable.

Is it really that difficult to add --json as a flag?

PaulHoule2• 2 years ago

I went through a phase when I really enjoyed writing shell scripts like

  ls *.jpg | awk '{print "resize 200x200 $1 thumbnails/$1"}' | bash

because I never got to the point where I could remember the strange punctuation that the shell requires for loops without looking up the info pages for bash whereas I've thoroughly internalized awk syntax.

Word is you should never write something like that because you'll never get the escaping right and somebody could craft inputs that would cause arbitrary code execution. I mean, they try to scare you into using xargs, but I find xargs so foreign I have to read the whole man page every time I want to do something with it.

hifromwork6• 2 years ago

I encourage you to give it a try again. Almost every use of xargs that I ever did looked like this:

ls *.jpg | xargs -i,, resize 200x200 ,, thumbnails/,,

I just always define the placeholder to ,, (you can pick something else but ,, is nice and unique) and write commands like you do.

projektfu• 2 years ago

Better is something like

  find . -maxdepth 1 -name "*.jpg" -exec resize 200x200 "{}" "thumbnails/{}" \;

which works for spaces and probably quotes in filenames I am not sure about other special characters.

chrsig• 2 years ago

Commands can have a maximum number of arguments. Try globbing on a directory with millions of files.

anthk• 2 years ago

Sane people will just use find and/or xargs.

arp2423• 2 years ago

Usually the pattern is "for f in [glob]", which doesn't have that issue. Running "ls" on a directory is little more than "for f in *; echo $f" so there's little advantage to using "ls".

Also: "find -exec {} \+" will take ARG_MAX into account, and may be much faster depending on what you're doing.

stephenr• 2 years ago

One advantage: `ls -i` gives you the file's inode in a POSIX portable way. If you glob and then look it up individually for each file, you'll need to be aware of which tool (and whether it's GNU or BSD in origin) you use on which platform.

In general yes globbing is better for iterating through files. But parsing `ls` doesn't necessarily mean the author doesn't know shell. It might mean they know it well enough to use the tools that are made available to them.

seanhunter• 2 years ago

It's not the case very often these days but it used to be quite simple to blow up your script globbing in a directory with a lot of files and you can still hit the limit if you pass a glob to some command because it can blow up trying to execve() Here's more details of the issue and some workarounds https://unix.stackexchange.com/questions/120642/what-defines...

amelius4• 2 years ago

I feel like Unix utilities should provide a standardized way to generate machine-readable output, perhaps using JSON.

db48x7• 2 years ago

The same information is already available in a machine–readable format. Just call readdir. You don’t need to run ls, have ls call readdir and convert the output into JSON, and then finally parse the JSON back into a data structure. You can just call readdir!

amelius4• 2 years ago

I know, but it would be so great if __every__ Unix utility just had the same type of output. By the way, ls does more than just readdir.

masklinn• 2 years ago

`find` is also an option, or shell globs.

emmelaich2• 2 years ago

Can you call readdir() from a shell easily?

WRT format, I'd prefer csv.

DonHopkins6• 2 years ago

That doesn't solve the problem that bash is completely useless for manipulating JSON.

It certainly would make writing Python scripts that need to interact with other programs easier. But Python doesn't desperately NEED to interact with so many other programs for such simple tasks like enumerating files or making http requests or parsing json, the way bash does.

Kwpolska3• 2 years ago

Bash is useless at JSON now. There's nothing stopping Bash from introducing native JSON parsing.

bpshaver• 2 years ago

https://kellyjonbrazil.github.io/jc/docs/parsers/ls

Aerbil3133• 2 years ago

What to do instead: Use Nushell.

I finally started really using my shell after switching to it. I casually write multiple scripts and small functions per day to automate my stuff. I'm writing scripts I'd otherwise write in python in nu. All because the data needs no parsing. I'm not even annotating my data with types even though Nushell supports it because it turns out structured data with inferred types is more than you need day-to-day. I'm not even talking about all the other nice features other shells simply don't have. See this custom command definiton:

  # A greeting command that can greet the caller
  def greet [
    name: string      # The name of the person to greet
    --age (-a): int   # The age of the person
  ] {
    [$name $age]
  }

Here's the auto-generated output when you run `help greet`:

  A greeting command that can greet the caller

  Usage:
    > greet <name> {flags}

  Parameters:
    <name> The name of the person to greet

  Flags:
    -h, --help: Display this help message
    -a, --age <integer>: The age of the person

It's one of the software that only empowers you, immediately, without a single downside. Except the time spent learning it, but that was about a week for me. Bash or fish is there if I ever need it to paste some shell commands.

db48x7• 2 years ago

Parsing, or the lack thereof, is not the point. The point is that standard shells already provide all the tools you need for dealing with lists of files. Want to do something for every file? Write this:

    shopt -s nullglob
    for f in *; do
       …
    done

But never this:

    for f in $(ls); do
       …
    done

They look similar, but the latter runs ls to turn the list of files into a string, then has the shell parse the string back into a list. Even if the parsing was done correctly (and it isn’t), this is still extra work. Looping over the glob avoids the extra work.

Aerbil3133• 2 years ago

I have to say this is very unintuitive. In Nushell, you'd do:

  ls | each { ... }

Another examples I don't need to explain, which would be far harder in stringly typed shells:

  ls | where type == file and size <= 5MiB | sort-by size | reverse | first 10

  ps | where cpu > 10 and mem > 1GB | kill $in.pid

It's immediately obvious what you need to do when you can easily visualize your data:

  > ls
  ╭────┬───────────────────────┬──────┬───────────┬─────────────╮
  │ #  │         name          │ type │   size    │  modified   │
  ├────┼───────────────────────┼──────┼───────────┼─────────────┤
  │  0 │ 404.html              │ file │     429 B │ 3 days ago  │
  │  1 │ CONTRIBUTING.md       │ file │     955 B │ 8 mins ago  │
  │  2 │ Gemfile               │ file │   1.1 KiB │ 3 days ago  │
  │  3 │ Gemfile.lock          │ file │   6.9 KiB │ 3 days ago  │
  │  4 │ LICENSE               │ file │   1.1 KiB │ 3 days ago  │
  │  5 │ README.md             │ file │     213 B │ 3 days ago  │
  ...

CJefferson2• 2 years ago

How do I replace:

    for f in $(cd subdir; ls); do
       ...
    done

?

probably_wrong• 2 years ago

I think there's a middle point where you want to do something that's complex enough that a glob won't cut it but simple enough that switching languages is not worth it.

I think the example of "exclude these two types of files" is a good case. I often have to write stuff like `ls P* | grep -Ev "wav|draft"` which doesn't solve a problem I don't have (such as filenames with newlines in them) but does solve the one I do (keeping a subset of files that would be tricky to glob properly).

In my experience 95% of those scripts are going to be discarded in a week, and bringing Python into it means I need to deal with `os.path` and `subprocess.run`. My rule of thumb: if it's not going to be version controlled then Bash is fine.

OrderlyTiamat• 2 years ago

You might enjoy a variety of `find` based commands, e.g. `find -maxdepth 1 -iregex ".*\.(wav|draft)" | xargs echo "found file:"`

This uses regex to match files ending in .wav or .draft (which is what I interpreted you to want). Xargs then processes the file. You could use flags to have xargs pass the file names in a specific place in the command, which can even be a one liner shell call or some script.

So the "find <regex> - xarg <command>" pattern is almost fully generally applicable to any problem where you want to execute a oneliner on a number of files with regular names. (I think gnu find has no extended regex, which is just as well- thats not a "regular expression" at that point)

bheadmaster3• 2 years ago

You might enjoy a variety of `find` based commands, e.g. `find -maxdepth 1 -iregex ".*\.(wav|draft)" | xargs echo "found file:"`

Find can even execute commands itself without using `xargs`:

     find -maxdepth 1 -iregex '.\.\(wav\|draft\)' -exec echo "found file:" {} \;

arp2423• 2 years ago

in zsh you can use:

  P*~*wav*~*draft*

This looks a bit obscure due to lack of spaces, but it's simpler than it seems; the pattern is:

  [glob] ~ [exclude glob]

  P* ~ *wav* ~ *draft*

The ~ being the negate operator, which can be added more than once.

It's essentially the same as "grep -Ev" or "find -iregex".

It's a lot less typing than find, and also something you're likely to use interactively once you're used to it, so it feels very natural.

bheadmaster3• 2 years ago

It's not necessary to bring Python into it, Bash can handle filenames with weird characters properly if you know how to use it.

E.g. instead of `ls | grep -Ev 'wav|draft'`, you'd have to do something like

    for filename in *; do
        if grep -E 'wav|draft' >/dev/null <<< "$filename"
        then : # ...
        fi
    done

Of course, it's more convoluted, but when you're writing scripts that might be used for a long time and by many people, it helps to know that it is possible to write robust things. Tools like shellcheck certainly help.

PeterWhittaker• 2 years ago

grep -q and you won’t need the redirect of stdout.

ykonstant8• 2 years ago

The above is perfectly fine for small directories, but in general the preferred way to loop over files is with find:

  find . ! -name . -prune                   \
           -exec grep -qE 'wav|draft' {} \; \
           -exec "${action}"             \; ;

Edit: I missed the herestring in the original code, so the above is wrong as mentioned in the comments; if your find has regex, you can use it to save one grep:

  find . ! -name . -prune              \
           -regex '.*wav.*\|.*draft.*' \
           -exec "${action}"        \; ;

Otherwise you can call sh to printf the filename into a grep.

However, the point of my post is that find can perform seek, filter and execute, and should be used for all three unless it is really impossible (which is unlikely).

some_random2• 2 years ago

At that point I think you need to ask yourself why you're using Bash to begin with. If it's just meant to be a quick script that's run occasionally then this is good but probably overkill. If it's going into prod to be run regularly as part of business critical, then it should be in a language that has a less convoluted way to _ls a directory_. There's an inflection point somewhere in there, where it is depends on you.

some_random2• 2 years ago

Before you write anything, you need to think about the cost of it breaking and the chance of it breaking, and Bash scripts in VC tend to maximize both. I like that heuristic a lot.

mcc1ane• 2 years ago

https://mywiki.wooledge.org/BashPitfalls#for_f_in_.24.28ls_....

teddyh• 2 years ago

Many people turn to globbing to save them, which is usually better, but has some problems in case of no matches. But, for Bash, you can do this to fix it:

  shopt -s failglob

billpg• 2 years ago

Why do you want to put LF bytes into filenames?

Using magic, I've renamed any files you have to remove control characters in the name and made it impossible to make any new ones. (You can thank me later.)

What can't you do now?

tredre3• 2 years ago

Preventing certain characters in filenames would solve a lot of issues, from security issues to wasted time all around.

But for whatever reason, when it is suggested, you get many people chiming in that "filenames should be dumb bytes, anything allowed except / !"

hifromwork6• 2 years ago

But for whatever reason, when it is suggested, you get many people chiming in that "filenames should be dumb bytes, anything allowed except / !"

I guess the issue is not what filenames should be, but what filenames are. In general, when interacting with files, you have to expect everything but `/` and nullbyte. Even if you forbid it on your machine, someone may mount a NFS drive and open you to the world of weird filenames. And you never know who uses your code.

And the unicode itself is weird anyway - for example you may have normalised and denormalised names which may be the same or different string depending on how you look at them[1]. And I hope you are not planning to restrict filenames to some anglocentric [a-z0-9_- ]*, because the world is much larger and you can't pretend unicode doesn't exist.

[1] https://eclecticlight.co/2017/04/06/apfs-is-currently-unusab... and many other cases

arp2423• 2 years ago

Latest POSIX "encourages" implementations to forbid using newlines in pathnames.

https://austingroupbugs.net/view.php?id=251

dTal• 2 years ago

I've put a file in my home directory named the entirety of your comment, newline included. Unfortunately I had to trim "except / !" to bring it to 255 characters.

Now at least when some tool or pipeline blows up horribly, it'll be hilarious.

badsectoracula• 2 years ago

I guess this is for shell scripts that need to work with "unsafe" filenames?

I've been using Linux since 1999 and i never came across a filename with newlines. On the other hand, pretty much all "ls parsing" i've done was on the command-line to pipe it to other stuff in files i was 100.1% sure would be fine.

kemitchell• 2 years ago

When teaching beginners shell, it's natural to teach `ls` for listing directory contents. It's also natural to extend from `ls` to `ls | ...` for processing lists of files

The important point to get across is that pipes let us build bigger commands from the commands we already know. If needed, you can back up later to teach patterns like `find [...] -exec`, `find [...] -print0 | xargs -0 [...]`, `find [...] | while read -r file; do [...] done` and so on.

There are all kinds of prerequisites to creating files with unusual names. Those barriers tend to mean beginners won't run into file name processing edge cases for a while. The exception will be files they download from the Internet. But the complexity there will usually be quote and non-ASCII Unicode characters, not newlines or other control codes.

In teaching, the one filename complexity I would try to get ahead of, preventively, is spaces. There was a time, way back when, when newbies seemed to expect to stick with short, simple filenames. These days they the people I've helped tended to be used to using spaces in file names in Finder and Explorer for office or school work.

zokier2• 2 years ago

I wonder if anyone has implemented kernel module or smth to limit filenames to sane set. Just ensuring that they are valid utf8 and do not contain any non-printables would be huge improvement. Sure some niche applications might break so its not something that can be made default, but still I think it would help on systems I control.

tmtvl• 2 years ago

Today I learned how neat find is:

  find ~/Music -iname 'p*' -not -iname '*age*' -not -iname '*etto*'
  find ~/Music -iname 'p*' -not -iregex '.*\(age\|etto\).*'
  find ~/Music -regextype posix-extended -iname 'p*' -not -iregex '.*(age|etto).*'

Not that I'm likely to ever use any of that in anger, but it's good to know if ever I do wind up needing it.

InsideOutSanta2• 2 years ago

Files and directories, once a reference to them is obtained, should not be identified by their path. This causes all kinds of problems, like the reference breaking when the user moves or renames things, and issues like the ones described in the article, where some "edge case" (and I'm using that term very loosely, because it includes common situations like a space in a file name) causes problems down the line.

You might say that people don't move or rename things while files are open, but they absolutely do, and it absolutely breaks things. Even something as simple as starting to copy a directory in Explorer to a different drive, and then moving it while the copy is ongoing, doesn't work. That's pathetic! There is no technical reason this should not be possible.

And who can forget the case where an Apple installer deleted people's hard disk contents when they had two drives, one with a space character, and another one whose name was the string before the first drive's space character?

Files and directories need to have a unique ID, and references to files need to be that ID, not their path, in almost all cases. MFS got that right in 1984, it's insane that we have failed to properly replicate this simple concept ever since, and actually gone backwards in systems like Mac OS X, which used to work correctly, and now no longer consistently do.

Kwpolska3• 2 years ago

IDs don't really solve many problems. The issues with scripts removing all your files were either caused by the absurd bash spaces and quotes rules, or by bash silently ignoring nonexistent variables. Those scripts would still need paths, since the ID of ~/.steam will be different for everyone. Scripts that need to work on more than one system, and human-authored config files, would still have paths. There are cases where you want to depend on the path, not the identity of the folder, and potentially swap the folder with something else without editing configuration.

Explorer needs to support local drives, with a lot of filesystems, including possibly third-party ones, but also network drives, FTP, WebDAV, and a bunch of other niche things. Not all of them have IDs and might not be possible to be extended. The cost is massive, solving it everywhere is impossible, and the benefit seems negligible to me (even though I fairly recently managed to eject a disk image (vhdx) in the middle of copying files onto it…)

InsideOutSanta2• 2 years ago

Earlier versions of Mac OS had APIs to retrieve the IDs of directories and files relevant for things like installing applications (such as the the System directory). It effectively never used paths to identify any files; if users opened a file, they'd use the system file picker, which would provide the application a file ID, not a path.

Similarly, things like config files would be identified by their name, not their path, because the directory containing configs was a directory the system knew about. As a result, no application needed to know the path to its own config files.

This meant there was no action that the system prevented you from doing to an open file, other than actually deleting that file. There was also no way for an installer to accidentally break your system because its code didn't take your drive, file, or directory names into account.

And, of course, there are file systems that don't use paths at all, like HashFS, a bunch of modern document management systems, or the Newton's Soup.

I get your point about interoperability with existing file systems, but I think it's perfectly acceptable to offer better solutions where possible, and fall back to paths for situations where that is not possible.

7bit• 2 years ago

Or use PowerShell where LS returns a bunch of objects, and say goodbye to string parsing forever.

chickenimprint2• 2 years ago

nushell is the superior structured data shell and it's cross-platform. https://www.nushell.sh/

redserk• 2 years ago

I've only used Powershell a little bit on Linux and Mac but it seems reasonably cross-platform.

On the surface, it looks like I'd be giving up the decently sized ecosystem of Powershell libraries for a new ecosystem without much support?

I'm interested in knowing what Nushell does differently since I'm wanting to find a better shell.

vips7L• 2 years ago

Nushell is way less powerful.

TacticalCoder• 2 years ago

Now of course having scripts and pre-commit hooks enforcing simple rules so that files must only use a subset of Unicode are a thing and do help.

Do you really think that, say, all music streaming services are storing their songs with names allowing Unicode HANGUL fillers and control characters allowing to modify the direction of characters?

Or... Maybe just maybe that Unicode characters belong to metadata and that a strict rule of "only visible ASCII chars are allowed and nothing else or you're fired" does make sense.

I'm not saying you always have control on every single filename you'll ever encounter. But when you've got power over that and can enforce saner rules, sometimes it's a good idea to use it.

You'll thank me later.

jcalvinowens• 2 years ago

Not sure how portable it is, but gnu ls has a flag to solve this problem trivially:

  --zero    end each output line with NUL, not newline

Aaron2222• 2 years ago

Not available in macOS `ls` unfortunately.

renewiltord2• 2 years ago

I just solve this by not having files like that on my computer. No spaces. No null chars.

hifromwork6• 2 years ago

Microsoft famously named "Documents and Settings" folder that way to defeat developers like you :).

Anyway check it, you might find you have files with spaces after all. For me it's:

  * /boot/System Volume Information
  * /proc/irq/126/PCIe PME
  * /sys/bus/platform/drivers/int3403 thermal
  * /etc/NetworkManager/system-connections/Hotel Xxx.nmconnection
  * /home/xxx/.cache/chromium/Default/Code Cache
  * and many others.

renewiltord2• 2 years ago

This is great. I `del mydocu~1` to kingdom come! Thank you. I meant in my home dir. I would never dare to presume anything about the rest. But it looks like Google demands that I be good at my job too.

waffletower• 2 years ago

Borkdude has a wonderful Clojure/Babashka solution in this space: https://github.com/babashka/fs

geophile3• 2 years ago

I wrote a pipe-objects-instead-of-strings shell: https://marceltheshell.org.

Not piping strings avoids this issue completely. Marcel’s ls produces a stream of File objects, which can be processed without worrying about whitespace, EOL, etc.

In general, this approach avoids parsing the output of any command. You always get a stream of Python values.

fooker2• 2 years ago

In general, this approach avoids parsing the output of any command.

Somewhere, there has to be validation phases. Just because you have objects, doesn't mean they are well formed.

https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-va...

It turns out proper validation is way harder than parsing. There is a reason text based interfaces and formats are so pervasive.

geophile3• 2 years ago

As long as Python does the right thing with globs, there is really no room for marcel to get it wrong. Not sure what additional validation you are thinking of.

lostmsu• 2 years ago

This is a problem I faced recently on Linux. You can use ip addr to see the list of your IPv6 addresses and their types (temporary or not, etc). But doing it programmatically from a non-C codebase is way more involved.

Nimitz14• 2 years ago

These sorts of pedantic exchanges are so pointless to me. We are programmers. We can control what characters are used in filenames. Then you can use the simplest tool for the job and move on with your life to focus on the stuff that actually matters. Fix the root cause instead of creating workarounds for the symptom.

tremon2• 2 years ago

Most of the time I avoid parsing ls, but I haven't found a reliable way to do this one:

  latest="$(ls -1 $pattern | sort --reverse --version-sort | head -1)"

Anyone got a better solution?

chenxiaolong• 2 years ago

This should work with any arbitrary filename:

    latest=$(printf '%s\0' <glob> | sort -zrV | head -zn1)

or with long args:

    latest=$(printf '%s\0' <glob> | sort --zero-terminated --reverse --version-sort | head --zero-terminated --lines 1

genrilz8• 2 years ago

What unix is this on? Neither the mac nor gnu manpages have a -z or --zero-terminated option for head.

genrilz8• 2 years ago

This ones a hard one. Since "--version-sort" isn't standard anyways, lets assume we can use flags which are common to BSD and GNU. Furthermore, lets assume bash or zsh so we can use "read -d ''".

In that case, how about:

  IFS='' read -d '' latest < <(find $pattern -prune -print0 | sort -z --reverse --version-sort)

midjji• 2 years ago

The bash code which creates the c file which gets the list of null terminated files in a directory and compiles it, and runs it, is easier to write and understand. Bash is a lousy language to do anything in, python is almost always available, and if not, then CC is.

cess11• 2 years ago

I don't know, this seems like a lot of words to avoid coming to the conclusion that there are many ways to skin a directory.

Most of the time it's fine to just suck in ls and split it on \n and iterate away, which I do a lot because it's just a nice and simple way forward when names are well-formed. Sometimes it's nicer to figure out a 'find at-place thing -exec do-the-stuff {} \;'. And sometimes one needs some other tool that scours the file system directly and doesn't choke on absolutely bizarre file names and gives a representation that doesn't explode in the subsequent context, whatever that may be, which is quite rare.

A more common issue than file names consisting of line breaks is unclean encodings, non-UTF-8 text that seeps in from lesser operating systems. Renaming makes the problem go away, so one should absolutely do that and then crude techniques are likely very viable again.

Tempest1981• 2 years ago

Recent discussion about the original "don't parse" page being referenced:

https://news.ycombinator.com/item?id=40692698 (10 days ago, 83 comments)

bandie91• 2 years ago

i searched through the page and have not found `find ... -printf "%M %n %u %g %s ...\0"` mentioned. this way you get ls(1)-like output, yet machine-parseable.

Why not parse `ls` and what to do instead

Related Stories