What's New in POSIX 2024

https://blog.toast.cafe/posix2024-xcu

by signa11 • 2 years ago

237 233 2 years ago

blog.toast.cafe

snvzz• 2 years ago

This is a surprisingly greedy POSIX update.

As someone who truly limits himself to POSIX when he can, I think they needed to push it forward to not become completely obsolete. I'm really sad `mktemp -d` and `set -o nullglob` didn't make the cut, but that's how it is, I guess.

ykonstant16• 2 years ago

A bespoke `mktempd` script is one of the first things I install in a new system. Fortunately, it is not too hard to make a `mktemp -d` compatible script with POSIX tools. `set -o nullglob` is another story :D

somat• 2 years ago

Hopefully nothing, posix is, or at least it should be, a descriptive standard. This is why posix is so terrible, and why posix is so great.

The way I feel posix, and other descriptive standards work best is when they describe what every one is already doing. This is opposed to prescriptive standards which try focus on how the "correct" way to do somthing, prescriptive standards tend to be over engineered and may or may not actually work.

see also: descriptive and prescriptive dictionaries. http://www.englishplus.com/news/news1100.htm

Flimm3• 2 years ago

Both prescriptive standards and descriptive standards have their uses. If POSIX is a prescriptive standard, then maybe another standard should exist that is descriptive.

lifthrasiir8• 2 years ago

Keep in mind that the Web standard eventually became prescriptive because descriptive standards failed to catch up. Likewise it can be argued that descriptive standards for the common OS interface are no longer usable.

zelphirkalt6• 2 years ago

That is also a way to never progress beyond the status quo.

Flimm3• 2 years ago

Yes! Finally! Let's treat filenames with new lines as errors! I'm so delighted with this decision.

enriquto6• 2 years ago

Next: spaces

lifthrasiir8• 2 years ago

Still much better than mojibaked names.

skissane2• 2 years ago

The original request was to ban all bytes between 1 and 31.

https://www.austingroupbugs.net/view.php?id=251

At some point they decided to narrow the change to just ban the newline character.

Which I personally think is a pity. Allowing escape in file names is a security risk because it enables you to embed ECMA-48 escape sequences in file names. Secure terminal emulators shouldn’t be made vulnerable by arbitrary escape sequences, but there are “too smart for their own good” terminal emulators out there that have escape sequences that let you do crazy things like run arbitrary executables.

ezoe3• 2 years ago

There are many non-UTF-8/16/32 character encoding used in the wild which use these value in multi-byte character encoding. These values are used in the wild.

I think the decision forbidding newline in pathname is also wrong. It may break tons of existing code.

IshKebab2• 2 years ago

Why is that an issue?

shakna• 2 years ago

Run a program to list a directory. Everything that interfaces with that, will assume newline delimiters. Similar assumptions are baked into a lot of software.

Enforcing that a newline isn't part of a path, ensures the security of those systems that are commonly relied on.

devit• 2 years ago

That's obviously impossible since it would break backward compatibility and the users' existing filesystems (and the Linux kernel will rightly never accept anything like that).

The only reasonable fix is to enhance bash and shell IDEs to track for each variable whether it could possibly include all filename-valid characters (e.g. if it comes from read with no options then it can't contain \n) and warn (off by default unless stderr is a terminal) if they can't and it's used as a filename (conservatively determined when used as arguments to processes), and also warn when using find without -print0, etc. noninteractively and perhaps interactively as well.

imrejonk• 2 years ago

This adds `set -o pipefail` to POSIX sh, which causes a whole pipeline to fail (non-zero exit code) if one or more of the commands in the pipeline fail.

throwaway984393• 2 years ago

Sad. Use of that option is almost always a mistake. It only leads to undebuggable silent failures.

Joker_vD4• 2 years ago

I'd rather both have this option and have it work reliably. It's ridiculous that

    export VAR=$(cmd1 | cmd2)

does not count as a pipefail when cmd1 or cmd2 fail but

    VAR=$(cmd1 | cmd2)

does, so the "correct" way to set an environment variable from a pipeline's output is actually

    VAR=$(cmd1 | cmd2)
    export VAR

ykonstant16• 2 years ago

Pipefail is useful and very hard to emulate on pure POSIX; you need to create named fifos, break the pipeline into individual redirections and check for error on each line.

And that is fine; but sometimes you want to treat a pipeline as a "single command" and then you can use pipefail to abort the pipeline on error. Then you can handle the error at the granularity of the entire pipeline without caring which part failed.

Lastly, I am confused as to the "silent" failures; maybe you are thinking of combining this with `set -e`? Then yes, that is bad and I recommend against the combination; but then again, I and most advanced scripters recommend against shotgunning `set -e` in the first place. Use it in specific portions of the script when appropriate, and use proper error handling otherwise.

akdor1154• 2 years ago

Holy balls that's like Christmas!

rightbyte2• 2 years ago

Really? Wont that break piping grep?

WJW3• 2 years ago

Probably, so don't `set -o pipefail` in scripts that pipe into grep.

zelphirkalt6• 2 years ago

Does it? It is not mentioned anywhere in the post. Can you post a reference to your source?

noselasd• 2 years ago

The post only have a few highlights. The Posix specs are only for paying IEEE customers though, but https://pubs.opengroup.org/onlinepubs/9799919799/ mentions it.

deskr2• 2 years ago

If you're writing scripts, use that and don't forget -e and -u

  -e      Exit  immediately if a pipeline (which may consist of a single simple command), a list, or a compound command (see SHELL GRAMMAR above), exits with a non-zero status

  -u      Treat  unset variables and parameters other than the special parameters "@" and "*" as an error when performing parameter expansion

ykonstant16• 2 years ago

For `set -u` I mostly agree. For `set -e` see my comment below and Greg's wiki: http://mywiki.wooledge.org/BashFAQ/105

enriquto6• 2 years ago

We’ve established that, yes, pathnames can include newlines. We have not established why they can do that. After some deliberation, the Austin Group could not find a single use-case for newlines in pathnames besides breaking naive scripts. Wouldn’t it be nice if the naive scripts were just correct now? Ok, that might be a bit much all at once. We’re heading there though!

Oh my god. This makes me so happy. This is the most lovely think I've read in the world of computing since the unix gods decided that newlines were to be a single character.

The philosophy underlying the sentence "Wouldn’t it be nice if the naive scripts were just correct now?" is incredibly positive. We are surrounded by arrogant jerks who break old code by aggressively enforcing stricter compliance of some stupid rules. But here come these posix heros who do the exact opposite: make old code correct! There is hope in mankind after all.

anal_reactor4• 2 years ago

It's a bandaid on a wider problem: the design of Unix shell is bonkers and the whole thing should be deleted. Why? Because I haven't seen any other tool ever have so many pitfalls. Take n random languages and m random developers and tell them to loop over a string array and print its contents, and count how many correct programs you get on average per language. There will be easy languages, then difficult languages, then a huge gap, then Unix shell because in your random sample you managed to get one guy who has PhD in bash.

blueflow9• 2 years ago

Someone needs to come up with a interactive shell first, one that is comparable in usability. Then we can think about replacing the unix shell.

I tried both python and lua interactively, but they are a pain when it comes to handling files. You have to type much more to get the same things done.

throwaway19972• 2 years ago

the design of Unix shell is bonkers

Compared to what?

enriquto6• 2 years ago

loop over a string array

Dear anal_reactor, what is a "string array"? I have used unix shells since nearly 30 years and never heard about them. And I consider myself a script-fu master!

There are two array-like constructions in the shell: list of words (separated by spaces) and list of lines (separated by newlines). Both cases are implemented as a single string, and the shell makes it trivial to iterate through its components.

dailykoder• 2 years ago

Works on my machine!

akira25012• 2 years ago

I haven't seen any other tool ever have so many pitfalls.

I haven't seen any other tool with so much general utility and availability.

to loop over a string array and print its contents

Is incredibly easy in bash and bash like shells. As highlighted the issue is that tools like 'ls' don't create "a string array." They create one giant string that has to be parsed. The rules in the shell are different than in other languages but it /will/ do most of the parsing for you, or all of it, if you do it carefully.

This is a fine tradeoff. As evidenced by it's wide usage and lack of convincing replacements.

vbezhenar3• 2 years ago

The main problem is using text as a common format between different applications.

First: text is not well defined. Is it ASCII? Is it UTF-8? Some programs can spew UTF-32 with proper locale configured, it's a mess.

Second: encoding and decoding of objects to text is not defined at all. Those problems with filenames is just one example. Using newline as a separator is a natural thing that is easy to implement, yet it is wrong.

In my opinion two things should be done:

1. Standardise on UTF-8. No other encodings allowed.

2. Standardise on JSON. It is good enough to serve as universal exchange format, tools like `jq` exist for some time now.

So any utility must read and write JSON objects with some standard env set. And shells can be developed with better syntax to deal with JSON. This way you can write something like

`ps aux | while read row; do echo ${row.user} ${row.pid}; done`

zelphirkalt6• 2 years ago

This should not be as downvoted as it is. In a way shell is broken. The brokenness is in that it requires each command to serialize and deserialize again, considering all the weird things that can happen with the "all is a string" kind of approach, instead of having a proper data interchange format or even sending objects to next steps in the pipeline. This behavior is what necessitates even thinking about the changes listed in the post. We wouldn't even have that problem, if the design of shell was better thought out. Now we are dealing with decades of legacy built on these shaky foundations. I hate to admit it, but seems at least this aspect Powershell got right, whatever one may think about the rest of it.

nneonneo4• 2 years ago

Rather unfortunately, I happen to have a handful of files on my machine with newlines in them (the filenames were programmatically generated from a summary of their contents). I loathe the possibility that my shell tools are going to suddenly crash when confronted with these weird files, rather than just producing some slightly silly output. I wish we'd standardized the behaviour of just escaping such characters as `\n/\r` or `^J/^M`...

nasretdinov• 2 years ago

The thing is, it's hard to predict what would happen to those scripts regardless... E.g. try naming your files "-rf" and see how many things break :)

ykonstant16• 2 years ago

In academia, I get (and used to create) pdfs with names like:

"On the number of

associative foobars

of degree blah -

Johnson and Anderson.pdf"

all the time. It is very convenient for non-technical academics to have a descriptive file name, and to be able to see it entirely in the navigator they use newlines.

jraph4• 2 years ago

They did the right thing for this: make the tools fail on file creation, but not on existing files.

I guess it's still advisable to rename those files, I don't know how things like cp, mv or rsync will behave when copying such files in the future.

stouset2• 2 years ago

Dude, just fix the filenames.

funcDropShadow• 2 years ago

Of course, there is an xkcd for that: https://xkcd.com/1172/

ezoe3• 2 years ago

Don't assume UTF-8 is the only character encoding used in the wild. There are character encoding with leading bytes not easily detectable like UTF-8.

arghwhat3• 2 years ago

In 2024, if you don't get the correct result decoding a text as UTF-8, the bug is the text, not the decoding. And luckily, adoption of UTF-8 in the past 30+ years have gone will enough that you don't need to worry.

Caveats for cursed hardware standards demanding two-byte encodings like USB.

account424• 2 years ago

Their proposed solution is not compatible with reality though where POSIX does not get to define what kind of files exist on filesystems you need to work with.

All they did is introduce new error cases in C programs while not actually fixing anything for shell scripts.

If anything, it's going to result in more exploits as people write shell scripts with the assumption that newlines cannot appear in filenames.

quotemstr5• 2 years ago

In the real world, nobody writes shell scripts that handle newlines in filenames.

zokier• 2 years ago

But they did not make old code correct. Filenames are still allowed to contain newlines. Shell scripts still need to be prepared to deal with that. Nothing really changed, they just added a feel-good half-measure.

quotemstr5• 2 years ago

It's a step in the right direction. You have to understand that for decades a vocal group of Unix die-hards has opposed any limitations whatsoever on the bytewise content of file names. The newline restriction in this latest version of POSIX may be modest, but it represents a dam breaking. When (obviously) the sky doesn't fall, the next version of POSIX will have a lot more filename cleanup.

janderland• 2 years ago

This is pretty standard for a human run system. Gotta make the human feel good about an idea before they can do said idea.

If you’re not familiar with humans, there are several manuals available online.

hwc• 2 years ago

Now do that with all whitespace!

relistan2• 2 years ago

The history at the beginning of this is not correct. Two examples: the assertion that there was one compatible UNIX prior to United States vs AT&T, the statement that GNU and BSD started that same year. Very, very off.

unixhero• 2 years ago

Okay, but you would add more value if you could also state what is the correct order if things.

relistan2• 2 years ago

https://en.m.wikipedia.org/wiki/History_of_Unix#/media/File%... is a good visual of (many of, not all) the various versions of UNIX and when they were released. BSD was first released in 1978. United States v. AT&T was implemented in 1984 (judgment 1982) GNU was first created in 1983.

johnisgood4• 2 years ago

Anyway, POSIX 2024 now requires c17, and does not require c89

I wish it would have been c99. What does c17 add exactly, more C++-esque complexity or not? Why was it not c99 (or perhaps even c11) over c17? Genuine questions.

lifthrasiir8• 2 years ago

What does c17 add exactly, more C++-ish bullshit or not?

Multithreading support and such (atomics, thread-local storage and a guarantee that `errno` is in TLS), explicitly aligned types and allocations, dedicated types for strings known to be Unicode, _Noreturn, _Generic, _Static_assert, anonymous structs and unions in the nested position, quick_exit, timespec, exclusive mode ("x") in f[re]open, CMPLX macros.

I'm not even sure which one can be C++-ish bullshit possibly except for about two points:

- Multithreading does seem farfetched for causal users. In fact, I do think it could have been minimized without any actual harm, but multithreading itself needed to be specified because it greatly affects a memory model. (Before C11, C had no thread-aware memory model and different threading implementations were subtly different beyond what the standard stated.) Even JavaScript, originally without no notion of threads, eventually got a thread-aware memory model due to shared web workers. But that never meant JS itself need multithreading support in its standard library, and C could have done the same.

- `_Generic` is even more debatable, though I believe it was the only way forward when we accept <tgmath.h>, which is known to be a response to Fortran (other responses include `restrict`) and was impossible to implement in the portable manner before C11. As long as it retains its scary underline and title case, I guess it's fine.

johnisgood4• 2 years ago

You quoted me before my edit, but fair enough. I do like the "atomics" support.

"guarantee that `errno` is in TLS"

I suppose that does not mean that I can just avoid setting errno to 0 before calling a function after which I check for errno, right?

Yeah, I do have an issue with stuff like "_Generic" but I assume I can just simply not use it.

What is "quick_exit" exactly and what does it solve?

As for multithreading, I stick to phtread. Is any of the new features a replacement for that or what?

At any rate, why C17 over C11 then?

gpderetta2• 2 years ago

Most importantly posix already has existing multithreading facilities in posix threads, so it is imperative that they are reformulated in term of the C++11/C11 memory model.

cryptonector5• 2 years ago

guarantee that `errno` is in TLS

I mean, that is already true.

ggm2• 2 years ago

File names with / in them

oguz-ismail5• 2 years ago

Nitpick re: https://blog.toast.cafe/posix2024-xcu#fn:6

    SRC != ls *.c

is fine in a makefile as far as POSIX is concerned, because:

Applications shall select target names from the set of characters consisting solely of slashes, hyphens, periods, underscores, digits, and alphabetics from the portable character set

guerrilla• 2 years ago

Why was `isascii()` removed?

(Listed in the Sortix article linked in OP.)

oguz-ismail5• 2 years ago

It would yield false-positives with non-UTF-8 encoded text. Big5 <https://en.wikipedia.org/wiki/Big5#Encoding> in particular was notorious for using ASCII values for trailing bytes. I don't know if it's still in use or if there are others.

EdSchouten• 2 years ago

strlcpy()!

pelorat• 2 years ago

TIL the POSIX standard is still updated. Does it still suffer from the issues that make Linux break POSIX compatibility in some areas because they consider it a flawed standard?

chasil5• 2 years ago

  - find(1p) now supports -print0
  - xargs(1p) now supports the -0 argument
  - newlines in filenames now should throw errors in many utilities
  - a complier implementing the c17 standard is now required
  - ulimit is expanded
  - renice can use relative values
  - a timeout utility has been added
  - make adds support for $^ $+ ::= :::= != ?= +=
  - logger is improved
  - gettext is adopted
  - readlink and realpath are adopted
  - rm now supports -d to remove empty directories and -v for verbose
  - various improvements to printf, sed, test

greyw• 2 years ago

Looks like the BSD-family will have some implementing to do.

sneed_chucker• 2 years ago

Strict adherence to POSIX isn't a goal of any of the current BSDs is it?

chasil5• 2 years ago

I just booted OpenBSD 7.0 (which is a bit dated).

The find utility has print0, and xargs has -0. Notibly, xargs also has -P for running processes in parallel.

rm has both -d and -v.

The renice command appears to be able to use relative adjustments with -n.

There is a timeout command.

There is a readlink command, but no realpath (but a manual page exists for it as a system call).

pabs3• 2 years ago

Since old-POSIX systems will be in use for some time, I wonder how many things will be able to switch to using the new capabilities. And how many OSes already support all of the new changes.

donatj• 2 years ago

To build an internationalized shell script I'll need to compile multiple .mo language files and distribute them along side the script itself.

For shell scripts part of a large system, that's probably fine. For small scripts, that's not very practical. You are not only adding a compilation step, you're also requiring distribution of multiple files. That's a pain.

It just kind of kills the convenience of a simple shell script. I would probably end up writing a makefile to manage all of this and at that point I am only a hop skip and jump away from using a compiled language instead of shell.

quotemstr5• 2 years ago

We’ve established that, yes, pathnames can include newlines. We have not established why they can do that. After some deliberation, the Austin Group could not find a single use-case for newlines in pathnames besides breaking naive scripts. Wouldn’t it be nice if the naive scripts were just correct now?

Finally. Now let's do the rest: https://dwheeler.com/essays/fixing-unix-linux-filenames.html

Filenames should be boring printable normalized UTF-8. I have never, not once, seen a good reason that a filename should be able to contain random binary gobbledygook

cryptonector5• 2 years ago

Filenames should be boring printable normalized UTF-8. I have never, not once, seen a good reason that a filename should be able to contain random binary gobbledygook

Ensuring normalization is hard. Where should you do it? There's only one good place: in the filesystem. But if you normalize on create then you'd better use the same form that everyone else uses, but, what's that? Input methods generally produce NFC, but there's no guarantee that they will not produce something else. HFS+ normalizes to NFD on create.

ZFS uses form-insensitivity -- much like case-insensitivity, but for form. The reason ZFS went this was exactly that HFS+ and input methods differ as to forms. I pushed hard for this way back when. IMO form-insensitivity is the best way forward.

But as for guaranteeing that filenames are UTF-8... that's much harder. The best thing to do is to not allow the use of non-UTF-8, non-ASCII, non-C locales -- not a guarantee, but pretty good.

quotemstr5• 2 years ago

Sure. Form-insensitivity is another good option. I'd actually argue for full case insensitivity too (like macOS), although I realize that it's probably a stretch.

nh2• 2 years ago

future editions will not require c17, but will simply require whatever C specification version is the most modern and already implemented by major toolchains

Is this really good?

If you can't rely on anything concrete being guaranteed, and it is open to interpretation what "modern" or "major toolchains" are, why have a standard?

cryptonector5• 2 years ago

The problem is that pathnames2 (as per section 3.254 of POSIX 2024) are just strings (meaning they can contain any bytes except the NUL character), [...]

Pathnames can neither contain NUL nor '/'.

Re: `find -print0` / `xargs -0`:

Previous POSIX releases have considered -print0 before, but never ended up adopting it because using a null terminator meant that any utility that would need to process that output would need to have a new option to parse that type of output.

What nonsense. Just add the `-0` or similar options as needed.

More precisely, this approach does not resolve our original problem. xargs(1p) can’t sort, and therefore we still have to handle that logic separately, unless sort(1p) also grows this support, even after read(1p). This problem continues with every other type of use-case. Importantly, it breaks the interoperability that POSIX was made to uphold.

More nonsense.

A bunch of C functions3 are now encouraged to report EILSEQ if the last component of a pathname to a file they are to create contains a newline (put differently, they’re to error out instead of creating a filename that contains a newline).

Ok, that's tolerable. Ditto utilities (notice here they were able to make a list of utilities).

chasil5• 2 years ago

Note that GNU sort has...

-z, --zero-terminated: end lines with 0 byte, not newline

InfiniteRand3• 2 years ago

I kind-of would like to see a POSIX-strict profile which incorporates commonsense (by commonsense I mean avoiding things that repeatedly over many years have tripped up programmers in frustrating ways) things like no newline in file names. Operating systems (or distributions) or could opt into this profile, and then someone programming on such an operating system could rely on the constraints of the profile and additional facilities could be added on that might need to rely on those constraints. Hopefully, gradually the use of the profile would spread.

BobbyTables2• 2 years ago

I really hate to say it, but the fretting about newlines used as delimiters after 50 years of misuse …

… makes PowerShell start to look damn good.

rurban• 2 years ago

EILSEQ for \n finally, but why not for unicode confusables? Path names are identifiers, and as such need to be identifiable. Meaning stricter rules than just buffers (not talking about strings).

Netch4• 2 years ago

Filename character set and its interpretation shall be controlled per directory or, at least, per FS. This pertains not only to permitted set like with or without LF, but to collation rules as well (including case insensitivity with cases like Turkish/Crimean/etc. I/ı and İ/i). Also this shall include workarounds for already existing problems: if a directory already contains files I1 and ı1, there shall be a technique to deal with them separately ever with Turkish locale.

But restricting this at syscall level is definite insanity, among with excuses.

What's New in POSIX 2024

Related Stories