lapsed_lisper

I've worked on projects with people who've insisted on these approaches, and I've never understood how this secures things in any deep way. As I understand the reasoning, open("/a/b",...) is discouraged in favor of openat(fd, "b", ...) because an attacker might be able to hijack /a. But as I see things, if an attacker has ever been able to hijack what /a contains or what "/a" refers to, they can change whatever "b" ends up naming, and the game is over.

As for the article's glnx_chaseat(), ISTM the 8 orthogonal flag bits are a warning indicator. If there need to be 256 ways to configure pathname resolution semantics, then there are 255 ways the flags compiled into program can be inappropriate for any particular use. Even if we stipulate that there's a problem here that both needs solving, if those flags aren't a hint that this is a flaky solution, I don't know what is.

So it has seemed to me as if the real security problem is the existence of a file system shared among unrelated sets of processes, and that if there is a secure alternative to that on existing operating systems, it probably looks more like embedding program configuration or data inside a program. (But this is handwavy: I'm envisioning stuffing data into ELF or Mach-O segments in signed binaries; some novel mechanism would need to be invented for shebang scripts.) But probably compartmentalizing all systems into distinct VMs is more practical than redesigning all software. (I would imagine that since the article's author works on Flatpak, they are motivated to want something less than VMs to serve as viable compartmentalization solutions, however.)

A while ago I stumbled across a technique for improving stream buffering that I wish more I/O library implementors knew about. Korn and Vo's sfio library (circa 1991) had a feature called "pooling", whereby distinct streams could be linked together. Any read or write operation to any stream in a pool implicitly synchronized all the other streams in the pool first. This way, when stdio and stderr were pooled, which was the default when both went to ttys, a write on stderr implicitly flushed stdout. I've implemented this feature for myself a couple times; it's fairly easy to do and basically eliminates the need to explicitly flush streams in client code.

Citation: https://archive.org/details/1991-proceedings-tech-conference... but note that the explanation of stream pools there is a little less precise and more general than really necessary. I believe that later versions of sfio simplified things somewhat, though I could be wrong. (I find their code fairly hard to read.)

Anyhow, ISTM a missed opportunity when new languages that don't actually use libc's routines for something reinvent POSIX's clunkier aspects.

Re: obtaining a legal copy of Genera, as of 2023 Symbolics still existed as a corporate entity and they continued to sell x86-64 laptops with "Portable Genera 2.0". I bought one from them then, and occasionally see them listing some on Ebay. (This isn't intended as an advertisement or endorsement, just a statement. I think it's quite unfortunate that Symbolics's software hasn't been made freely available, since it's now really only of historical interest.)

I don't know about the Unix certification process itself, but the Single Unix Specification explicitly mentions case-insensitivity among non-conforming file system behaviors that are allowed as extensions (in 2.1.1 item 4, third-to-last bullet):

https://pubs.opengroup.org/onlinepubs/9799919799/

So a conforming OS has to make case-sensitive file systems available (which MacOS does: you can create case-sensitive HFS or APFS volumes). But I'm not sure if a conforming OS instance (i.e., running system) has to have any case-sensitive mount points, and either way, AFAIK there's no practical and race-free way for a conforming application to detect whether any particular mount point behaves case-sensitively.

So I believe that as far as the standard goes, a conforming application might run on a conformingly-extended OS where no portion of the the file namespace behaves case-sensitively. IOW, a conforming application cannot rely on case-sensitivity for file names.

And then, since the type NIL is a subtype of all types, it's a subtype of CHARACTER. So because the type STRING is the union of all array types whose element type is a subtype of CHARACTER, an array that can't store any values is also a string. Oops.

(Also, just for onlookers, in ANSI Common Lisp, but not its ancestors or its sorta-sibling Emacs Lisp, characters are disjoint from integers. That's why the intersection of BIT and CHARACTER is empty.)

Yes, in Common Lisp, NIL is a value (it's a symbol, and by convention also the empty list).

But when used as a type specifier, NIL denotes the empty set. So no Lisp object is of that type, and an array with that element type cannot store any object.

I worked in Perl for ~2.5 years in the mid-2000s. It wasn't the language for me, but I liked, respected, and am still friends with colleagues who loved it. However, I was always dumbfounded by the experience that none of them could or even professed to be sure of what most code fragments did at a glance, even fragments that only used constructs in the base language. When they worked on existing code, they'd run it, tweak it, run it again, maybe isolate some into a small file to see what it did, look at perldoc, ask questions on IRC, etc. As a Lisp guy, I'm all for interactive & iterative development, but I also like learning tools so as to use them more effectively over time. I didn't find that learning Perl made me more productive with it over time (on a modestly sized, pre-existing code base that needed both maintenance and new feature development), and the Perl lovers I knew didn't seem to mind not having this as part of their work.

Anyhow, toward the end of my time there, I had to interview candidates. Because I came to believe that the above is how one had to work with Perl, I took to asking the ones who said they knew Perl, "Can the reverse builtin reverse a list?" (I won't spoil the answer for you.) Most would answer with "Yes" or "No"; 75% of them were mistaken. Either way I'd ask them "Suppose you weren't confident about that answer. How would you determine the truth?" IIRC, 90% of them said "I'd write this one-liner..." and (I swear) 100% of the one liners would give any reasonable person an impression of an answer that turns out to be incorrect. The ones that said "I'd check the perldoc" were the only ones I'd approve for subsequent interviews.

IDK how much this matters, but the Common Lisp standard doesn't mandate tail call elimination. So although many implementations offer it (usually as a compiler thing), it's conceptually an implementation-dependent detail that borders on a language extension: if your code actually depends on TCE for correct execution, that code might exhaust the stack under ordinary interpreter/compiler settings, and differently across different implementations. So for Common Lisp, if you want to use standardized language features standardly, it's quite reasonable to reach for iteration constructs rather than tail recursion.

I used to think this, but now mostly (but weakly) don't. Long options buy expressiveness at the cost of density, i.e., they tend to turn "one-liners" into "N-liners". One-liners can be cryptic, but N-liners reduce how much program fits on the screen at once. I personally find it easier to look up flags than to have to page through multiple screenfuls to make sense of something. In this respect, ISTM short options are a /different/ way of helping a subsequent reader, by increasing the odds they see the forest, not just the trees.

Among conceptually Unix-like OSes, at least one tried to do something along these lines: see http://www.bitsavers.org/pdf/apollo/SR10/011021-A00_Using_Yo... PDF pp 151 and following.

Unix's standard error is definitely not the first invention of a sink for errors. According to Doug McIlroy, Unix got standard error in its 6th Edition, released in May 1975 (http://www.cs.dartmouth.edu/~doug/reader.pdf). 5th Edition was released in June, 1974, so it's reasonable to suppose Unix's standard error was developed during that 11 month interval. By that time, Multics already had a dedicated error stream, called error_output (see https://multicians.org/mtbs/mtb763.html, dated October 1973).

All the same, I'd be willing to believe that Unix's standard error could have been an "independent rediscovery" of one feature made highly desirable by other features (redirection and pipes). It's not clear how much communication there was among distinct OS researcher groups back then, so even if other systems had an analogue, Bell Labs people might not have been aware of it.

S-expressions only really address a smallish part of what people seem to want out of configuration file syntaxes: a simple, recursive syntax.

Among the things that S-expressions don't address per se are the interpretation of tokens (e.g., numeric vs non-numeric token syntax), how non-numeric tokens may be interpreted as booleans, symbols, timestamps, etc.; whether to use alists vs. plists for associations and the semantics of any duplicate keys within an associative construct; how to specify the schema for a configuration object (required vs. optional elements & the types of each, etc.)

IOW, merely saying "use S-expressions" over-emphasizes syntax while under-emphasizing semantics.

ISTM one could use the same premises to reach the opposite conclusion, namely, that because awk is basically a subset of Perl (excluding CPAN, of course), many things are easier to read, often resulting in more regular, if sometimes longer code. :-)

(FWIW, I learned Perl before sed and awk, and when I was using Perl every day, it was easy enough to whip up one-liners and throwaway scripts. However, I find that as I stopped using Perl on a day-to-day basis about 17 years ago, I can't produce Perl without re-learning the language; but I can produce sed and awk a few times per year without any refresher. I suspect that -- for me -- the smallness of each of sed and awk has something to do with it. YMMV, of course.)

Right, good questions. Pure conjecture on my part: maybe it's just that 36 is the smallest integral multiple of 6 that also had enough bits to represent integers of the desired width?

If you're keen to go down the wikipedia hole, https://en.wikipedia.org/wiki/Six-bit_character_code and then https://en.wikipedia.org/wiki/BCD_(character_encoding) explain that IBM created a 6-bit card punch encoding for alphanumeric data in 1928, that this code was adopted by other manufacturers, and that IBM's early electronic computers' word sizes were based on that code. (Hazarding a guess, but perhaps to take advantage of existing manufacturing processes for card-handling hardware, or for compatibility with customers existing card handling equipment, teletypes, etc.)

So backward compatibility is likely the most historically accurate answer. Fewer bits wouldn't have been compatible, more bits might not have been usable!

(Why 6 bit codes for punch cards in 1928? Dunno. Perhaps merely the physical properties of paper cards and the hardware for reading them. This article talks about that stuff: https://web.archive.org/web/20120511034402/http://www.ieeegh...)

Does it really make that much difference if comments exist "inside" the embedded sublanguage vs. "outside"? After all, you can always construct a format specification by string concatenation, and in CL, you can trick the parser into doing this for you:

  (format stream t
          #.(concatenate 'string
                         ;; comment about the first piece
                         <piece 1>
                         ;; comment about the second piece
                         <piece 2>)
          ...)

  (format t "~
  ~0{ Because the previous line ended with a tilde, 
  the following newline is ignored. All of this text
  occurs inside an iteration construct that loops 
  zero times, so will not be output. However, it
  will consume one element from the list of arguments,
  so after this loop, we'll use tilde-colon-asterisk
  to back up one element, and let what comes next
  decide how to format it. So this is more or less
  a comment inside a CL FORMAT string.
  ~}~:*~
  ~S" 'Foo)

Out of curiosity, how often do you find yourself using CL-PPCRE's S-expression notation? (This is a genuine question: I've never felt a desire for an S-expression notation for regular expressions, so I'm curious what I'm missing out on.)

Anyhow, while it's certainly possible to parse FORMAT control strings into S-expressions, ISTM that if you want them to be invertible back into FORMAT strings, you'll end up with control structure and constant strings being contained within the S-expression, with data extraction as a separate concern. IOW, you won't get McDermott's preferred style of interwoven control, data extraction, and constant strings. For instance, you could have this FORMAT control string

  "~<~{~A~^, ~}.~:>"

  (:paragraph-fill ()
    (:iterate ()
       (:aesthetic)
       (:exit-if-list-exhausted)
       ", ")
    ".")

Richard P. Gabriel (one of the co-creators of Common Lisp, later a founder of the Lisp vendor, Lucid) has a few interesting things to say about CL's FORMAT in his Patterns Of Software, starting on page 101:

https://www.dreamsongs.com/Files/PatternsOfSoftware.pdf

What are the trade-offs? Format strings don’t look like Lisp, and they constitute a non-Lispy language embedded in Lisp. This isn’t elegant. But, the benefit of this is compact encoding in such a way that the structure of the fill-in-the-blank text is apparent and not the control structure.

IMO McDermott's OUT macro has precisely the drawbacks Gabriel predicts. While McDermott seems to think it's an advantage that

we no longer have to squeeze the output data into a form intelligible to format, because we can use any Lisp control structure we like

his example PRINT-XAPPING basically duplicates the logic for extracting data from the xapping structure as Steele's FORMAT call, but buries the data extraction into control structure alongside constant strings that go into the output.

And that's where I think FORMAT really a win: a FORMAT control deliberately separates the control flow and constant text from the data extraction logic. Presumably you could write functions that do the same thing using McDermott's OUT macro, but they'll be more verbose and no more enlightening; what's the point?

IMO, the bad parts Unix are of two semi-distinct kinds: the skin-deep kind, and the genetic defects.

The superficial problems are everywhere, easy to spot, and fun to complain about! The names of commands are obscure, the flags are inconsistent. Any utility's feature set is inherently arbitrary and its limitations are equally so. Just how to accomplish any particular task is often a puzzle, the traedeoffs between using different utilities for the same task is inscrutable, and the time spent contemplating alternative approaches within the tool kit is utterly worthless. Utilities' inputs, outputs, and messages aren't for humans, they're either for coprocesses or for the control layer running the utility; and so users are supposed to learn to conform to the software, rather than vice versa. There are a hodgepodge of minilanguages (sed, find, test, bc, dc, expr, etc.), but they're unnecessary and inefficient if you're working in an even halfway capable language (let's say anything at about the level of awk is potentially "halfway capable"); and so "shelling out" to such utilities is a code smell. The canonical shells are somewhat expressive for process control, but terrible at handling data: consequently, safe, correct and robust uses of the shell layer Unix is hard, maybe impossible; so nowadays most any use of the Unix shell in any "real" application is also a bad code smell.

I say these are skin-deep in the sense that in theory any particular utility, minilanguage, or shell can be supplanted by something better. Some have tried, but uptake is slow/rare. The conventional rationales for why this doesn't happen is economic: it's either not worth anyone's time to learn new tools that replace old ones, or the network-effect-induced value of the old ones is so high (because every installation has the old ones) that any prospective replacement has to be loads better to get market traction. I have a different theory, which I'll get to below.

But I also think there's a deeper set of problems in the "genetics" of Unix, in that it supports a "reductive" form of problem solving, but doesn't help at all if you want to build abstractions. Let's say one of the core ideas in Unix is "everything is a file", i.e., read/write/seek/etc. is the universal interface across devices, files, pipes, etc.). "Everything is a file" insulates a program from some (but not all!) irrelevant details of the mechanics of moving bytes into and out of RAM... by forcing all programs to contend with even more profoundly irrelevant details about how those bytes in RAM should be interpreted as data in the program! While it is sometimes useful to be able to peek or poke at bits in stray spots, most programs implicitly or explicitly traffic in in data relevant to that program. While every such datum must be /realized/ as bytes somewhere, operating on some datum's realization /as bytes/ (or by convention, as text) is mostly a place to make mistakes.

Here's an example: consider the question "who uses bash as their login shell?" A classical "Unixy" methodology to attacking such a problem is supposed to be to (a) figure out how to get a byte stream containing the information you want, and then (b) figure out how to apply some tools to extract and perhaps transform that stream into the desired stream. So maybe you know that /etc/passwd one way to get that stream on your system, and you decide to use awk for this problem, and type

awk -F: '$6 ~ "bash$" { print $1 }' /etc/passwd

That's a nicely compact expression! Sadly, it's an incorrect one to apply to /etc/passwd to get the desired answer (at least on my hosts), because the login shell in the 7th field, not the 6th. Now, this is just a trivial little error, but that's why I like it as an example. Even in the most trivial cases, reducing anything to a byte stream does mean you can use any general purpose tool to a problem, but it also means that any such usage is going to reinvent the wheel in exact proportion to how directly it's using that byte stream; and that reinvention is a source of needless error.

Of course the sensible thing to do in all but the most contrived cases is to perform your handling of byte-level representations with a dedicated a library that provides at least some abstraction over the representation details; even thin and unsafe abstractions like C structs are better than nothing. (Anything less than a library is imperfect: if all you've got is a separate process on a pipe, you've just traded one byte stream problem for another. Maybe the one you get is easier than the one you started with, but still admits the same kinds of incorrect byte interpretation errors.) And so "everything is a file", which was supposed to be great faciltiy to help put things together, is usually just an utterly irrelevant implementation detail beneath libraries.

And this gets me back around to why I think the superficial stuff hasn't changed all that much: I doubt that the "Unix way" of putting things together has really truly mattered enough to bother making the tools or shells substantially better. I got started on Unix in 1999, by which time it was already customary for most people I knew to solve problems inside a capable language for which libraries existed, rather than to use pipelines of Unix tools. (Back then there was lots of Perl, Java, TCL, Python, et al.; nowadays less Perl and TCL, more Ruby and JavaScript.) Sure, you've needed a kernel to host your language and make your hard drive get warm, but once you have a halfway capable language (defined above), if it also has libraries and some way to call C functions (which awk didn't), you don't need the Unix toolkit, or a wide range of the original features of Unix itself (pipes, fork, job control, hierarchical process structure, separate address spaces, etc.).

And that's just stuff related to I/O and pipes. One could look at the relative merits of Unix's take on the file namespace, Plan 9's revision of the idea, and then observe that "logical devices" addressed much of that set of problems as early by the early-to-mid 70s on TOPS-20 and VMS, without (AFAICT) accompanying propaganda about how simple and orthogonal and critical it is that there be a tree-shaped namespace (except that it's a DAG) and everything in the namespace works like a file (except when it doesn't).

My point is that people have said about Unix that it's good because it's got a small number of orthogonal ideas, and look how you can make those ideas can hang together to produce some results! That's all fine, though in practice the attempt to combine the small number of ideas ends up giving fragile, inefficient, and unmaintanable solutions; and what you need to do to build more robust solutions on Unix is to ignore Unix, and just treat it as a host for an ecology of your own invention or selection, which ecology will probably make little use of Unix's Unix-y-ness.

(As to why Unix-like systems are widespread, it's hard not to observe some accidents of history: it was of no commercial value to its owner at a moment when hardware vendors needed a cheap operating system. Commercial circumstances later changed so that it made sense for some hardware vendors to subsidize free Unix knockoffs. Commercial circumstances have changed again, and it still makes sense for some vendors to continue subisidizing Unix knockoffs. But being good for a vendor to sell and being good for someone to use can very often be different things...)