bodyfour

It's really easy to run for yourself: https://github.com/DoctorWkt/pdp7-unix/

Don't expect it to do much, but it's fascinating if you're interested in OS history.

The one exception I'd make to this is Sodium-ion which does seem to have some chance of reaching manufacturing scale: https://www.batterytechonline.com/materials/5-key-takeaways-...

Indeed, that was the way it originally worked in all UNIXes: https://github.com/dspinellis/unix-history-repo/blob/Researc...

It was a then-important optimization to do the most common operations with macros since calling a function for every getc()/putc() would have slowed I/O down too much.

That's why there is also fgetc()/fputc() -- they're the same as getc()/putc() but they're always defined as functions so calling them generated less code at the callsite at the expense of always requiring a function call. A classic speed-vs-space tradeoff.

But, yeah, it was a mistake that it originally used a "char" to store the file descriptor. Back then it was typical to limit processes to 20 open files ( https://github.com/dspinellis/unix-history-repo/blob/Researc... ) so a "char" I'm sure felt like plenty.

The creator of MINIX, Andrew Tanenbaum, asked the community to choose between Stevie and Elvis to be adopted as the main text editor for their OS. Elvis was chosen and it's the default text editor on MINIX until today.

Point of order: Minix switched to BSD nvi in 2013 https://github.com/Stichting-MINIX-Research-Foundation/minix...

Not that it matters -- Minix itself hasn't had a commit since 2018 -- but the last five years of its life were spent without Elvis

Waterloo&City seems to be missing too

The issue is that "const" didn't exist in the earliest forms of C... and even when it became available not everybody started using it.

So you might have a function that doesn't have proper "const" qualifications in its prototype like:

  void my_log(char *message);

  my_log("Hello, World!");

A few months ago I happened to install Debian/unstable on a G4 mini. ppc32 is no longer a supported architecture -- purely "what you get is what you get".

Still, the process was mostly painless. Everything I needed worked out of the box.

What is annoying to me is that in this internet-connected age, the storage units I see still don't have better per-unit security.

Just a phone alert to say "door to unit #xyz has been opened" would be a huge improvement. Wire up a cheap webcam for extra credit.

in that case, a compiler still must set something up to fulfil the main `noexcept` promise - call `std::terminate()`

This is actually something that has been more of a problem in clang than gcc due to LLVM IR limitations... but that is being fixed (or maybe is already?) There was a presentation about it at the 2023 LLVM Developer's meeting which was recently published on their youtube channel https://www.youtube.com/watch?v=DMUeTaIe1CU

The short version (as I understand) is that you don't really need to produce any code to call std::terminate, all you need is tell the linker it needs to leave a hole in the table which maps %rip to the required unwind actions. If the unwinder doesn't know what to do, it will call std::terminate per the standard.

IR didn't have a way of expressing this "hole", though, so instead clang was forced to emit an explicit "handler" to do the std::terminate call

not sure about other *nixes

Should be available on any UNIX, it was added to V7 UNIX back in the 1970s: https://github.com/dspinellis/unix-history-repo/blob/Researc...

Even before that, it existed as a standalone text file https://github.com/dspinellis/unix-history-repo/blob/8cf2a84... This still exists on many systems -- for instance as /usr/share/misc/ascii on MacOS

Surely you mean Ultrix, not OSF/1? Unless you're misremembering the hardware model...

After seeing how term limits work in the California state legislature, I stopped being a fan of them.

The first (small) problem is that it caused unneeded intra-party drama. Effective assembly members would end up forced to seek a "promotion" into the state senate if they wanted to stay in politics. This often meant challenging members of the same party. This ends with people constantly fighting their "allies" to scramble up the greased pole, rather than doing more useful work.

Of course, that can happen even without term limits (politicians are the ambitious sort) but they definitely accelerate the effect.

The worse problem is that all of the legislators are now short-timers. But do you know who aren't newbs? The lobbyists! Since they're now the only ones around with deep experience, they invariably get even more involved with crafting laws.

This actually dove-tails into the other problem with the lobbying industry: the revolving door from legislator to the lobbying firm. Even without term limits this happens all of the time. It's very common for a retiring US House member to immediately get a lucrative job lobbying their former colleagues. However in a term-limited legislative body this only gets worse. Not only do lobbyists become more powerful, but term limits provide a guaranteed flow of politicians needing a new job.

So, if you find yourself as a newly-elected politician in such a system and you actually want to make a difference, probably your best bet is to immediately find some lobbyists to get sweet with. They are the only ones with the experience to make the political machinery work, and they're probably your future employer as well.

So at least in my observation, the ultimate effect of term-limits is to transfer power from democratically elected representatives to well-funded special interests. By un-entrenching the politicians you're accidentally making another group even more entrenched.

By contrast, I think the very top legislators are ones that become a true expert in their field of interest. Imagine somebody who has been working on, say, education policy for decades. They know every policy detail, all of the stakeholders, all of the experts. In a world of term limits, how will such a person ever emerge?

All of the above is specifically about legislative term limits. I believe the case for executive term limits is much stronger.

IFUNC is supported on several architectures, including ARM64.

The malicious code that the xz backdoor inserts into the library is a compiled x86_64 object file so it only is targeting one platform.

I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup (and not in "curl | sh" scripts).

Yeah, I've been banging on that same drum for ages too... for example on this very site a decade ago: https://news.ycombinator.com/item?id=7213563

I'm honestly surprised that this autoconf vector hasn't happened more often... or more often that we know of.

Probably the person you're replying to is just confused because before it was standardized some snprintf() implementations returned -1 on overflow. If you were trying to be portable and defensive you'd need to check for either error return.

Not really a concern inside musl because those implementations are probably long gone and because it's calling its own snprintf() anyway.

I can understand reasons they won't (re-)enter the server space themselves.

However it is a shame they don't sell bare boards for specialist integrators to build these sorts of products with.

A great point, but even over a teletype you'd use "ex", not "ed".

It remained a valuable skill into the 1990s when many UNIX machines still had a separate /usr partition.

Many times I'd have to fix a server with a configuration issue causing /usr to not mount correctly on boot. When you're at a single-user shell and no /usr/bin/vi is available suddenly knowing the basics of /bin/ed becomes very useful.

Of course these days most UNIXes moved away from a separate /usr partition so if the root filesystem mounts you have a full-featured vim.

That much is true -- the difference between 6 and 7 registers is much larger than the benefit of going from 14 to 15.

However, even under zero register pressure having a frame pointer is still an extra register that needs to be touched on every function invocation, extra instructions taking space in the I-cache, etc. It's a small thing, but it's still a cost that has to be paid by all compiled code.

I'm not going to claim that re-enabling frame pointers was the wrong choice -- the people involved in the debate know the tradeoffs and I have to start with the assumption that I would have made the same decision if I were in their position. It does make me slightly sad, though. The idea behind removing frame pointers isn't that backtraces aren't important, it's that computing the frame pointer after-the-fact is possible -- i.e. for normal functions without alloca() or dynamically-sized stack arrays map %rip -> frame size.

The problem seems to be that despite years of experience with "no-frame-pointer" being the default I guess the profiling tools never got as reliable or good as the with-frame-pointer variants. My personal hope was that the problem would fade over time as tools improved, but it seems that's unlikely to ever happen. After all, once no-frame-pointer stops being the default there won't be any pressure for tools to improve. The towel has been thrown in.

when I had two or more files of the same size, and the size was larger than a few MB, they likely had the same content.

Yes, if the number of files is small enough, then "notice unique file sizes" is really the only optimization that ends up mattering much. If you have a few thousand files and they're each multiple gigabytes then hopefully you'll get lucky and no two will have the same size.

But the ideal tool should also try to handle the opposite case well too.

First, imagine if you have a huge collection of unique ~100KiB files. Now the "birthday paradox" means that size collisions are inevitable so optimizing the total I/O needed to prove two files are different starts to belp.

But the pathological case is even worse -- what if nearly all of your files are about the same size? For instance, suppose you have a program that is recording time-series data from a sensor and rotating the file every time it grows to 10MB. This sort of thing happens all the time dealing with scientific data sets -- you might have a directory with thousands of large files, all exactly the same size. If you want to quickly verify none of the files are dups, reading one block from each is far more efficient than hashing them all.

"The math" only matters for random collisions, which are effectively impossible (less likely than the CPU malfunctioning). However that tells you nothing about maliciously constructed files. Even if a hash function has no known collisions today, doesn't mean that they won't be found someday.

But as I tried to describe (probably in way too much detail) the real problem with "hash everything, compare hashes afterwards" is that it implies that you'll be doing I/O to read all of the file's contents even when it isn't needed to prove uniqueness. For a lot of common use cases (big files, few dupes) this can mean doing 1000x more I/O than you need.

Once you design the solution around avoiding unneeded I/O, you find that hashing also stops being useful.

(Warning: rambling ahead, since in the past I've spent a decent amount of time on the same problem)

using blake3 for hashing the content

Using any hash algorithm isn't a good design, at least for SSD storage.

First, it invites some degree of cryptographic risk. i.e. if a collision is ever discovered in the hash then your program can be tricked into discovering a false duplicate. Whether that is a problem depends on the use case, but isn't ideal.

Worse, though, is that it just doesn't make any sense algorithmically. Consider the simple example of having two 1TB files and you want to discover if they're identical. You could do a cryptographic hash of each of them and (barring any malicious collisions) tell whether they're the same.

However, now imagine that those two files differed in the very first byte. Now it seems that you could have figured out they're different a lot faster, right?

So what you really want to do is read both files chunk-by-chunk (probably some number of disk blocks at a time) so you can detect the files-are-different case early. (After all, the common case of files that differ is that they'll differ early!). You could still compare the chunks using a cryptographic hash, but now there is no benefit: you can just compare the two blocks of memory directly faster than you can take a crypto hash of them. C's memcmp() works fine but since you are probably working on fixed-sized aligned blocks you can do slightly better with a hand-rolled SIMD loop.

The one advantage that a cryptographic hash gives you is that it provides a memory-efficient way of reading all of file A and then all of file B. Therefore if disk seeks are expensive, it can be a benefit (again, if you can accept the risk of a malicious false-positive). However if the files are SSD backed and you have enough RAM to read a decent sized chunk of each file into memory simultaneously this ceases to be a problem.

To extend this from 2 files to many, first stat() all of the files and group them by file size. After all, two files of different sizes aren't going to ever be equal. You can think of the size as a "hash" of the contents that you get for free. Any files that are 0 bytes are (of course) duplicates of each other so you can just return those as "hits". Any file that has a unique size (which thankfully is often the common case) is not a duplicate of anything and you don't even have to open it. If you care about hardlinks, you also want to track the inode numbers at this step so you can avoid comparing two files that are actually the same inode.

Then for any group of files with the same size, read each block in turn. The tricky part is that you want to subdivide the group by the contents of the file. i.e. if you have 4 same-sized files and two have contents "AAAA..." and the other two have contents "BBBB..." then you didn't find any unique files yet, but you need to split the set of 4 files into two new subsets of 2 files each. Data-structure wise, keep a worklist of items containing of (1) a set of (at least two) files that could be duplicates and (2) how many bytes you've already verified are the same among them. Then when you encounter this "split" scenario you can just push a new worklist item and continue working on one of the groups.

The bit you need to be careful of is not introducing bad worst-case performance here in a hard case (e.g. you have a million potentially-duplicate files, and reading one block separates them into 5000 groups each with 200 members). Just a decent hash-table is enough with the key being the whole disk block.

Because maintaining this hash table adds a bit of complication, it can seem worthwhile to build a special-case for groups of exactly two files, where you can just do the simple read-and-compare. But then you can re-combine this by observing that the case where you have N files that are all the same is also worth optimizing for. So instead, just do the read-and-compare on all of the files until you find the first two that are different -- only then start building the hashtable when you have two different blocks and more files still to read. That way the common-ish case where you have many files all the same can be handled as fast as possible.

There are things that operating systems could provide that would make this even better:

1. It would be nice if it were easy to estimate the likely seek cost before picking an algorithm. If the file system would simply indicate whether it thought it was backed by spinning rust that would be great.

2. Also if you could ask for the filesystem to read a non-fixed number of bytes (without resorting to async-I/O) that would be helpful. ("Give me up to 1MB of the file, but stop early if you have to seek to a new extent"). Having the ability to basically read the file extent-by-extent instead of block-by-block would mean we could be seek-efficient while reading multiple files in parallel.

3. Finally, it would be great if there was some portable way to access any block-hash metadata the filesystem itself keeps. A filesystem that does its own deduplication work might already know that two blocks must have different contents without reading them because it already scanned them. On the flip-side, if the filesystem supports copy-on-write file snapshots then it could tell us in advance that two inodes are really the same underlying file before we open them.

and/or add overhead power delivery to major motorways, allowing trucks to have (relatively) small batteries for the non-motorway parts of their journeys: https://www.youtube.com/watch?v=_3P_S7pL7Yg

If you don't consider compute cost [...]

Yes, but what if you do? Imagine your hyper-specialzied API-heavy model takes 10x less resources to answer a question (or at least a question relevant to the task at hand) Won't it be more powerful to have a model that can run 10 times as fast (or run 10 instances in parallel)?

What if the ratio turns out to be 100x or 1000x?

So I agree that the cutting edge of "best possible AGI" might mean building the largest models we can train on massive clusters of computers and then run on high-end hardware. My hunch, though, is that models that can be run on cheap hardware and then "swarmed" on a problem space will be even more powerful in what they can perform in aggregate.

Again, it's just my hunch but right now I think everybody's predictions are hunches.

I'll actually go one bit further: even for a linear task that can't be "swarmed" in the same way, it could be that cheaper-per-token models could even do better on linear problem-solving tasks. Existing models already have the ability to use randomness to give more "creative", if less reliable, answers. This is inherently parallelizable though -- in fact Bard seems to be exposing this in its UI in the form of multiple "drafts". So what if you just ran 100 copies of your cheap-AI against a problem and then had one cheap-AI (or maybe a medium-AI) judge the results?

Or at the risk of a getting too anthropomorphic about it: imagine you as a human are writing a program and you get stuck on a tricky bit -- you know that the problem should be solvable but you've never doing anything similar and don't know what algorithm to start with. Suppose then you could tell your brain "Temporarily fork off 100 copies of yourself. 10 of them go do a literature review of every CS paper you can find related to this topic. 10 of you search for open source programs that might have a similar need and try to determine how their code does it. The other 80 of you just stare off into the middle distance and try to think of a creative solution. In two human-seconds write a summary of your best idea and exit. I'll then read them all and see if I/we are closer to understanding what to do next"

For us, this type of mental process is so alien we can't even imagine what it would feel like to be able to do. It might come completely natural to an AI, though.

There is no way around the bitter lesson.

Isn't there? I'm certainly not sure, based on the results published over the last weeks and months.

The giant GPT-{3.5,4} models show that if you make the model big enough and throw enough data at it you can produce an AI capable of conversing on basically any topic, in dozens of languages. There are plenty of different takes on how near-human its abilities are on specific tasks, but it's worth stepping back and appreciating how super-human the breadth of this knowledge is.

But it's also not clear if a mega-model is anything close to the most efficient way of storing knowledge. After all, you don't need to memorize every fact in Wikipedia if you know how to effectively search it.

And we're currently seeing a daily explosion in these capabilities. Today's flavor is interfacing with Wolfram, but we've also seen web searches, python coding, etc. That, I think, it the real superpower that comes out of this: you or I can answer a question by "doing a web search" or "query a database" or "use wolfram" or "develop a python program that finds the answer" However, an AI could do tasks like this just by "thinking" about it. Maybe it would be as natural as we find blinking.

That to me is the real breakthrough in stuff like Alpaca -- start with a mega-model and prompt it with something like: "After this paragraph, you are going to be speaking to a AI model similar to yourself but much more primitive. Its task will involve interfacing with English speakers, so converse with it only in that language. It has access to the same {X,Y,Z} APIs you have so any time it has trouble answering a question, prefer to give hints about how it could find the answer using those APIs rather than providing the answer directly yourself. Only give an answer directly if it repeatedly fails to be able to answer it by using an API. I've provided a large set of standardized tests used by humans at this URL -- start by asking it questions intended for a preschool-aged child. Each time it is able to answer new questions at a given level correctly 99% of the time increase the material's level until it is able to achieve that score on a test designed for a Computer Science PhD candidate"

How large would the "student" model have to be to succeed at this deep but narrower task? I think the answer right now "we have no idea". However if the model has the advantage that it can rely on external knowledge and tools from the start (and is rewarded by the "teacher" for doing just that) I bet it'll be a lot smaller than these mega-models. Sure, you wouldn't be able to disconnect the "student-AI" from its APIs and expect it to converse with you in Hungarian about the history of yacht design, but that might not be a capability it needs to have.

My personal hunch is that we're going to find these "AI-taught specialist AI, with API access" models will be a lot smaller than most people are expecting. That's the moment when things REALLY change: instead of pairing a human with a mega-model AI, if specialized models are cheap someone can say "spin up 100K expert-programmer AIs and have them supervized by 5K expert-manager AIs and have them build XYZ"

Or if you need it to work on an existing task you'd specialize further -- you'd go to your AI vendor and say "I'd like to license the weights for your expert-programmer model, but first have it read these 200 books I consider important to my problem domain and then show it every commit ever made by a human to my git repo and every design document I have"

That's a good instinct to have. However in this case the input is 0..63 and the output comfortably sits in a single byte. It can fit in just one cache line if you bother to align it.

The miss cost is, therefore, not really relevant: if this code is hotpath at all the cost of a single cache miss is amortized across millions of calls. Your lookup table will be in cache as surely as the code that reads it is.

X: yes, minix has X11R7 binaries: https://wiki.minix3.org/doku.php?id=usersguide:introductiont...

I don't think I've run an X server on it ever, but I can verify you can at least get it to pop up xterm's on a remote machine.

gcc: I don't think any recent gcc's work, but the latest minix (3.4.0rc6) does have clang-3.6 as /usr/bin/cc

Because then you can only use that instruction if you're sure the target CPU supports it... or else your program will crash.

Sequences like "or r1, r1, r1" are going to be harmless when run on a processor that is unaware of its special meaning, which is exactly what you want for a "hint" instruction.

stdlib is also where things go to get used, though.

Coding against a pile of external python modules is perfectly reasonable when you're building something "app scale" -- i.e. something that is going to be spread across many files and installed in a container or venv.

However, when writing something at "script scale" I just don't want to deal with all of that. I want to write something that I can deploy as a single file and not end up dealing with missing python dependencies every time. This means I'm using the old urllib and such more often than I'd like.

It's a shame that there doesn't seem to be much flow of functionality into stdlib at all any more. For my needs, I wish "requests" and "yaml" were in that set, although I'm sure other people have their own opinions on that.

Once we have AI-generated laws I'm sure this will be sorted out.