The operating system: should there be one? (2013) [pdf]

https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.715.9197&rep=rep1&type=pdf

by evanlh • 5 years ago

168 202 5 years ago

citeseerx.ist.psu.edu

Causality1• 5 years ago

Title needs the [PDF] tag.

UnlockedSecrets• 5 years ago

The PDF: Should there be one?

aitchnyu• 5 years ago

There is a HTML-bundle alternative to PDF in my utopia. User agents can render documents optimized for any screen instead of hardcoded two column A4 pages. Nouns and verbs are highlighted.

https://rewordify.com/index.php

p2t2p2• 5 years ago

On one had - yes please, screw OSes, gimme something like Smalltalk environment (in it's Pharo incarnation preferably).

On other hand - define OS. Isn't that Smalltalk environment an OS if it runs bare metal?

ModernMech2• 5 years ago

Yes, there is a very blurry line between operating systems, programming languages, and we can include databases as well. Some operating systems as you point out are PLs. Some PLs are implemented as databases. They’re all just really complex accounting systems by nature. Look at the implementation of a lot of OSes, PLs and DBs they are really just made up of tables keeping track of relationships all the way down.

yjftsjthsd-h• 5 years ago

Also VMs; the JVM is targeted by multiple languages as if it were an OS

infogulch4• 5 years ago

I would be soooo happy if OSes were just absorbed by databases. The file system is just a heirarchal key value store: genius! Let's make it untyped so you have to parse everything manually just to interpret it: not so great.

Yes I'm saying the Unix file system is a worse mongodb. Please at least give me a postgres or something.

hnlmorg• 5 years ago

Some operating systems as you point out are PLs

None that immediately spring to my mind. There are operating systems that incorporate programming languages as their default user interface (like UNIX that drops into a Borne shell and LISP machines) and there are computers that have programming languages baked into their firmware (like the 70s and 80s micro computers did with BASIC). But in neither instance is the language itself the operating system.

The micro computer instance is definitely nuanced though. Some do argue that Microsoft BASIC et al were an operating system. Some don't. I'm in the latter category because I think "BIOS" more closely describes the function of the software on those old micro computers.

travisjungroth2• 5 years ago

(Only read the abstract). I’m learning Emacs and it’s making me think that we missed on the correct abstraction level of the operating system. There are programs that are OS-like (Emacs, browsers, Electron, anything that takes plug-ins). Maybe operating systems could have been lower level, and we could have developed more/better options on the pseudoOS level on top. This seems somewhat inline with the Smalltalk vision.

pjmlp19• 5 years ago

Well, have a look at

http://www.softwarepreservation.org/projects/LISP/interlisp_...

Then follow up with "The Interlisp Programming Environment"

https://www.computer.org/csdl/magazine/co/1981/04/01667317/1...

"Graphical Programming in Interlisp-D"

https://www.youtube.com/watch?v=wlN0hHLZL8c

"Eric Bier Demonstrates Cedar"

https://www.youtube.com/watch?v=z_dt7NG38V4

Cedar was the very first graphical workstation for something like .NET and Java would later kind of try to become.

LargoLasskhyfv2• 5 years ago

While that seems to be like almost lost history, and a forgotten art, it is not!

Because

[1] https://interlisp.org/

[2] https://github.com/interlisp

travisjungroth2• 5 years ago

Thanks!

ducktective• 5 years ago

So...It Has Come To This...

https://xkcd.com/1022/

overgard4• 5 years ago

I can't help but think about what happened with graphics APIs - the older ones (DX9, OpenGL) held your hand more, and the new ones are super bare metal and not written for the faint of heart. I think most would agree this is better though; for an application developer you can simply use a much higher level library built on top, and for the library/engine developer you have a lot more power and control. The old APIs were in an awkward middle ground of not being particularly easy to use, but also not being nearly low level enough. Nobody was happy.

I think modern OS stacks are in a similar boat, at least Windows and macOS. True low level control is kind of a pain, but the OS libraries are not particularly pleasant or high level enough either. So largely things like Electron or Qt are acting as the defacto OS anyway to varying degrees of success but undeniable levels of resource waste. I wonder if modern languages and tooling and such are advanced enough that the OS should become more primitive and minimal. In a way it feels like we're moving there anyway with web tech becoming the defacto user space and toolkit. That's fine, but god if it wasnt all built on 50 layer of cruft our machines would be screaming fast.

Smalltalk, if you get a chance to use it, really does act like its own OS in so many ways. It's great as a programmers OS really, although it doesn't really hit the right notes for average use. Still though, if someone had the urge to build it I could easily see a smalltalk based userland built on like a linux kernel (with no gnu user space) being a totally viable thing.

tehsauce• 5 years ago

very interesting take, thanks for sharing

pjmlp19• 5 years ago

Nah, Android, macOS and Windows are doing just fine, it is the classical UNIX desktop that cannot get their act together.

The existing issues aren't technical, rather political, like the WinRT crusade that ended up bombing and now we have plenty of GUI toolkits to chose from on Windows.

Swift, Java, Kotlin and the .NET languages are more than high level enough.

otabdeveloper44• 5 years ago

the classical UNIX desktop

CDE? That hasn't existed for decades.

EllieEffingMae• 5 years ago

Could you expand on this more? I don't know a lot about GUI programming, but I'd be interested in learning what the current failings are within the desktop space.

raxxorrax3• 5 years ago

MS .NET burned me hard with WPF. After finally looking into it, it more or less was cancelled for greener pastures.

I don't think the newer Windows UI-Frameworks are loved that much and UWP is not really convincing for desktop applications. Just too much hassle for too little gain and the threat of vendor lock-in.

AussieWog9310• 5 years ago

I think modern OS stacks are in a similar boat, at least Windows and macOS.

I come from a background as a EEE mainly writing embedded code before somehow stumbling into frontend code (in win32, Cocoa and Qt) at my old job.

From my experience, OS-Level APIs definitely have more in common with Electron or Qt than they do "bare metal". To make the top left pixel red on a bare metal system or RTOS, you simply write `framebuffer[0] = 0x00ff0000`.

For win32 or Cocoa, you need to create a window, alter its flags to ensure it's borderless, change its size and posistion to 1x1 and [0,0] (or [0, screen_height] on the Mac), strip away the drop shadow, respond to regular paint events and pray like hell that nothing else draws over the top of it.

True low level control is kind of a pain

It depends on what you're trying to achieve. If you're trying to get something highly responsive that looks exactly like the UI spec, most libraries tend to make this job harder when the abstractions leak and you're forced to rely on bizarre hacks that make no sense and feel genuinely demoralising to implement.

Dear ImGui, which exposes far more of the low-level internals than any other API, is an absolute joy to build GUIs in and significantly easier than something which "holds your hand" at every step.

nyanpasu64• 5 years ago

Dear ImGui does not support Arabic (right-to-left, glyph shaping) and will not support Arabic (https://github.com/ocornut/imgui/issues/1343). I don't think it supports accessibility either.

As much as I struggle with Qt's APIs (aging janky Widgets, half-baked QML on the desktop, the item models are an impenetrable byzantine enterprisey API that's simultaneously too open-ended, not flexible enough since it imposes its add/remove/move schema on your model, and too buggy), building a GUI from the ground up is not going to give you internationalization or accessibility, and Qt is.

nine_k• 5 years ago

But most of the time you don't want to make individual pixels red. You want to blit bitmaps and fill rectangles, preferably without pegging the CPU. Here the simple memory-mapped framebuffer is less helpful, especially if you have specialized hardware to do such work.

Of course this does not apply if all you have is a screen connected via SPI to an MCU's GPIO pins. But you won't expect much graphical sophistication from such a device anyway.

lavabiopsy8• 5 years ago

IMO, Dear ImGui is great for its original purpose (debugging GUIs, simple demos), but once you start adding complexity, custom layouts and custom widgets on top of it the state tracking gets to be just as bad as retained mode if not worse. Unless there was some best practice that I missed, one has to take great pains to avoid re-running the layout multiple times per frame, whereas a retained mode GUI would be able to handle that easily.

Certhas• 5 years ago

Isn't this obvious? I mean if you need that kind of control of the screen then you shouldn't be working inside a WM by definition, and the Toolkits are built for building interfaces in the context of WMs. Right? I definitely don't want your random application to be able to draw on arbitrary parts of the screen and then prevent anyone else from drawing over it. Your use-case might sound simple at first glance, but it's truly odd if you think about how I interact with applications, and how I expect to be able to manage them.

simonh• 5 years ago

Making the top left pixel red just isn't something you should be doing in a GUI environment anyway, it doesn't make any sense. For something like that you should probably be dropping to a full-screen mode like most graphics heavy games anyway, in which case it's pretty easy on modern OSes.

The desktop GUI is just one subsystem in a modern OS, and you don't have to use it if you don't want to. If you do use it then there are tradeoffs, of course, but that's always true of anything you use.

dkersten2• 5 years ago

To make the top left pixel red on a bare metal system or RTOS, you simply write `framebuffer[0] = 0x00ff0000`.

Really? In my brief experience it was more like sending a bunch of commands over SPI/I2C/UART to some LCD module. Which isn't thaaaat different from how an OS driver communicates with a GPU, only the commands are far, far more complex, so using an OS-level API avoids having to do it by hand.

raxxorrax3• 5 years ago

Just start to implement the ~300-400 functions listed here and the underlying systems supporting them. How long could that take, a whole weekend?

https://github.com/torvalds/linux/blob/master/arch/x86/entry...

cmrdporcupine• 5 years ago

Arguably win32 GUI or Cocoa are not "OS level" APIs; yes some aspects of them are invoked through syscall type interfaces (esp win32) but not consistently and the operating system itself is entirely usable without them. Especially Cocoa. You can boot a Mac into a BSD shell without ever running Finder and the GUI.

Writing into a framebuffer is arguably as much of an abstraction these days as many of these toolkits. In the modern world of GPUs and display controllers a framebuffer itself is already N levels away from the "physical" display hardware. We're not in VGA land anymore. Some aspects of the OS or the OS's interaction with the display driver may present something like a linear framebuffer to you, but underneath that is a world of GPU textures and buffers and display controller abstractions.

When I think of OS-level APIs I think of ioctl, mmap, file descriptors and sockets. I guess that's my Linux bias showing?

kumarvvr• 5 years ago

True low level control is kind of a pain, but the OS libraries are not particularly pleasant or high level enough either

Wait. In that case, library developers or OSS projects can fill the gap. What am I missing here? QT comes to mind. A high level library for GUI programming. Also, the modern .NET stack.

I always thought the popularity of Electron is because GUI programming is insanely easy on it, and if you have apps that are heavily data dependent, you go with it because the alternatives (Native apps, WPF or Winform apps) do not provide any value addition.

klodolph6• 5 years ago

Electron is popular because it makes cross-platform desktop apps easier, or because developers have existing JavaScript code that they want to share between a desktop app and a web app.

I would say that if you just wanted to spit out a Windows application, and you already knew the technologies, WPF or Winforms would probably be better (despite the fact that WinUI is the new hot stuff).

bsdubernerd2• 5 years ago

The OS is all about resource sharing and allow a minimum common level of interoperability.

Smalltalk can provide a storage abstraction, but to interoperate with other things it needs to be able to save data in a way that can be loaded back into your Java program.

OSes can provide you with the desired abstraction level.

You could stop at the block layer, which many DBs can take advantage of, but that won't help your Java and Smalltalk program work together easily. A filesystem is a common abstraction which should provided by someone.

Your example about the graphics stack is no different, the problem is that various components of the system are not "shared" in the same way all the time. The actual low-level abstraction of the GPU being provided to you by the kernel has more complexity today than what it had years ago in order to allow virtualization and compartmentalization of graphics memory. This wasn't needed some years ago, as it was assumed X11 was the only program to ever going to access the framebuffer, and would allow any single program using GL to trash the entire GPU.

What has been simplified is the user-level facing code, so that you can more precisely control allocation of the resources. Akin to the page-level memory allocation support we always had. This was done to allow /some/ programs (3d engines) to push the hardware to the limits (I'm relatively sad the GL1.x is "deprecated" - I still consider it to be an excellent API for prototyping and what would be 90%+ of applications).

Consider audio instead. OSS always provided pretty much all the hardware features, but allow no resource sharing. Every single evolution on that front has been an increase in complexity to allow for resource sharing. And it's interesting because the effort went in and out of the kernel a couple of times. Describing the evolution of OSS/ALSA in linux/freebsd would be too long, however we now have very fat daemons sitting in front of the HW just for resource sharing.

account422• 5 years ago

Consider audio instead. OSS always provided pretty much all the hardware features, but allow no resource sharing. Every single evolution on that front has been an increase in complexity to allow for resource sharing.

None of the API changes were required for resource sharing though so that complexity could (and should) have been contained to the implementation and those applications that actually need new features.

PaulDavisThe1st5• 5 years ago

Consider audio instead. OSS always provided pretty much all the hardware features, but allow no resource sharing. Every single evolution on that front has been an increase in complexity to allow for resource sharing.

This is not true. Early evolution of ALSA included lots of work on handling high channel count, memory-mapped devices (and their opposite, a few devices around in about 2000 that require active CPU involvement in data transfer). At some point, more evolution was required to deal with asynchronous interfaces like the USB ones that are now ubiquitous. The Intel HDA "specification" created a huge amount of work to model the topologically indistinct hardware mixers that it allowed. It remains the case that ALSA provides no facilities for resource sharing that are used by more than a handful of obstinate folk, so there has been essentially no evolution within ALSA itself toward that goal (see next para).

And it's interesting because the effort went in and out of the kernel a couple of times.

Once OSS was dropped, the only resource sharing left on the kernel side in ALSA was the dmix layer. That continues to exist today, but has never worked reliably, which is part of the reason why PulseAudio came into existence.

Describing the evolution of OSS/ALSA in linux/freebsd would be too long, however we now have very fat daemons sitting in front of the HW just for resource sharing.

I object to JACK being called "a very fat daemon". It's lean and mean and fairly clean (even the version I didn't write). Perhaps you mean PulseAudio, but that does much, much more than just "resource sharing".

As a side-note, Apple did move resource sharing out of the kernel. At some point that was all kernel-side, and at some point, coreaudiod showed up as a user-space daemon that was eventually clearly doing the same thing.

oshiar53-0• 5 years ago

You might find the Exokernel approach interesting.

https://en.wikipedia.org/wiki/Exokernel

paulddraper• 5 years ago

So largely things like Electron or Qt are acting as the defacto OS anyway to varying degrees of success

That's the natural (and arguable optimal) state of things: lower levels and higher levels.

High-level developers use high-level things, low-level developers use low-level things.

Every once in awhile someone gets the bright idea to combine all the layers into one, but finds out that in the real world that just isn't practical.

But we get developers pulling apart and adding layers over and over, driven by a love for "simplicity." Moving forward, or in circles? Hard to say, but probably forward.

mjevans2• 5 years ago

Unix generally has the correct basis for the design.

I agree that a filesystem as we know it in unix-like systems is a set of objects (directories) and files.

Would it not be preferable to retain this simple interface and instead conceive of interaction between programs and threads not as a stream of distinct objects in memory, but rather as a data tree within each program's control which can be exposed as input or output for other programs to map against and write into / consume?

overgard4• 5 years ago

I think the problem with the unix model is that there isn't a rich enough array of types -- everything is just a stream of bytes and the closest thing you have to lists and records are strings with spaces and newlines. It was revolutionary in the 70s but compared to what you can do with something like powershell (which has its own issues) it's just not a rich or easy to use model anymore.

mjevans2• 5 years ago

Simplicity is a good baseline. In Unix systems, off the cuff scripting works really well for live human interaction.

Complexity, particularly NOT-self-documenting complexity, becomes a nightmare.

Imagine instead that each program controls a special area under /proc/(pid)/ which behave like tmpfs to that program: named interfaces are subdirectories, messages are files.

Reflection of supported interfaces becomes easy. It integrates with existing software, and it's self documenting for both software and users.

Complexity can be opt-in rather than the only way to operate.

AnthonyMouse3• 5 years ago

"The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem."

- Claude Shannon, A Mathematical Theory of Communication

Everything is inherently a stream of bytes. You can use programs that interpret them as typed data or newline or null-delimited arrays all you like.

strictfp• 5 years ago

Having tried many techniques for program-to-program interfacing, I must say that text still is the lingua franca, and I think it's the best "non-standard" to set.

pjmlp19• 5 years ago

Even in the 70's it wasn't that great, Powershell "novelty" was a standard way of working across Interlisp-D, Smalltalk, XDE/Mesa and Mesa/Cedar at Xerox.

ZealousIdeal• 5 years ago

I enjoyed Benno Rice's talk, "What UNIX Cost Us". I think he provides some examples of where the simple interface breaks down when compared to other systems (i.e. Windows, macOS) take on it.

lavabiopsy8• 5 years ago

Unix generally has the correct basis for the design.

I can't disagree with this more, Unix is broken in so many ways and its designs continuously cause me pain. I would not describe a modern Linux or BSD as "simple", maybe the core of it is, but to actually build a modern OS there are so many layers on top of it. And when you do that, the crumminess of the foundation really starts to stand out.

hvasilev• 5 years ago

This was correct in the 1980s, it is definitely not correct today.

sedev• 5 years ago

Title should probably have a [2013] marker.

vbezhenar2• 5 years ago

I would love more collaboration between operating systems. There should be more layers with standard interfaces.

For example BIOS used to be a driver layer back in the day, providing simple API to underlying OS. This concept should be expanded. I don't see no reason for every OS to rewrite those drivers again and again. There should be some standard for common device APIs, like disks, keyboards, mouses, GPU and so on. I should be able to buy an Nvidia GPU, download and install UEFI driver and it should provide Vulkan API for any underlying operating system, Windows, Linux, OpenBSD, ReactOS.

Another example is file systems. All major operating systems are very similar when it comes to file systems, at least from the user PoV. Yes, there are Windows ACLs, Linux permissions, but it should be possible to abstract those away enough. So Windows could install its latest Windows 11 NTFS driver into UEFI and Linux should be able to just use it to access NTFS safely, without any gigantic efforts to reverse-engineer and reimplement everything from scratch.

At one point I thought about virtualization being an answer to those demands. These days you can write completely virtualized operating system which needs to implement virtblk, virtfs, virtio-* drivers and run it on any host operating system with perfect driver support from manufacturer. GPU is not there, but I don't think that it's impossible to implement. Of course running a full-blown ad-ridden Windows 11 just as a hypervisor seems like not the most elegant solution.

Unification will not play nicely with new progressive features like BTRFS, though. That's something to think about as well.

klodolph6• 5 years ago

“BIOS” is really just a bunch of drivers in ROM chips somewhere.

Thing is, most devices you actually don’t need special drivers because they are so standardized. Mice and keyboards are USB HID. There are a handful of ways you typically interact with storage, like using SCSI. Surprisingly, a lot of storage devices are actually SCSI, in some form or another.

One of the big problems is that drivers are going to closely interact with other operating system services, like memory allocators or the scheduler. The interface has gotten a lot more complicated over the years. And the semantics that operating systems present to the user are sometimes different in significant ways. Back in the BIOS days, what you could do through the BIOS was rapidly falling behind what you could do with a proper driver.

To pick on the filesystem example, there are some fairly major differences between Windows and Linux/macOS/BSD filesystems APIs from the developer perspective. Windows has mandatory locks and files are identified by name. On Linux/macOS/BSD, there's more of a separation between a file and its name—files do not even need to have names at all, and locks are advisory. For this and other reasons, there are some boundaries between files you use in WSL and files you use in Windows.

I’m just very skeptical here that you could introduce another layer between devices and the OS, that would be thinner and more direct than using a hypervisor, more flexible than just using a standard protocol (like USB HID) which your OS already supports, and work with multiple operating systems.

pabs32• 5 years ago

Standard protocols between hardware and operating system drivers are the way to go, then you get to implement drivers for each hardware class (mouse, disk, keyboard etc) once and use all new hardware, instead of having the hardware vendor need to write new proprietary blobs for every new piece of hardware and every OS.

For example I read on LWN that hardware vendors are starting to look at Linux virtio for hardware agnostic protocols.

https://lwn.net/Articles/805235/

ksec• 5 years ago

For example I read on LWN that hardware vendors are starting to look at Linux virtio for hardware agnostic protocols.

It is interesting that abstraction always comes a full Circle.

KronisLV• 5 years ago

In an ideal world, we'd essentially have a number of software vendors that provide devices that conform to certain standards (we are mostly there in regards to peripherals like mice and keyboards, but not really in regards to Wi-Fi dongles/cards and other more complex devices).

Then, we'd have reference implementations for every single one of those, that would give examples of how to access this standardized functionality, possibly with abstractions over those. So on one end you'd have something like a wi-fi-networking-huawei library and on the other you'd have wi-fi-networking-generic library that could interface with the former and access its common functionality. Of course, as for how this could be written, and what the situation would be in regards to the C standard library or any other primitives for other sufficiently low level languages, i have no idea.

Then, you'd have a kernel module or a microkernel, that would contain the code to interface with the hardware and add all of the controls and permission management functionality, as other bits to integrate this solution with the OS. If one would go with the GNU Hurd approach, then integrating that within OS distros would suddenly get even more simple, except that Hurd seemed to fail at the actual implementation stage.

And finally, you'd have the OSes, which could implement code to interface with these modules, microkernels or add any number of necessary shims inbetween to get everything working together and provide the higher level OS functionality that's needed to use these devices within the OS context.

Of course, i'm probably oversimplifying some things, skipping a few steps here and there, but i'd chalk that up to the current approaches to using hardware in software being overcomplicated beyond need or belief, rather than me mostly being a business software developer, as opposed to systems software developer or hardware engineer. In short, there are abstractions in place already, i'm just not sure those are the right ones, but rather what was historically needed to get things working. So where's the big rewrite and learning from our past mistakes?

To me, it just feels like what we need is a set of meaningful abstractions and standardization, and not just in regards to making sure that there are security mechanisms in place for booting so the devices on which Windows runs can be limited to create more e-waste (snark) and motivate consumerism, but also to actually be able to use Wi-Fi in all hardware in 2021 in both Windows and Linux distros (not snark, literally had to compile custom drivers for my no name netbook).

mkup• 5 years ago

Oh no, I prefer BIOS/UEFI firmware to be as lean as possible, like, "boot my OS and get out of the way as quickly as possible" (because all software has bugs, larger software has more bugs, and fixing/updating firmware is a gross PITA, I mean inability to roll back and risk of bricking the hardware e.g. due to power loss). At least in Linux I can apply kernel patches individually, and in Windows I can replace a file here and there, but firmware is a monolith, so it is a "hit or miss" kind of thing. More complexity in firmware rapidly shifts this wholistic thing to "permanent miss" state.

chriswarbo• 5 years ago

Sounds like you'd enjoy OpenFirmware https://en.wikipedia.org/wiki/Open_Firmware

waynecochran• 5 years ago

I see the paper uses the term "postmodern" multiple times and references a paper "Notes on Postmodern Programming" which begins with a Manifesto. What does this term mean in this context? I don't like what I smell.

injidup• 5 years ago

Postmodern in terms of programming refers to that body of thought that posits a replacement of the rigid classist structures with a new functional hermeneutics based not simply on truth or falseness but on an interpretation of access defined as a trajectory of internal and external states across the pattern. Most new PDM (post modern development) is actuated in a bias free programming language called "Chomsky" which is a rigid left recursive grammar untainted by Boolean logic.

raxxorrax3• 5 years ago

Yes, I heard postmodernists say that everyone should stop boolean them.

TheOtherHobbes3• 5 years ago

This is now evolving into Green software, where everything is a tree.

jonathanstrange• 5 years ago

I know you're joking, but just to clarify for others: Nothing about Chomsky's work is postmodern. On the contrary, Chomsky is a well-known and fierce critic of postmodernism, both in his political work and in linguistics. He's been openly criticizing "postmodern" thinkers and authors for decades. See for instance [1]. Moreover, his approach in linguistics has frequently been criticized as being too "scientistic" because he re-oriented linguistics towards the natural sciences. After all, he's the one who brought formal, recursive grammar formalisms and grammar hierarchies to the field.

[1] https://www.youtube.com/watch?v=772WncdxCSw&t=1s

stephenrkell• 5 years ago

As far as this paper goes, the main idea is pluralism. Much software lore leans quite heavily on grand narratives about how things "should be done". The "postmodern" view is a reaction against that: in favour of multiple, coexisting, smaller stories. These do not have to be mutually consistent.

Of course there are lots of other overlaps between software and postmodernism in other culture, and that was the wider observation of the Noble & Biddle essay... true that it is a bit oblique. You might enjoy this a bit better: https://www.se-radio.net/2006/11/episode-38-interview-james-... -- it's now slightly dated, e.g. how "programming by Google / StackOverflow" was a much newer idea then than it is now.

perbu3• 5 years ago

I've had some experience with running applications without an operating system. First and foremost I like the simplicity. Instead of having state distributed between the kernel and the application all the state is in your application. No longer having this dance where an application and the kernel interacts there is only one place where state is kept.

This can lead to some spectacular results for things like TCP. Having the ability to serialize a TCP connection and sending it over the network means you're able to transfer a TCP connection from one host to another, even with SSL on top.

Now, this can be done with a POSIX kernel as well, as long as you move the networking from the kernel into the application. But if you're doing this you're eroding away the benefits of the kernel, basically turning your OS into a bootloader. Same thing with scheduling.

Another benefit is that, at least on compiled languages, code that isn't used isn't there. So the linker can strip away everything that is never called.

This still feels like utopia. The average development environment is pretty complex and ripping out and replacing all the layers below your language runtime is too much effort.

jiriro• 5 years ago

Is there a demo showing how the Plan 9 is similar to Smalltalk regarding the “meta” stuff (programmabilty, message passing, ..)?

jacquesm2• 5 years ago

Let's start off with defining what an Operating System really is because depending on where you draw that line the outcome will be substantially different. Ignoring mainframes and other large installations for the moment, strictly from a 'personal computer' point of view:

If you see the operating system as the task-schedular + IPC clearinghouse then the system is simple, well defined and it can be expected to perform in a deterministic and verifiably correct way (assuming working hardware).

If you define the operating system as 'everything and the kitchen sink' including media players, browsers and so on then there is no way to make any guarantees and determinism is going to be impossible to achieve.

Personally I prefer the leaner-and-meaner operating systems that have a larger ring of userland software built on top of them to create as small as possible a core on which the actual end-user applications are run.

Those systems tend to find good middle ground between reliability, speed and security because the amount of surface area exposed is going to be the minimum at every level.

Something like Windows is on one end of that scale, something like MacOS is somewhere in the middle. Linux can be stripped (or more accurately: it could be stripped) to something a lot slimmer than the default distros, and something like IRIX (no longer used in production as far as I know) was roughly where MacOS sat (custom hardware, custom OS tied closely to that hardware).

An ideal OS that limits itself to scheduling and IPC is currently not on the market for desktop usage, and the arrival of the mobile device market has blurred many of the lines that we thought were drawn pretty clearly in the past.

The 'lets move everything into the kernel' philosophy is as far as I'm concerned fatally broken and likely things will have to get a lot worse before they will finally get better, and it would not surprise me if everything that we do today will be discarded once that happens.

psyc• 5 years ago

Where do you rate the part of the OS that makes 100,000 types, brands, and models of devices all look about the same to our apps? I always saw hardware compatibility and abstraction as the greatest value proposition, not only for the user but for the PC industry as a whole.

jacquesm2• 5 years ago

Hardware abstraction layer. Can be done in userspace, with the exception of a timer. You may need to add some interrupt arbitration logic.

amelius• 5 years ago

Related question: do we need device drivers?

qayxc3• 5 years ago

That's a definite yes.

I do NOT want to go back to the times when we had to write and ship individual drivers for each graphics card, sound card and peripheral we wanted to support in our software.

Could you imagine having to select the physical port, IRQ and DMA channel of your sound device for each app that wants to output sound? Because that's what had to be done in DOS and screens like this https://flaterco.com/kb/audio/PCI/DS-XG_IRQ_error.png

were commonplace. Good enough if you had well-defined devices like SoundBlaster(Pro) or GUS that you could auto-detect without crashing the PC, but a living nightmare with clones that didn't quite behave the same way.

So unless you only plan to support a single set of well-defined hardware, device drivers are a must.

OrvalWintermute2• 5 years ago

I do NOT want to go back to the times when we had to write and ship individual drivers for each graphics card, sound card and peripheral we wanted to support in our software.

This 1000x - I remember this time and it was quite painful

Could you imagine having to select the physical port, IRQ and DMA channel of your sound device for each app that wants to output sound? Because that's what had to be done in DOS and screens like this https://flaterco.com/kb/audio/PCI/DS-XG_IRQ_error.png were commonplace. Good enough if you had well-defined devices like SoundBlaster(Pro) or GUS that you could auto-detect without crashing the PC, but a living nightmare with clones that didn't quite behave the same way.

I also remember the application working one second, unrelated hardware change (1), and back to the drawing board all over again

Koshkin• 5 years ago

In the (post)modern world, every device should have its driver built-in (along with a CPU to run it on). The communication protocol (the command language) should be standardized for each given class of devices. The protocol should support the ability to upload custom “shaders,” whenever this makes sense, to tweak or extend the default behavior.

yencabulator• 5 years ago

In the (post)modern world, every device should have its driver built-in (along with a CPU to run it on). The communication protocol (the command language) should be standardized for each given class of devices.

We are currently in this world. NVMe, USB, etc. NVMe for example is "just" DMA'd ringbuffers of requests and responses.

One challenge is that vendors tend to want to have "special features" that go beyond the standard command set. SCSI and SMART are examples of that pain.

The protocol should support the ability to upload custom “shaders,” whenever this makes sense, to tweak or extend the default behavior.

This part is crazy complex and not very likely to happen. "Shaders" in the context of GPUs can work because it's a very large & expensive component and very vendor-specific already. That approach won't work for a chip that costs $2, with the additional burden of needing to have the same programming interface for competing vendors.

unixhero2• 5 years ago

When was this written? I cannot find any meta information in the paper itself.

perbu3• 5 years ago

(bottom left corner)

unixhero2• 5 years ago

Ah there it was, thanks!

arnaudsm• 5 years ago

Pardon the biological analogy, but diversity has 2 main benefits : resistance to epidemics (malware) and faster evolution (innovation).

I don't want to converge towards a single OS, but welcome standards and bridges between them.

kaba03• 5 years ago

Continuing on your analogy, I would argue we have less diversity due to the all-encompassing three major OSes.

Similarly to how you don’t really get competitors to eucaryotes and prokaryotes, the initial investment (device drivers) to create an OS (organism) that is actually usable is so high that one basically can’t overcome it when others are available.

qPM9l3XJrF• 5 years ago

"an operating system is a collection of things that don’t fit inside a language; there shouldn’t be one"

I've wondered the same thing about databases. Just like an OS, a database has to manage data on disk and figure out when to cache it. The files & folders metaphor is arguably a relic of pre-computer times--instead of searching through folders, it is easier to search and sort based on particular attributes, something that databases are well-suited to. I think it'd be interesting to start with a database and try to build an OS around it, see how far you could go.

wsc981• 5 years ago

But humans generally like to organise things hierarchically and I would think database are not well suited for this purpose.

That is, unless there exists some kind of tree structured database that I don't know about ...

Many years ago Microsoft tried to implement some kind of database oriented file system (it was a big dream of Bill Gates IIRC and probably meant for Longhorn), but this project never became part of an official Windows release AFAIK. It'd be interesting to know why Microsoft abandoned this initiative.

qayxc3• 5 years ago

It'd be interesting to know why Microsoft abandoned this initiative.

There were technical issues. They simply never managed to get the system performant enough to be use as a FS replacement. There also were tons of issues with the concept itself, like proper tooling for visualising the stored objects to the user and a compatibility layer for legacy software.

It seems as if parts of it was merged into MS SQL server products [0].

[0] https://docs.microsoft.com/en-us/archive/blogs/winfs/

dgb233• 5 years ago

humans generally like to organise things hierarchically

People like doing that until they have sufficiently essential complexity that it breaks down, then they hate it with passion.

AnIdiotOnTheNet• 5 years ago

That is, unless there exists some kind of tree structured database that I don't know about ...

Believe it or not, hierarchical databases were actually quite popular once upon a time, mostly in IBM mainframe land as I understand it. Today the Windows registry is a hierarchical database in common use.

cpach• 5 years ago

Didn’t BeOS have some features like that?

t435622• 5 years ago

https://www.haiku-os.org/legacy-docs/practical-file-system-d...

  The Be operating system (BeOS) uses BFS as its native file      
  system. BFS is a modern 64-bit journaled file system. BFS 
  also supports extended file attributes (name/value pairs) 
  and can index the extended attributes, which allows it to 
  offer a query interface for locating files in addition to 
  the normal name-based hierarchical interface. The 
  attribute, indexing, and query features of BFS set it apart 
  from other file systems and make it an interesting example 
  to discuss.

So yes it has been done. Personally I like the AS400 idea most - an object store where you don't even have to really "think" about the difference between memory and storage - you just have objects that you can cause to "persist" when you "commit" them and anything they are linked to by pointers will also be persisted.

pjmlp19• 5 years ago

Oracle does that,

https://docs.oracle.com/en/database/oracle/oracle-database/2...

Then use APEX for Web applications, that make use of stored procedures,

https://apex.oracle.com/en/

Other enterprise RDMS have similar offerings.

dgb233• 5 years ago

getting a 404 on the first link

slx262• 5 years ago

This is basically the problem of organizing knowledge and setting up rules to easily interface with it through computational systems. There was an interesting article that discussed the topic some time ago, but can't find it right now.

Neither hierarchical organization nor tags nor databases work. You need a mix of everything, and you need to make it nice enough for the user to be able to select the best option at the right time. Think about how you think and access memories. There are a ton of ways you do it, not only hierarchically or "table-based".

This could also be likened to the problem of categorization that meta-rationalists like to talk about so much. Basically, that (at least in human terms) no categorization works under all possible contexts.

We are not gods, so it should be obvious that there's no possible organization of knowledge that makes it all perfectly available. It can't be frictionless, instantaneous and complete at the same time. Then, it should be obvious that if we want to make access to information as effective as possible, multiple methods need to be combined.

In fact, a lot of the technology that has been most "revolutionary", has been technology that improved the way we access certain types of information. Graphical interfaces, databases, spreadsheets, hyperlinks, google search, etc. We have also used calendars, lists, schedules, diagrams, button panels, etc., long before computers were a reality.

There are still a few steps left. As we attempt to tackle even more and more complex systems, we are pushed to look for new ways to manage all that.

selestify• 5 years ago

I've never heard of that meta-rational problem before. What is it called? I'd like to read up on potential solutions to it

jakeinspace• 5 years ago

I can imagine a hybrid file system where a hierarchical structure is used for several levels, before bottoming out in a directory of databases. Ideally this would be accessible/searchable by DB index from all levels of the directory hierarchy. I'm sure something like this has been implemented of which I am ignorant.

spijdar• 5 years ago

You might be interested to read about IBM's OS/400, aka IBM i. It's... sort of like that. Essentially an OS built around DB/2.

ilaksh• 5 years ago

OSes do have shared abstractions such as file systems and process input and output. I think the trick is to make the common abstractions richer in a way that enough people agree on and are willing to adopt. Most of the shared OS features are basically grandfathered in. So a big part of this is that this stuff is not really a rational choice usually. It's just the way it's always been.

So upgrading the capabilities and creating new common abstractions in an OS is not that hard. But almost impossible to get the majority to go along with it.

Browsers demonstrate an additive way of getting a lot of that. In that we now have essentially two highly compatible widely deployed VMs -- Firefox and Chrome. Which can be programmed in a number of languages that compile to JavaScript or web assembly and share a rich API.

Web assembly modules seem relevant.

https://github.com/WebAssembly/proposal-type-imports/blob/ma...

gatestone• 5 years ago

“Programs = data structures + algorithms”

If you have more than one program, they need shared data and access procedures, like the “user” and “the hardware”. Who manages them: The OS?

Maybe you want to share a bit more like some common higher level functionality. That becomes part of the OS, de jure (resident kernel) or at least de facto (loadable kernel modules).

Kalanos• 5 years ago

i just want protocols between workflows of serverless services

mikewarot3• 5 years ago

If you have one job to do, and it doesn't involve other computers, you don't need an OS. MS-DOS or equivalent program loader/file system API will work just fine. Windows, MacOS, and Linux work fine in a stand alone environment (unless an APT {advanced persistent threat, like the NSA} is after you)

On the other hand, if you're even thinking about touching the internet, or any other computer, you need to have every single task limited ahead of time to a specified set of resources, like mainframes do. Doing anything less is foolish, and I certainly wouldn't build a civilization on such folly.

Yes, in an era of mobile code, persistent internet, you need an operating system that doesn't give any default permissions to any executing task, ever. It must also safely and securely multiplex access to the hardware resources, all of the hardware resources, in all of the subsystems in the SoC, etc.

There are no Operating Systems, as of yet. I weep for the future.

lloydatkinson• 5 years ago

Isn't what you just described Fuschia?

mikewarot3• 5 years ago

Isn't what you just described Fuschia?

It would be, if I could get it to run my PC. Same with Genode. Of course at this point nobody expects HURD to come back to life.

marcus_holmes2• 5 years ago

I've been playing with Unikernels for Go server deployment - so far so good. I'm not sure if this counts as "no OS" - as I understand it (and I'm not sure I do) the Unikernel core provides the calls that the Go runtime is expecting.

But the central idea that the server system can only run one program makes a lot of sense to me. Obviously it won't work for a desktop, but then the whole idea that desktops and servers run the same OS doesn't make much sense to me.

OrvalWintermute2• 5 years ago

I've been expecting the unikernel approach to catch on more too - although not the "no OS" it can be pretty stripped down compared to a regular OS. Now that unikernels are starting to gain more traction in the DOD I expect this will continue.

perbu3• 5 years ago

I've worked with Unikernels for a couple of years. They are great, but they are very, very pure and this can be limiting.

Say I have an web API written in Go. For some reason I need to do some openssl stuff before I start up my main program. In a container I can just wrap my program in a little shell script which generates/signs a cert before starting the API.

Such a simple task. It is gonna be a lot harder with a Go Unikernel as I could have to bake this functionality into the application itself.

This Swiss Army knife that is Unix often has it's uses.

pjmlp19• 5 years ago

Somehow I would state that serverless and PaaS for managed languages are where unikernels are going.

In those cases 99% of the time you really don't care what runs underneath the language runtime.

PaulHoule2• 5 years ago

IBM never had a plan for a mainframe operating system that made sense -- given the idea that the "360" was supposed to encompass the whole circle of applications.

What happened was some academics figured out it was possible to make virtual machines on the 370 and that you could run a few different operating systems at the same time. If you were doing software dev on the 370, for instance, you would start up a VM running VM/CMS which was a single-user OS a lot like CP/M or MS-DOS, work there, possibly using services offered by another OS in another VM by some kind of message passing.

This turned out to be the "master plan" in the long term, since they could have a few different mainframe OS, even some applications that run on "bare metal", but have it all run on one machine or one cluster of machines and be managed under the same pane of glass.

mikewarot3• 5 years ago

The reason it worked out was that each virtual machine was given a set of capabilities, and the hypervisor restricted access for that machine to only those resources.

Eventually, this lesson will spread. I think about 5 more years of chaos until people finally catch on.

kaba03• 5 years ago

We sort of have that with cloud computing, don’t we?

PaulHoule2• 5 years ago

Yep.

Also there are many x86 and other systems where people use VMWare or something like that for "server consolidation" without going all the way out to the cloud.

The difference today though is that the guest operating system today is a "full-service" operating system such as Linux or Windows. In many of these cases the partitioning of the service could be accomplished through user id's, processes, sessions, containers, and similar facilities offered by the guest OS.

schiem• 5 years ago

Maybe this is a hot take, but it seems like browsers have essentially become what the operating system should have been, and it's the reason that everyone crams things into web apps / Electron instead of building native applications.

Browsers have a set of APIs that are decided on by a standards committee, and you have to implement those standards if you want your browser to be compliant. As an application developer, all I need to do is check a page to see how widely a standard has been adopted and then use it - I can be sure that my code can ship to (nearly) every device and behave uniformly, with minimal tweaking.

If the operating system had gone that route, we wouldn't have had to build more layers on top of it to abstract away the differences.

reidjs• 5 years ago

From this perspective it makes sense why Chromebooks are doing so well, especially among general computer users.

pjmlp19• 5 years ago

Browsers are just yet another take on remote access to timesharing systems.

kazinator• 5 years ago

As soon as two or more separate applications share some common body of code that helps them deal with The Computer, you have an operating system. It doesn't matter whether that common body is a table of software traps that dispatch machine language routines, or whether it is Smalltalk objects.

Some game on a MS-DOS floppy disk that boots itself from the boot sector and uses BIOS calls has an operating system.

The set of requirements which describe the host environment (the "virtual machine" in the classic sense) that forms the backdrop against with an application is developed is the operating system.

The only way you don't have an operating system is when the application itself does everything it needs with regard to the outside world by itself poking at I/O registers, and handling every interrupt and so on. Anything independently organized for that purpose, given something resembling an API, is an operating system.

There is, or used to be, a sensible version of "there should not be an operating system" I first heard some decades ago. Namely this: "there should not have be an operating system with a hardware-protected user/kernel boundary". The idea is that there doesn't have to be kernel which the hardware conceals using privilege levels and fences. The idea is supported by the notion that all security can be handled at the software level. The system, as such, is somehow verified to be correct. The only way to execute code is via compiling it via the system's trusted compiler, which generates only safe instruction sequences. Those sequences can call functions in the system, and those functions sandbox what can be done. All notion of trust and privilege is software-level without any privileged instructions having to fall down a trap into a privileged kernel which meticulously validates arguments.

(All that rhetoric needs to be somehow adjusted for 2021, which finds us jaded by knowledge of side channel attacks, cache attacks and whatnot.)

fouc• 5 years ago

I think that there should be a humane operating system that operates for the human.

Operating systems shouldn't get too hung up on supporting applications or browsers, instead they should focus more on serving humans.

Common operations that humans need should be standardized across the operating system.

The operating system: should there be one? (2013) [pdf]

Related Stories