Octox: Unix-like OS in Rust inspired by xv6-riscv

by o8vm • 3 years ago

190 119 3 years ago

Santosh83• 3 years ago

Why does it seem like 80 to 90% of hobby OS projects that are started are "Unix-like?" Don't we already have a huge variety of Unix-like OSes out there? Why not explore new models?

packetlost2• 3 years ago

We need more Plan 9-likes!

But more seriously, the Unix/POSIX-like OS is a pretty good model, but IMO we could do better, especially in the microkernel front.

o8vm6• 3 years ago

I would like to implement a plan9-like OS someday!

bakul4• 3 years ago

I'd love to see an OS with a Unix API but plan9 like kernel! I will never get around to writing one myself though!

boricj2• 3 years ago

There are a lot of things wrong with plain standard POSIX:

- it's hopelessly out of date and incomplete w.r.t. modern expectations

- fork() is very problematic for multiple reasons, especially in a modern environment (https://www.microsoft.com/en-us/research/uploads/prod/2019/0...)

- process management functions take PIDs and not capabilities, which can cause race conditions

- POSIX hasn't standardized anything better than poll(), yes it works fine in a hobby context but it's not 1987 anymore (and don't get me started on select(): https://github.com/SerenityOS/serenity/pull/11229)

- signals are a mess

- Unix processes have a huge amount of ambient authority, which is problematic when trying to isolate them (chroot(), https://fuchsia.dev/fuchsia-src/concepts/filesystems/dotdot, ...)

- the C POSIX library has a lot of cruft while also missing stuff what programmers actually need (for example, POSIX took nearly 6 years to standardize strlcat()/strlcpy(), the process itself starting 17 years after OpenBSD introduced those functions: https://www.austingroupbugs.net/view.php?id=986)

- ...

Granted, modern production-grade Unix-like operating systems have extensions to deal with most of these issues (posix_spawn, kqueue, pidfd_open...), but they are often non-standard and can be quite janky at times (dnotify, SIGIO...). It also doesn't fix the huge amount of code out there using the legacy facilities like it's still the 1980s.

There are other models out there (Plan 9, Windows, Fuchsia...), but what we really need is to stop putting Unix/POSIX on a pedestal like some holy scripture that shall not be questioned. It's the pinnacle of 1970s operating system designs and it has fossilized so much it's actively turning into oil.

Or at the very least, please stop teaching the next generation that fork() is the greatest thing ever. It's a 50 year old hack kept alive through gratuitous amounts of copy-on-write that should've been scrapped the day Unix was ported to computers with virtual memory and paging.

pkphilip• 3 years ago

Redox OS implemented in Rust aims to be Plan-9-like:

https://www.redox-os.org/faq/#how-redox-is-inspired-by-other...

crickey• 3 years ago

What is even Unix-like ? I usually associate it with POSIX but maybe thats naive

OJFord2• 3 years ago

The general design of users, processes, files, etc.

It's obviously not that well-defined since it's implicitly 'not Unix', so can diverge. Just anything that's subjectively similar to that OS design.

nextaccountic• 3 years ago

What is an OS good for if not to run programs? You either port existing programs to your API (lots of work) or design the OS with an existing API in mind. Or forget about existing programs and write entirely new programs

"Unix-like" describes the second approach, you provide a POSIX API but can have different internals

Now, maybe you wanted to have two APIs: one native API for the apps you write, and a translator from another, more popular API to your native API. Sounds like a lot of work when almost all software that will ever run on your OS will the translator

vilunov• 3 years ago

Often the user-space API limits the design of internals and available features. I for one would like to see the classical hierarchical filesystem gone, but POSIX (and UNIX) at its core is about the filesystem.

If you need to run the existing programs, feel free to develop a translation layer, at the same time allowing a new type of OS paradigm to emerge.

jijji• 3 years ago

its better than being OS400-like, isnt it?

kjs32• 3 years ago

OS/400 is actually really interesting and very well thought out, particularly as an example of "Not Unix". I wouldn't want to make my living there, but lots of people do.

kjs32• 3 years ago

Why does it seem like 80 to 90% of hobby OS projects that are announced here as "Unix-like" invariably get someone who replies with "that thing you're doing as a hobby, for fun...you're doing it wrong and I don't approve". Zero content, zero insight posts about someones else's toy aren't useful. You want a 'new model', start coding.

sneed_chucker• 3 years ago

Because we effectively live in a Unix monoculture world, in terms of operating systems that you can actually use and study the inner workings of.

It was a legal anomaly that resulted in early Unix (aka Research Unix) being distributed with its source code for free to universities, which was enough to get the ball rolling on stuff like BSD and Lion's annotated V6 source code, that by the time AT&T decided that closed source commercial Unix was the game it wanted to play, the cat was already out of the bag.

By the time the free software and open source movements had gained some ground, enough people had studied or worked on some kind of Unix kernel and Userland source code that projects like Linux, Minix, and Free/Net/Open BSD were feasible. The fact that Linux running on x86 subsequently ate the world was probably something few people saw coming.

The other lineages of operating systems, e.g. Windows NT, OpenVMS, IBM's various offerings, either never have their source released or only have their source released long after they're obsolete.

linguae2• 3 years ago

Through various market forces, Unix (and its descendants/clones) and Windows killed most of the rest of the OS ecosystem over 20 years ago. There are generations of software engineers and computer scientists who’ve never studied operating systems that weren’t Unix- or Windows-based. Most leading OS textbooks (Tanenbaum’s books, the dinosaur book, Three Easy Pieces) have a Unix slant. Even the systems software research community is heavily Unix-centric; I say that as someone who used to be immersed in the research storage systems community. The only non-Unix or Windows operating systems many practitioners and even researchers may have used in their lives are MS-DOS and the classic Mac OS, and there’s a growing number of people who weren’t even born yet by the time these systems fell out of common use.

However, the history of computing contains examples of other operating systems that didn’t survive the marketplace but have interesting lessons that can apply to improving today’s operating systems. The Unix Hater’s Handbook is a nice example of alternative worlds of computing that were still alive in the 1980s and early 1990s. VMS, IBM mainframe systems, Smalltalk, Symbolics Genera, Xerox Mesa and Cedar, Xerox Interlisp-D, and the Apple Newton were all real-world systems that demonstrate alternatives to the Unix and Windows ways of thinking. Project Oberon is an entire system developed by Wirth (of Pascal fame) whose design goal is to build a complete OS and development environment that is small enough to be understood for pedagogical purposes, similar to MINIX but without any Unix compatibility. Reading the history of failed Apple projects such as the planned Lisp OS for the Newton and the ill-fated Pink/Taligent project are also instructive. Microsoft Research did a lot of interesting research in the 2000s on operating systems implemented in memory-safe languages, notably Singularity and Midori.

From learning about these past projects, we can then imagine future directions for OS design.

lproven• 3 years ago

Beautifully put. I couldn't have said it better myself.

amedvednikov• 3 years ago

Project Oberon is amazing. Shame it isn't more popular.

mananaysiempre• 3 years ago

How would one go about exploring “other” real-world systems? I’ve read some old OS textbooks, poked at Symbian and OS/2 books some, and have texts on Oberon and Symbolics in my queue, but the docs for RSX-11 and VMS seem to bury me in operational minutiae without really explaining the design choices, and the Multics docs look like a huge pile of research notes, which is going a bit too far in the other direction. The current bytecode on IBM i is apparentily outright NDA’d, and the RPG docs are eager to presume I know how to operate the original punch-card tabulators. Any pointers?

flyinghamster2• 3 years ago

Another key issue: the concept of open source barely existed. Sure, you got a copy of the source code used to SYSGEN your RT-11 or RSTS/E system, but it had comments removed and was not for redistribution. In the end, the closest in feel to the old DEC world today would be the traditional Windows command line (not PowerShell).

The rise of GNU and the general open source movement was a reaction to the rug-pull when access to Unix source was restricted, and that gave us Linux (MINIX wasn't initially open source, but could be put into a state that made Linux possible).

The Unix paradigm stuck with us, IMO, because Unix (and later Linux and BSDs) have been ported to so many vastly different architectures. So many other operating systems have never moved beyond (and often died with) their initial platforms.

kazinator• 3 years ago

So far, historically, no OS has been successful unless (in addition to whatever whiz bang funtionality it had to offer) it provided a process abstraction in which the application thinks it has the real hardware machine to itself. This is the big thing in Unix and others: not how the file system is, or that there are pipes and whatnot: but that you get an address space in which you run machine code for the Intel, SPARC, PPC or whatever. The machine isn't hidden. The OS is in the background; you tap into it via some privileged instruction or trap.

It was not possible to develop just an OS which does this; it needs hardware support. There has to be virtual memory if there is a requirement to juggle multiple such applications each of which thinks it has a whole machine all to itself. There has to be protection: user/supervisor separation, and that protection cannot depend on a trusted compiler, because the application can execute arbitrary machine code produced by a compiler of the developer's choosing (or even by hand). It has to be hardware.

Okay, so successful OSs so far have been ones that provide decent multiprogramming, by following the hardware's model for virtual memory and privilege separation. What about the rest of the shape of the OS; the environment? Like files being just arrays of bytes?

Files being arrays of bytes is a form of democracy. Any sort of structured storage introduced on one OS will look indistinguishable from a proprietary file format which silos your data.

Files are exchanged between operating systems in bytes. No matter what you do, that's not going to go away.

Even some modern enhancements and complications in file representation cause interoperability headaches for the users, when they need to take their files to a different system.

The rich object features are provided by applications. For instance, an object-oriented database can just store its stuff in a block of bytes. Same with a graphical office suite, or a Lisp system saving its image.

Nobody wants an opinionated operating system in which files are fancy trees of objects done in One Way that everyone has to follow who codes for that operating system.

guerrilla3• 3 years ago

If it ain't broke, don't fix it.

All kinds of new OS ideas can be implemented on UNIX like Mach[1], FLASK[2] and 9P[3] while internally a UNIX-like system doesn't need to be anything like a UNIX[4]... So who cares? What are you worried about losing? What can't be implemented on a UNIX-like system?

1. see MacOS

2. see SELinux

3. see v9fs

4. see Windows and BeOS which both have POSIX APIs

LoganDark• 3 years ago

Didn't Windows drop the POSIX APIs and then eventually introduce WSL instead?

josephg7• 3 years ago

The problem is that anything you build on top of Unix will always be a second class citizen in the Unix world.

For example, suppose you want a database-like filesystem. Either you implement it in the kernel, and now your special apps barely work on anyone’s computers. Or you implement it in userspace - preferably as a library. And now your apps can run anywhere without special kernel features but the terminal, and all the other applications on the computer can’t / won’t understand your new abstraction. And you’ll be fighting an uphill battle to get anyone to care about your new thing, let alone integrate it.

It’s like saying - why rust? Why not just add a borrow checker to C? Why didn’t C# just add a garbage collector to C++? Sometimes starting fresh and establishing a clear, clean environment with different norms is the most effective way to make something new. You don’t have to fight as many battles. You can remove obsolete things. You don’t have to fight with the platform conventions, or fight the old guard who like things as they are.

It’s a shame with operating systems that modern device drivers are so complicated. Hobbyist operating systems seem inordinately difficult to make these days as a result, and that’s a pity. There’s all sorts of good ideas out there that I’d love to see explored.

flykespice• 3 years ago

If it ain't broke, don't fix it.

I always hated this caveman mentality.

So should we hinder progress/try out new ideas because things as is aren't broke?

im_down_w_otp2• 3 years ago

We created a funky little OS on top of seL4 & Rust that's most certainly not Unix-like, and is more akin to an RTOS-like approach to building & bundling software. More for purpose-built appliances than a general purpose OS.

cmrdporcupine2• 3 years ago

Tell us more. Who is we?

chaxor• 3 years ago

I think the main selling point for Unix-like OS, but in Rust, is focused on Rust. It's to ensure that the memory related errors are less likely, and hopefully with enough work, the system can be essentially what we have today, but with less CVEs.

It's honestly a decent goal and I'm in support of it.

I know that there will inevitably be many now that come to state the obvious "well it doesn't guarantee safety" and "there are other reasons for CVE", etc. Nonetheless, it's not a bad idea.

zvmaz• 3 years ago

There's XINU (Xinu's Not Unix). There's a book that walks you through the complete implementation of the OS in C [1].

[1] https://xinu.cs.purdue.edu/

bsder• 3 years ago

Because the only large, available, free ecosystem of software (both applications and hardware drivers) is completely built around the Unix abstractions.

If you don't want to do Unix, you have to reduplicate ALL of it. And that's like trying to boil the ocean.

smasher164• 3 years ago

It’s hard enough to learn osdev. If you have a concrete design to fall back on, you can focus on implementation.

Coming up with an original design for OSes is like a 3rd project thing.

lmm• 3 years ago

Unix is a lot more amenable than most older OSes to being implemented by a disparate group of people with limited communication, hence why GNU was originally set up as a reimplementation of unix.

jacquesm3• 3 years ago

Because if you manage to get to some level of POSIX compatibility you can leverage that into having a whole toolchain and lots of other goodies up and running in a relatively short time. This limits the amount of effort required to get to 'first base', a self hosting environment.

OJFord2• 3 years ago

Is it perhaps interoperability?

If you're not that serious about it, it's a way to get a bit more for free?

Contrast Redux, which is quite a serious (not in the 'we compete with Windows' sense, but still) project I gather, which... I don't know how they describe it, but it's sort of Unix-ish, Unix-rethought? I've never actually played with it, but loved the idea of 'everything is a url' (not file) when I heard it described & explained on The Bike Shed podcast I think.

inkyoto• 3 years ago

Because of the availability of the source code under a permissive licence that UNIX has been distributed under for a long time. Other operating systems source code still remains unavailable, sometimes decades after the hardware they used to run on had disappeared.

The second reason is the simplicity of abstractions and the ease of their implementation. In the original DEC documentation on RSX-11M (the predecessor of VAX VMS), for example, there is a whole chapter describing how to create and fill in a file descriptor block (a complex record data structure) required just to open a file where the user has to decide beforehand where to locate the file, whether they want to access the data [blocks] randomly or sequentially, whether the file is being opened merely for updating the data but not extending the file size or for updating the data and extending the file, whether allow or not other processes to access the same file whilst it is open, the number of in-memory buffers for the kernel to allocate for file access etc etc. Many complex decisions have to be made before a file can be opened. In UNIX, on the other hand, it is a mere «int fd = open ("myfile", O_RDONLY);», the end.

Granted, not every OS has had such complexities (the opposite is also true, tho). Yet, the simplified (one can argue that it has been oversimplified) UNIX abstractions have been influential for a reason.

Brian_K_White• 3 years ago

At both extreme opposite ends of the scale/funding/people spectrum we already have TempleOS and Fuscia, and probably countless in between. You tell me why they aren't going anywhere even though any properties you might say about one, the other has the opposite quality and is also going nowhere.

Maybe "unix-like" is really just a principle that has no expiration date, like "murder is wrong".

Depending on how slavishly you define "unix-like", for instance, I would not say that the principle philosophy dictates there shall always be a command named "rm" that takes these options and does this task a la posix specs.

But for today and certainly any forseeable time, it's perfectly useful to "merely" reimplement posix.

hulitu• 3 years ago

It is only marketing. It does not seem to implement anything from POSIX or SUS.

pjmlp3• 3 years ago

Because it is less effort to copy already existing stacks, than be creative in trailing not yet discovered paths.

mysterydip• 3 years ago

What does "written in safe Rust as much as possible" mean? Are there functions with no equivalent in rust?

OtomotO• 3 years ago

Interfacing with hardware means you have to drop to "unsafe" rust at some (few) points.

"Unsafe" rust isn't named good, because it's still safer than e.g. C.

Some invariants you cannot break in rust, no matter if "safe" or "unsafe"

xeonmc• 3 years ago

Would it be more aptly named “risky rust”?

tialaramex• 3 years ago

it's still safer than e.g. C.

This is arguable and I think overall it's actually harder to correctly write unsafe Rust, even if sometimes maybe in some sense safer than C when you screw up.

In Rust everything has to obey Rust's semantic constraints. For safe Rust that's fine because the language itself promises you're obeying. You can't introduce anything which would be a problem, so you needn't even care what those problems are.

But in unsafe Rust you are responsible for the same guarantees that safe Rust gave everybody. And the rules you're responsible for obeying are truly difficult so that you may not properly understand them. If you screw up, that's instantly Undefined Behaviour.

Let's take a fairly old but brutal real example from Rust's standard library. core::mem::uninitialized<T>(). This function is labelled deprecated (as well as unsafe) in your Rust, but once upon a time it was the usual way to make some uninitialized buffer in which to construct something.

But it was actually UB almost always†. Because what it says is, OK, I know I didn't initialize a T, but trust me, I'll sort that out later, lets say this is a T anyway. And for a time people persuaded themselves that this is OK for at least some types. After all, if T was u8 (a byte) then who cares what its value is, any value is valid, isn't it? Well, yes, but "uninitialized" isn't a value, it's a 257th possible state, the compiler knows we didn't initialize this, and therefore all optimisations are valid even if they wouldn't be valid for any possible initialized state of the memory - we didn't initialize it so we're not entitled to assume it had any of those values.

In C you will get away with this but in Rust you've created Undefined Behaviour, which is not OK. Today you would use the MaybeUninit<T> type so that you can explicitly initialize it (once you have something to initialize it with) and then MaybeUninit::assume_init() to get your T instead now that it's initialized, and (if you did it correctly) that is safe.

† If T is a Zero Size Type then this function isn't dangerous, because it makes nothing and then says this nothing is actually a T, and the compiler says well, thanks for telling me, I don't really care but whatever. No UB. Likely this only happens in generic code, but it's safe.

dathinab2• 3 years ago

Some invariants you cannot break in rust, no matter if "safe" or "unsafe"

a better description IMHO is that unsafe rust enables additional functions which are normally not usable as they can create unsoundness if used incorrectly

because this are just additional functions it means also the checks and type safety of all other code is still always there and normally sound, as long as you don't misuse the additional functions to brake invariants

I do count pointer dereferencing as a function converting a pointer to a reference in this context, it's technically not quite right, but conceptually not really wrong either.

So e.g. unsafe rust don't allow you to write to a &T (immutable reference), but using unsafe you could technically bit-wise transmute the &T into a &mut T (100% guaranteed unsound!!!) and write to that. Still at any point all constraints when handling &T did still apply and so do constraints of handling &mut T, you just (unsoundly) converted one into another using an additional function unlocked by using unsafe.

cnuts• 3 years ago

That's really cool, great job!

prydt• 3 years ago

Awesome! What are some good resources on making a simple UNIX like operating system? I know osdev wiki exists, anything else?

o8vm6• 3 years ago

Thanks！For me https://pdos.csail.mit.edu/6.S081/2020/xv6/book-riscv-rev1.p... was quite helpful! but Someday I will write a book on how to implement this OS step by step in Japanese.

bakul4• 3 years ago

How does this compare with https://github.com/dancrossnyc/rxv64 ? From its README.md:

rxv64 is a pedagogical operating system written in Rust that targets multiprocessor x86_64 machines. It is a reimplementation of the xv6 operating system from MIT.

snvzz4• 3 years ago

I immediately notice that o8vm targets RISC-V, whereas rxv64 targets a legacy ISA.

yjftsjthsd-h4• 3 years ago

Is it really a "legacy" ISA if it's actively developed by two different companies (and will be for a long time to come) and is utterly dominant in servers, desktops, and laptops today?

o8vm6• 3 years ago

octox is also different in that the kernel, userland, mkfs, and build system are all implemented in Rust, and I think it is also different at the type level, for example, octox using mpmc for Pipe, and using OnceLock and LazyLock for static variable initialization.

bakul4• 3 years ago

Thanks. rxv64 kernel (the much harder part), syslib and ulib are in rust. Replacing userland C with rust would not be hard. Ideally a general purpose OS should support programs written in any of the common programming languages so it would make sense to have some programs in another language to keep the OS API "honest"!

xolve• 3 years ago

OP, would be nice if you add a license to the repo.

o8vm6• 3 years ago

Thanks! Is this notation in the README not enough: https://github.com/o8vm/octox#license ?

Arnavion• 3 years ago

Both those licenses require you to add the license text as a file to the codebase. Eg see the "How to apply the Apache License to your work" section in the Apache license link that you have there.

Since it's dual-licensed you can add one as LICENSE-MIT and the other as LICENSE-APACHE.

tbillington• 3 years ago

If you like you can copy what I did: https://github.com/tbillington/bevy_toon_shader, which I copied from https://github.com/bevyengine/bevy/ (just that my repo has was less stuff, so might be easier to copy from).

freecodyx• 3 years ago

Not related to the project. Most of the code is unsafe. I really find rust counterintuitive.

This is just a note to myself. * rust uses llvm as a backend * rust tries to solve the memory issues commonly found in C bu enforcing a programming paradigm which allow the compiler to detect them at compile time. * it tries to provide 0 cost abstractions

It works but the code is ugly

segfaltnh• 3 years ago

Wait, most of _what_ code is unsafe? Aside from this one comment it sounds like you just came here to shit on Rust, lol.

stevefan1999• 3 years ago

Oh you have did what I did in the shadow...I wonder if I later GPL'd it the license won't be compatible to take the code in...But I runs in x86_64 with custom QEMU UEFI loader anyway

yasuoyamasaki• 3 years ago

Just one hacker developed this OS, ya know!

Octox: Unix-like OS in Rust inspired by xv6-riscv

Related Stories