Writing an OS in Rust: Advanced Paging

https://os.phil-opp.com/advanced-paging/

by ingve • 7 years ago

336 140 7 years ago

os.phil-opp.com

gbrown_2• 7 years ago

Previous discussion https://news.ycombinator.com/item?id=18903235

Edit: Discussion of previous post.

zaarn13• 7 years ago

That's actually a different post.

sctb• 7 years ago

As an exercise in optimistic parsing, the OP could've meant “Previous discussion of previous post”. It's a handy link in either case, thanks!

gbrown_2• 7 years ago

Whoops, so it is!

rurounijones2• 7 years ago

For someone who is completley not into OS programming but has heard a lot about safety and rust etc. etc.

Assuming the rust safety lives up to the type.

Is this (or any other project) likely to ever become a "real" useful os. What about including rust modules in the linux kernel? What are the barriers?

jerf• 7 years ago

Is someone likely to build a Rust OS that is real and useful? Yes. Projects are already in progress, and even if they all shut down, I'm sure more will exist in the future.

Is that Rust OS going to experience "success" by traditional metrics, such as taking a significant chunk of Linux's marketshare away, and becoming a viable thing you can run on some real piece of hardware for some real purpose? I won't say "no" but it will be at least a decade before that will happen. Not because Rust is bad, or the resulting OS will be bad, but all the entrenched advantages existing OSes have are just staggeringly huge. If a perfect OS jewel of perfection appeared today, but with no other support for it (no software, drivers, applications that can only run on it, etc.) it would be a long, long time for it to get any traction.

AnIdiotOnTheNet7• 7 years ago

The key to getting there (aside from the driver issue), I think, would be creating an environment that is simple, easy to work with, and friendly to creating new tools. The faster you can spin up new tooling, and the more you want to, the more willing people will be to move to the new system. Then you provide VMs, emulators, and/or compatibility environments for running software from "legacy" systems.

Do not attempt to make a new posix system, because we already have giant, entrenched, mature posix OSs and you'll bring nothing new or interesting enough to the table to warrant switching while being held back by the creaking old standard.

I'd be willing to make a lot of sacrifices to move to an OS that met my personal list of wants.

Verdex• 7 years ago

Fully agree.

I think that now is the time for not only new system level programming languages, but now is also the time for new OS implementations using said programming languages. That being said, "now" actually refers to a period of time that will be anywhere from 5 to 25 years long (and you might want to weight that towards the long end).

bogomipz4• 7 years ago

"Yes. Projects are already in progress, and even if they all shut down,"

Might you have any recommendations of specific ones worth paying attention to right now?

andrewflnr• 7 years ago

a perfect OS jewel of perfection appeared today, but with no other support for it...

Sounds like seL4 to be honest, and yeah, it's taking a while to see adoption.

jononor2• 7 years ago

As far as I understand, the goal of the original post is to be an educational resource.

Redox OS on the other hand is a real OS in Rust, already at a basic useful level: https://www.redox-os.org/

As for Rust modules in the Linux kernel, I expect the barriers there to be primarily political. Other proposals for non-C modules in the main tree have not been very welcomed.

swebs2• 7 years ago

There is a demand for a FOSS, realtime, safety critical OS for embedded systems. Linux and QNX are the main options right now, but they both have their downsides (QNX being proprietary and Linux being monolithic).

seren• 7 years ago

FreeRTOS maybe ? But I am not quite sure what is the exact license, or if there are restrictions in terms of features for the free version.

pjmlp7• 7 years ago

I doubt Linux faires that well against something like INTEGRITY OS.

sushibowl• 7 years ago

Redox OS is the most mature rust-based operating system that I know of. Exactly at which point you can call a project such as this useful or real is of course debatable, but redox is advanced enough to run a GUI.

The linux kernel is very unlikely to accept anything other than C. It would further complicate their build system, and raise the (already high) barrier to entry for developers. Rust also does not have the breadth of architecture support that the Linux kernel needs.

std_throwawayay3• 7 years ago

I think none of your points are really set in stone.

* The kernel already contains some C++ which (if you ask C programmers) is not C. And many C programmers are more likely to accept Rust than having to deal with C++.

* AFAIK there is not gcc front-end for Rust yet but LLVM tools can compile the kernel, too. When Rust stabilizes gcc support may come, too.

* The barrier for entry could be lowered by using Rust for Rust programmers if it is employed in some specific modules. The resulting code could be more clear. Not everything in the kernel is pointer pointing and bit shuffling.

* The number of architectures that Rust supports are growing daily.

ChrisSD• 7 years ago

I think it would take a lot of persuasion for Linux to be written in anything but C (and assembly). All their tooling and expertise is geared towards GNU C.

Rust is a young language and LLVM based too so I doubt it'll be accepted any time soon.

yoshuaw• 7 years ago

The MesaLock Linux project aims to structurally replace parts of Linux with safer equivalents. For example OpenSSL: https://github.com/mesalock-linux/mesalink

I think projects such as MesaLock are interesting because they don't try and start from scratch[1], but instead incrementally improve the status quo.

[1]: Though I love Redox & pals. I'm definitely rooting for their success.

gardaani• 7 years ago

steveklabnik: "Not even Rust people think that the kernel should adopt Rust."

https://news.ycombinator.com/item?id=16547450

steveklabnik5• 7 years ago

The Linux kernel, yes.

jcranmer4• 7 years ago

In OS programming, there is a strong bias towards favoring C as the only serious programming language. Rust is one of the few programming languages that has the potential to break through this barrier, especially because the ability to support bare-metal programming is one of its design goals.

But the key word there is "potential"--Rust, as it stands right now, is not ready to be used in production OSes. Features such as inline assembly or other developments for #[no_std] support need to be worked on and supported in non-nightly modes for any production OS to take notice, for example.

mehrdadn11• 7 years ago

I feel like if C++ didn't break through this barrier then I wouldn't hold my breath for Rust.

rhacker• 7 years ago

Just replying to your first sentence: I'm actually curious how many NEW OS projects are being written in C. I would suspect your comment is more geared towards entrenched operating systems where C was already the groundwork.

TheCondor• 7 years ago

Rust modules in Linux? Anyone could start to do it now, but it won’t be mainlined, probably not ever.

A real useful OS in rust? I think this is a complicated question. I don’t see a rust based posix kernel ever taking off in an interesting way, the C based kernels are stable and fairly safe, the community quickly fixes issues when they come up. A new kernel won’t have the architecture and driver support so it will have those downsides with the upside being maybe more safety which is harder to measure. It’s just that compelling. Now a new hypervisor? Maybe some sort of unikernel core? Maybe some sort of container hosting microkernel or some new thing I’m not dreaming up right now? I could see that taking off. I think it would have to be a new thing.

rurounijones2• 7 years ago

Hypervisor / container hosting was what I had in mind yes. Seems that would be the place where a from-scratch OS, safe, would immediately gain usefulness.

vectorEQ2• 7 years ago

>> Assuming the rust safety lives up to the type

^ this... :-)

steveklabnik5• 7 years ago

Fuchsia isn’t pure Rust, but has a lot and it’s only increasing.

zaarn13• 7 years ago

Atm the barriers are all the unstable features you need to rely on as well as rust and LLVM bugs.

The LLVM bug I've encountered makes it impossible to properly handle interrupts with error codes unless you use [naked], which is unreliable in my experience.

The Rust bugs are on one hand a None variable being initialized to a Some() value and a bug where rust injects a few instructions between inline assembly that corrupt the stack.

These problems can usually be worked around in some ways, it's annoying and means the rust compiler is not quite there yet.

However, with rust it's easier to reason about why an error occured than in C, atleast in my experience, although some things can only be found out by trial and error (corrupted stack or hitting undefined instructions), as a result I would predict that Rust could result in a lot more fun designs and architectures than simply writing a Microkernel (Redox).

steveklabnik5• 7 years ago

a None variable being initialized to a Some() value

Did you file a bug about this? this seems extremely bad.

pcwalton2• 7 years ago

The Rust bugs are on one hand a None variable being initialized to a Some() value

Wait. There has to be more going on there than just that—perhaps it's some interaction with naked functions?—or else we'd be seeing crashes everywhere from this.

agumonkey• 7 years ago

Well well well.. as a special kind of user, I often wish for a tinier even limited OS that would part from C type system.

For rust modules.. BSD added lua support for kernel modules.. surely linux can tolerate gc-less rust.

Thaxll• 7 years ago

Probably not, at least on the desktop, good luck getting drivers working when you see the state of Linux / BSD after 20 years and thousands of engineers that worked on them. It's 2019 and we still have issues on Linux that were fixed in Windows XP era.

Also I think people are blinded by that "rust safety" thing for an OS, it doesn't really matter, Redox that is written in Rust had a challenge a year ago about breaking it, it took a few minutes for someone to kernel panic with bad arguments in a user space CLI, so how long do you think it's going to take to have something as secure / stable as Linux for Redox, 10 / 15 years and at the same time you need all the features / drivers to have a usable OS, just forget about it. The only way you can have a "successful" OS nowdays is to target a very specific niche / usecase.

tracker1• 7 years ago

Given the amount of software that runs on servers with a limited footprint need, or orchestrating virtual machines or containers running similar/other software I wouldn't count out the potential. It won't be a desktop application anytime soon, but there is room for the OS is the Service type of OS.

buildbot• 7 years ago

AWS Firecracker, there VM behind Fargate and Lambda is written in Rust, if that counts as an "OS".

https://aws.amazon.com/blogs/aws/firecracker-lightweight-vir...

shmerl2• 7 years ago

One of the areas still in development in Rust now is handling memory allocation failures. That's important for kernel development.

steveklabnik5• 7 years ago

This is both kinda true and kinda not; the allocator work will help with using the standard library data structures on a kernel heap, but many kernels write their own specialized structures anyway, so you’d make the kernel allocator fallible and it all works out.

Gajurgensen• 7 years ago

Why wouldn't it be a good solution to write something like Box, but with a new function that returns `Option<Box<T>>`? I'm not sure what Rust is lacking here.

timClicks• 7 years ago

The barriers are mostly cultural, but even the technical ones are huge. Including non-C code into the Linux kernel would require a lot of internal momentum, because of inertia. On the technical side, it would require everyone downstream of the kernel to update their build process.

That said, there is little stopping you from experimenting with writing your own kernel modules in Rust that wouldn't need to make their way into mainline.

fotcorn• 7 years ago

I started to create the counterpart, a x86_64 bit emulator written in Rust: https://github.com/fotcorn/x86emu

Disclaimer: Almost no comments, unidiomatic Rust, tons of hacks. Currently can run the uncompress code of Linux and some initial startup code. Not much more than instruction decoding & execution and basic page handling is implemented, the rest of the hardware is missing.

Sadly I do not really have time right now to work on it. I am still torn if Rust helps more than it makes everything more complicated, but recent Rust version should have made some things I wrangled with much easier.

bluejekyll• 7 years ago

I am still torn if Rust helps more than it makes everything more complicated

Do you have an example in your code of what is complicated? I took a quick look, and didn’t see anything crazy.

vectorEQ2• 7 years ago

creating recursive page tables is bad idea.

convolvatron• 7 years ago

this does seem really complicated. my guess is that it doesn't save on TLB space since the hierarchy is kind of baked into that cache. it saves a little bit of memory?

DSMan1952762• 7 years ago

It really doesn't save anything, and makes some operations like finding the physical address for a virtual address more complicated. His description of "identity mapping" is also a bit too focused on page-tables and kinda misses the bigger picture. x86_64 has a 48-bit memory space, so you can literally get away with identity-mapping all of your physical memory into a higher-half location with no issues, because the virtual-address space is much larger then the amount of physical memory a 64-bit computer is going to have. Ergo, even if you went with his described approach and just made a single identity mapping for each page table, it would still use so little virtual memory space you could just allocate from elsewhere for your memory-mapped files and such with no worry of them touching eachother. And by doing a single large allocation of all of physical memory, you can use very large page sizes to setup the identity map, significantly reducing the number of pages required.

There is also a presumption that you can only choose one mapping technique, which isn't the case with x86_64 since it has such a large virtual address space and you can map the same page in more then one location. So, as long as you keep track of what's happening with your pages, you can have an allocator that just allocates contiguous pages in the identity mapping, and also have an allocator that maps separate physical pages into a contiguous virtual mapping, and both allocators can coexist perfectly fine as long as you ensure pages are only allocated to one purpose at a time (Which isn't very hard to do. You can even have your virtual allocator just call your physical allocator).

twtw• 7 years ago

Why?

Rusky• 7 years ago

It's not a general solution- you often need to access the page tables of other address spaces, and those aren't picked up by a recursive mapping.

And while a single recursive mapping is a relatively small fraction of the address space, you probably don't want to do it for all address spaces simultaneously.

So you need something more general anyway, and you may as well just use it all the time.

Writing an OS in Rust: Advanced Paging

Related Stories