HarriL

Plan for a Rust-Based Fork of the Linux Kernel

Background & Motivation

Linux is a monolithic kernel, making it difficult to incrementally introduce Rust without significant modifications. While Rust is being integrated into the kernel for drivers, deeper adoption requires a more fundamental shift. A full Rust-based fork would allow leveraging Rust’s safety features across core kernel components.

Key Goals 1. Process Management in Rust – Rewrite the scheduler and process handling to improve memory safety. 2. Synchronization Primitives – Implement mutexes, spinlocks, and RCU using Rust for safer concurrency. 3. Filesystem & VFS Layer – Enhance security and stability with Rust-based abstractions. 4. Kernel Drivers in Rust – Replace legacy C-based drivers with Rust implementations where feasible.

Challenges • Rewriting large portions of Linux is costly and time-consuming. • Ensuring compatibility between Rust and existing C subsystems. • Gaining support from the Linux development community.

Approach

Instead of a complete rewrite, a modular approach can incrementally introduce Rust subsystems, ensuring compatibility while leveraging Rust’s benefits. Although existing Rust-based kernels like Redox OS are not yet competitive, a forked Rust-Linux could lay the foundation for a safer and more maintainable kernel.

Summary: Why Rust Cannot Be Fully Integrated into the Linux Kernel

Rust-for-Linux (R4L) is an experiment, but it faces fundamental obstacles due to the Linux kernel’s monolithic structure, constantly changing APIs, reliance on GCC, and existing development model.

Key Challenges: 1. Unstable Kernel APIs – Linux kernel APIs frequently change, making Rust integration difficult without a stable ABI, which contradicts Linux’s development model. 2. GCC vs. LLVM Conflict – The Linux kernel primarily relies on GCC, while Rust requires LLVM. This creates fragmentation in the toolchain. 3. Dual-Language Complexity – Developers must master both C and Rust, leading to recruitment and maintenance challenges. 4. Memory Model Incompatibility – Rust’s ownership model does not align well with many kernel subsystems, requiring workarounds that reduce its safety benefits. 5. Monolithic Kernel Issues – Linux is designed as a monolith, where all components deeply interact. Introducing Rust without a complete rewrite results in complex dependencies and maintenance overhead.

Only Viable Solution: A Full Fork

A Rust-based kernel requires a complete fork from Linux, rewriting everything in Rust. However, such a project would no longer be Linux but an entirely new OS.

Thus, Rust cannot become a true part of the Linux kernel without fundamentally breaking its principles. The real question is: Should a new Rust-based OS replace Linux, or should Rust-for-Linux be abandoned?