Ask HN: Recommended resources to learn the Linux kernel and OS theory?

https://news.ycombinator.com/item?id=20809666

by non-entity • 7 years ago

408 83 7 years ago

news.ycombinator.com

So recently I did a couple of minor patches on the FreeBSD and NetBSD kernels and played with some linux kernel. It was the first time in a a few years, I've been excited about programming.

Unfortunately I'm still completely lost. It seems that there's so much to learn and every kernel update breaks whatever you just finished writing. I see terms in unfamiliar with and often find myself googling basic OS concepts.

Are there any recommendations reading materials to get a better grasp on OS theory and / or the Linux / UNIX kernels and programming for them?

lignux• 7 years ago

This looks like a good start: https://0xax.gitbooks.io/linux-insides/content/index.html

rapphil• 7 years ago

In the past I did this: http://eudyptula-challenge.org/

Not sure if they are accepting new applicants, but you can try. Also, you can try to find the assignments online.

neuland• 7 years ago

The assignments are online in various places. Here is a copy I made from someone's Github (not linking GitHub b/c it had answers too): https://pastebin.com/cUXuHcDc

yourbandsucks2• 7 years ago

The O'Reilly book Understanding The Linux Kernel is fantastic. Starts you off in the deep end with memory management/addressing and fans out from there. Notably excludes networking, as the book is long enough without it.

DanAtC• 7 years ago

Any suggestions for Linux networking? I have Understanding Linux Network Internals but it’s a bit dated.

yourbandsucks2• 7 years ago

I think it comes down to just understanding IP and TCP in general. Which I don't really (IP) or do sorta (TCP).

As far as kernel architecture, the rest of the system can work with the assumption "writes go into a buffer, reads come from a buffer, and the networking subsystem does the right thing in between", plus some stuff about DMA optimizations.

slinger• 7 years ago

not exactly focused on Linux but a good resource on networking nevertheless: http://beej.us/guide/bgnet/

weinzierl• 7 years ago

Not very Linux specific but Tanenbaum's "Modern Operating Systems" is an excellent source for OS theory and very readable as well. It's an expensive book, but I found it worth its money.

jascii• 7 years ago

I agree! I took Andy's class that he wrote the book for and it was one of the most influential classes I have ever taken. It is also one of the few books I have kept over the years and throughout all my travels.

Noe2097• 7 years ago

Seconded. I would add "Structured Computer Organization", as a preamble book. I read both as if they were novels - they really are interesting and written in such a way that can hook you just as a good story

slipwalker• 7 years ago

+1 for Andrew Tanenbaum

lorenzfx• 7 years ago

If you are still interested in FreeBSD, "The Design and Implementation of the Freebsd Operating System" is a fantastic book. It's a great guide in how a UNIX system and especially FreeBSD works and written by some of FreeBSD's most respected developers.

non-entity5• 7 years ago

I actually did get a copy of that! Unfortunately, I only have the ebook at the moment, which I find difficult to read

ashafer• 7 years ago

If you want something more hands on you can try "FreeBSD Device Drivers: A Guide for the Intrepid" by Joseph Kong. It is much easier to read, and my guess is you ran into it while making those BSD patches.

He also has a short book on building rootkits for BSD, which is a very fun read and demonstrates how things like system calls work.

If you want a guide to reading actual kernel code, here is a great resource explaining where some of the foundational pieces are. With a tool like cscope, reading the FreeBSD kernel is actually pretty easy.

http://www.watson.org/~robert/freebsd/reading/

Hope that helps!

mjb• 7 years ago

This is a great book, and while FreeBSD and Linux differ in a lot of ways, there are enough similarities to make it worth reading even if you're only interested in Linux.

bytematic• 7 years ago

OSTEP is the best book on Operating systems easily. http://pages.cs.wisc.edu/~remzi/OSTEP/

apacheCamel• 7 years ago

Not sure if it is the best book overall, but I used it for one of my college classes and I think it is a great book for learning much more about operating systems. And it is available for free online so my college student wallet loved it!

yodsanklai• 7 years ago

Not Linux, but I'd recommend this lab from MIT.

https://pdos.csail.mit.edu/6.828/2014/overview.html

I completed this one, but they may have more recent versions. They give you some initial code that you need to complete, as well as tests to check your work before going to the next step. It covers a lot of material. It takes some patience and it's not for complete beginners but it's a great fun project. Not sure, but I'd say it took me 1 or 2 weeks full time.

geofft3• 7 years ago

Yes, there is just so much, and it is not thoroughly documented. "BPF maps" in particular are a) a special feature of a special feature that's Linux-specific, not anything that generalizes across OSes, and b) a feature for userspace anyway, not for kernel internals. But even longstanding kernel-internal features aren't well documented (e.g., the other day I was trying to figure out what struct file's f_version does, and I think there's genuinely no docs for it.) So honestly I think the answer there is to not feel bad about not knowing everything.

The basic trick of dealing with the kernel is becoming comfortable working in a large codebase most of which you don't understand, and figuring out how to find what you need. Honestly, git grep is one of the best tools here. Get some practice finding some specific thing and where it's implemented, e.g., find a syscall (git grep SYSCALL.*foo) and trace what it calls. Find the definition of a structure inside include/ and see who uses it. Get comfortable with the kernel's OO-ish system of operations structs, and get some practice tracing both "this function makes a generic call, here's a sample driver that implements it" and "this is an implementation of a generic function, here's the syscall that calls it."

Beyond that, reading https://lwn.net 's articles is invaluable, partly for the clear prose coverage and partly for the breadth of what they talk about. (You don't need to pay unless you care strongly about this week's updates - you'll learn plenty from reading articles a week behind - but support them if you can, they're an important resource.) Again, you're not going to follow exactly why e.g. Google wants a new syscall for "restartable sequences" on your first read, but you'll get a sense of what is involved in adding a syscall, how various concurrency models work, what other kernel features are relevant, etc.

What sort of OS concepts are you finding yourself Googling? I will say that actually doing a college OS class is what made things like virtual memory management click in my head. It's an intensive approach but writing code in a much smaller kernel than Linux is a valuable way to understand concepts without being drowned in real-world optimizations and edge cases and portability.

non-entity5• 7 years ago

Perhaps I should've said more. I do often end up googling basic OS stuff, especially related to memory (software interrupts still confuse me though, I understand the concepts, but never the implementation), but even more so have been my ignorance of hardware level concepts. For example trying to patch the code for a network card the other week, I learned what watchdog timer is, I've never heard of that prior.

geofft3• 7 years ago

Working through an academic OS class where you implement interrupt handling is super helpful for that - hardware interrupts and software interrupts are similar but not quite the same and actually writing the interrupt handlers makes some things clear.

I can endorse MIT's 6.828 (I've taken it and taught a version if it), which (at least before this semester, they seem to be redoing things) has a lab about setting up interrupt handling: https://pdos.csail.mit.edu/6.828/2018/labs/lab3/ It's entirely possible to complete the labs on your own, and doing labs 1-3 is worthwhile, but it's also a significant time commitment.

Watchdog timers fall in the category of "there is so much" - they're not crucial to how network drivers work, but drivers in a production OS have to care about every feature that might interact with them. :(

livueta• 7 years ago

The basic trick of dealing with the kernel is becoming comfortable working in a large codebase most of which you don't understand, and figuring out how to find what you need. Honestly, git grep is one of the best tools here.

That's a very good point. If we're recommending tools, cscope (especially if you're a vim/emacs user) or opengrok (if you're not or prefer a web frontend for other reasons) are super helpful for navigating and comprehending large, mysterious codebases.

DTrace/eBPF are also valuable tools for grokking kernel functionality.

hatsubai• 7 years ago

One important part about learning how the Linux kernel works is understanding the details of the system's interface the kernel provides. In my opinion, there is no better book out there than the "Linux Programming Interface" by Michael Kerrisk: http://man7.org/tlpi/

It provides extremely detailed information about everything going on in Linux, as well as example programs and exercises to help you further your knowledge. While it doesn't get deep into kernel theory like Tanenbaum's books tend to do, it will provide you with a greater understanding of how things work, IMO.

tbrock• 7 years ago

This book is great. Seconded.

whowhatwhy• 7 years ago

This book is great, so is Advanced Programming in the Unix Environment, which covers much of the same material but details the differences in standards and implementation. I'd also add `Linux Kernel Development` by Robert Love.

aidos• 7 years ago

£50 on kindle! Looks like a great book, but that’s punchy. (Maybe that’s standard for a text book like this, not something I would normally buy)

geofft3• 7 years ago

It's a huge book (1552 pages), much larger than even your average OS textbook. It's more like a Linux encyclopedia.

sruffell• 7 years ago

I was going to suggest this book when I see I'm late to the party, so I'll just have to add my voice to the chorus.

"Linux Programming Interface" is one of the best technical books I've ever read.

koolba• 7 years ago

That’s a fantastic book.

gravypod• 7 years ago

These are some good resources on the things that go into creating your own kernel:

  - http://www.brokenthorn.com/Resources/OSDevIndex.html
  - http://www.osdever.net/bkerndev/Docs/intro.htm

For learning about Linux specifically:

  - https://0xax.gitbooks.io/linux-insides/content/index.html

ndesaulniers• 7 years ago

Join us in /r/kernel. As a mod I'm always helping folks with questions like this, and our community frequently beats me to the punch.

non-entity5• 7 years ago

Didn't know that existed. Will definitely check it out.

qntty• 7 years ago

If you want to understand Linux systems programming, I've never found anything better than the book The Linux Programming Interface. For Linux internals, Linux Kernel Development by Robert Love is good. For something more removed from current systems, The Design of the UNIX Operating System and Operating Systems: Design and Implementation are good.

n_t• 7 years ago

From my previous comment - https://news.ycombinator.com/item?id=18451260

duckqlz• 7 years ago

Beyond the exceptional mentioned texts dedicated to the subject; my favorite resources are:

lwn.net

github.com/torvalds/linux

pm215• 7 years ago

My somewhat off-beat recommendation is "Lions' Commentary on UNIX 6th Edition with Source Code". The sources to a 1976 version of Unix written in an archaic dialect of C and targeting a long-dead CPU architecture are obviously not of immediate relevance to the modern world, and may well not be to your taste unless you already have some sympathy with the idea of retrocomputing. But if you do: 6th Edition is small enough that you really can read through the whole set of sources and understand pretty much how the whole thing is put together; the basic bones of the design are not so far away from Linux, so it gives you a workable conceptual model of what the shape of the Linux sources are and what the various important parts are; the commentary is really good, explaining the inscrutable but nudging you to figure things out yourself too; and overall it is practice in reading a large volume of somebody else's code, which is something you're going to be doing all the time if you work on Linux or any modern kernel. You probably also want a more theoretical and general book too, of course.

jiveturkey2• 7 years ago

And you can witness the glory of swtch() and the original "You are not expected to understand this" comment.

Highly recommended.

alfiedotwtf2• 7 years ago

I've commented below... there's an old book called "Linux Core Kernel Commentary" which is based on the Lion's book. It's a great read.

There's a Xinu book which is similar, but without the accompanying source code.

throwawaypls• 7 years ago

Linux Kernel Development By Robert Love and Operating Systems Concepts by Silberschatz are great resources.

waddlesplash6• 7 years ago

The OSDev wiki can probably teach you anything you need to know about "OS theory" (and practice, mostly on x86): https://wiki.osdev.org/Expanded_Main_Page

Depending on your style of programming, I'd recommend maybe not working on Linux if you are trying to learn kernel development and OS internals.

Have you looked at Haiku (https://www.haiku-os.org/)? We have a very well organized and commented modular-monolitic kernel, and a pretty active development team with a wide range of experience levels. I'm more than happy to help you (or anyone else) learn OS development!

Some example Haiku kernel code:

* "load_image_internal()", which is responsible for creating processes: https://github.com/haiku/haiku/blob/master/src/system/kernel...

* a recent change by me to replace a global lock with two local ones: https://github.com/haiku/haiku/commit/37eda488be1c9fee242e8e...

non-entity5• 7 years ago

I have heard of Haiku,but haven't looked at it in a while. However, I have heard great things about it and would certainly be interesting in learning and contributing

waddlesplash6• 7 years ago

I and most of the other developers hang out on Freenode#haiku, and of course on our mailing lists. Come say hello and get involved, indeed. More than happy to teach :)

xz0r• 7 years ago

I'd recommend maybe not working on Linux if you are trying to learn kernel development and OS internals.

Could you also tell why ?

waddlesplash6• 7 years ago

The Linux kernel is really, really large, and changes extremely rapidly, to the point of breaking such "experiments" (as the original poster already noted.) It's also rather loosely commented and, aside from a handful of core systems, not so well documented.

The BSDs do much better here; they have excellent manpages for almost all kernel APIs, manuals for how the internals work, and very good source code commenting and organization practices. Haiku is not as good as the BSDs on manpages and manuals, but that's the only real way we differ from them (in this respect.)

kjs3• 7 years ago

Complexity.

Start with something where you can substantially wrap your head around the whole thing, like XV6 or Minix, and it's vastly easier to look at chucks of something like Linux and see "oh, this fits into the mental picture here". For me, in the mists of the old days, learning Unix V6 out of the Lyons books (very slim volumes of annotation...wonderful stuff) make it easy to understand what was going on in V7, which made 4.2BSD and progeny accessible, which made SVR[234] somewhat weird and different, but still conceptually understandable, which made most everything Unix-ish afterword (including Linux) something you still had the vernacular to understand.

Now...to be fair...this leaves out a lot of really valuable OS knowledge. There's no IBM large system OS (e.g VM/370->z/OS, VM/CMS, TPF). No Seven Dwarfs OS (DEC VMS, Burroughs B-series, DG Nova, CDC NOS, etc.). Nothing for the really interesting Lisp or Smalltalk machines, the Perq or the Mesa/Cedar workstations. No interesting small systems OS: OS/2, classic MacOS, OS-9, much less the Windows NT family. Much less any of the embedded/RTOS world.

But you can still learn alot.

okl• 7 years ago

Function with 9 arguments, 200 lines. Mix of abstraction levels. Basically untestable. Could use RAII.

waddlesplash6• 7 years ago

Function with 9 arguments

That's nothing, let me show you some of the ones in the VM...

Mix of abstraction levels

What are you referring to here?

Basically untestable.

Unit-testable, no. Integration-testable, absolutely; POSIX testsuites are not so hard to write. And this function is called (tens of) thousands of times per hour while you are using the OS, remember.

Could use RAII.

It is already using RAII reference and lock objects, what improvement do you think could be had here? There are no exceptions in the kernel, so that's not a concern.

If this is your criticism of Haiku's kernel code ... go take a look at some of the equivalents in the BSDs or Linux, then. We are by far and away better than those by your metrics (especially as, you know, those are C and this is C++.)

But do these specific metrics matter? Probably not. "9 arguments" is not a mark against a function in all cases, neither is length or abstraction mixing. Those can and usually are bad, but kernels are a different ballgame altogether.

no-dr-onboard2• 7 years ago

modular-monolitic kernel

This sounds like a bit of an oxymoron. What is a modular-monolithic kernel?

linguae2• 7 years ago

The Linux kernel, as well as the FreeBSD kernel, supports the loading and unloading of kernel modules, which are compiled binaries that can be linked to the kernel while it is running. These modules are still in the address space of the kernel; the term "monolithic" refers to the fact that all portions of the kernel, even kernel modules, are running in the same address space. This is different from a microkernel like Mach where various operating system subsystems such as networking and the file system are running as separate user-level processes (thus having their own address spaces).

waddlesplash6• 7 years ago

"Monolithic" refers more to the way drivers and the kernel interact, i.e. if drivers live in the kernel address space, can read/write all kernel memory, call functions in other drivers, etc., then it's a monolithic kernel. If instead each driver is its own process with address space, and calls into the kernel via syscalls like any other process, then it is a microkernel.

"Modular" refers to linkage and overall design. The Linux kernel is mostly "statically" linked (there is DKMS but it isn't so widely used) with most major systems in the "kernel" itself, and something similar is true in the FreeBSD world.

Haiku, on the other hand, has a rather small (2.2MB binary on i386) kernel with no drivers built in at all (not even PCI); and loads all its drivers as relocateable shared-objects. This means unloading and reloading kernel modules at runtime (or boot) is very easy; so as long as you don't crash or hang the system, you can keep reloading drivers as much as you like without rebooting.

I know macOS also uses dynamic kernel modules, but I'm not as familiar with that system.

apples_oranges• 7 years ago

From not too long ago: https://news.ycombinator.com/item?id=19225268

kernyan• 7 years ago

https://github.com/s-matyukevich/raspberry-pi-os

Matyukevich page on GitHub has lessons with accompanying code to develop a Linux OS for raspberry Pi. The author goes into kernel, processor initialization, interrupt handling, scheduler, implementing syscall, and virtual memory.

I like his approach for these reasons 1. Minimal workable code, 2. Points you to entry function in linux repo 3. Line commentary of those entry function

Note that it's also a somewhat short read as he focuses on the practical implementations instead of the theory.

I myself was hoping that the author continue developing the chapters on file system, drivers, and networking but seemed to have been on hiatus.

gamescodedogs• 7 years ago

Years ago in school, I decided to figure out how the Linux kernel works. So I've downloaded the sources and open them with a notepad++. After an hour I've figured out that I have no idea what am I reading in the code, close the notepad++ and never open it back. Buy the sources are still awaiting in some old directory at my home's PC :)

aeyes• 7 years ago

The Linux kernel is unfortunately pretty far from UNIX nowadays but if you want to understand the basic concepts behind UNIX I think Lions' Commentary on UNIX is still the best resource out there:

https://www.amazon.com/Lions-Commentary-Unix-John/dp/1573980...

whydoyoucare• 7 years ago

"Operating Systems: Three Easy Pieces" is a more modern book to OS concepts, explained with real source code as opposed to algorithms, and is regularly updated.

http://pages.cs.wisc.edu/~remzi/OSTEP/

(Online version above, you can also order a print copy).

0x262d• 7 years ago

I took this class (although not from him). it's a good textbook and was pretty much sufficient.

snoocat• 7 years ago

This is what I learned from, along with some xv6 labs. Just wanted to say that both Remzi and Andrea were exceptional professors and their book is clear and concise.

thinkmassive• 7 years ago

ostep.org is an easy-to-remember shortcut

molteanu• 7 years ago

Operating Systems: Three Easy Pieces [1]

XinuOS [2]

MIT's xv6 OS [3]

[1] http://pages.cs.wisc.edu/~remzi/OSTEP/

[2] https://xinu.cs.purdue.edu/

[3] https://github.com/mit-pdos/xv6-public

apengwin• 7 years ago

+1 for xv6. The commentary book is really well written, and is a straightforward introduction to a lot of important ideas.

https://pdos.csail.mit.edu/6.828/2014/xv6/book-rev8.pdf

jiveturkey2• 7 years ago

The old standards.

Stevens to start. If you don't already know Stevens' thoroughly first, I wouldn't go deeper. The kernel is not separable from the userland-facing API.

I'm not convinced you actually need to / want to understand actual internals. But assuming you do, next is Tanenbaum (maybe start with the newer "Modern" book).

"Operating Systems: Three Easy Pieces" also seems well regarded, although I haven't read it myself.

The FreeBSD book is great, if you're specifically interested in BSD. There's also a video lecture you can buy on McKusick's site. I had the pleasure of an on-site presentation and it is absolutely fantastic.

ekr• 7 years ago

Start with something smaller, such as lk.

filereaper• 7 years ago

Many folks here are recommending reading up on production grade operating systems which might be a very steep learning curve.

I'd recommend starting with an academic operating system to nail the fundamentals down, once you have a solid foundation you can then use all the excellent links provided by everyone else and ramp up on production grade kernels.

I have linked to multiple course syllabus'es below from excellent institutes, pick whichever feels most comfortable.

Harvard with MIPS based OS/161 (I learned this personally): http://www.eecs.harvard.edu/~cs161/syllabus.html

Berkeley EECS 162 (John Kubiatowicz is amazing and is behind RISC V) https://inst.eecs.berkeley.edu/~cs162/sp19/

MIT OCW: https://ocw.mit.edu/courses/electrical-engineering-and-compu...

djhworld• 7 years ago

I've been making my way through this course https://classroom.udacity.com/courses/ud923

It's about 9 years old I think so might be too out of date for you, but I've been really enjoying it. I think they have an advanced version of the course as well if you're already familiar with the basics

Mxtetris• 7 years ago

That appears to be the Intro to Operating Systems course from the Georgia Tech OMSCS program.

Udacity also hosts the GT Advanced Operating Systems course: https://www.udacity.com/course/advanced-operating-systems--u.... AOS is not a "how to" course, but rather a tour of the challenges operating systems may face (distribution, fault tolerance, scale) and of academic papers attempting to solve them (mostly from the 90s and 00s). As such, the material might not be a good fit for the original post, but this commenter found it fascinating.

The reading list is here: https://www.udacity.com/wiki/ud156-readings

Jach• 7 years ago

Zhao Jiong released this excellent work in English not too long ago: http://www.oldlinux.org/download/ECLK-5.0-WithCover.pdf If you know Chinese it was originally done in that language, details are on http://www.oldlinux.org/ It heavily comments / explains everything in the linux kernel v0.12. Quite some time ago, but I agree with the assessment on its relevance for an "inductive understanding". The growth since then is largely in new subsystems, new drivers, but maybe not so much in core theory...

If you're more into the core theory and less about Linux in particular, a fun project-oriented approach to get more familiar with the basics would be to get a little development board for an ARM CPU and try to make a multi-tasking (real-time) OS for it from scratch.

gabrielblack• 7 years ago

Thanks, I haven't this one, it seems interesting.

peterkelly• 7 years ago

I recommend starting with something smaller.

Here's something I put together some years ago - it's about 5,000 lines of code and supports a number of key Unix features. There's a PDF of lecture notes accompanying it.

http://adelaideos.sourceforge.net/

Once you've gone through this you'll be in a better position to tackle the Linux kernel and books on OS architecture. My personal favorite is "The Design of the UNIX Operating System" by Maurice J. Bach. It's an old book but explains things well, and I believe it was one of the references Linux Torvalds relied on when creating Linux.

Diederich• 7 years ago

This is a somewhat different take, and quite possibly impractical.

The main thing I've done over the years to get reasonably 'good' at the Linux kernel (and quite a few other things) has been to read every technical article appearing in https://www.lwn.net/, every week, in full, even if I didn't really understand the material. I've been doing that since 1997, and it's paid off quite well.

Also: please buy a membership, even though you don't have to. It's well worth supporting.

mesarvagya• 7 years ago

In addition to OSTEP, Linux From Scratch [1] is also a good resource.

[1] http://www.linuxfromscratch.org

rubyn00bie• 7 years ago

This book has been immensely helpful to me, it's "Writing an OS in Rust" https://os.phil-opp.com/ and I can't recommend it enough.

Also, Redox OS is an OS written in Rust and has a ton of wonderful documentation on it that's been helping me get into systems development: https://www.redox-os.org/docs/

rpark• 7 years ago

A classic: The Design of the UNIX Operating System by Maurice Bach. https://www.amazon.com/Design-UNIX-Operating-System/dp/01320...

I learned more about OS from this book than any academic material used at the university and gave me a good foundational handle on *NIX's.

theonewolf• 7 years ago

The Minix OS was designed as an academic, teaching OS: https://www.minix3.org/

There is a textbook that goes along with it: https://www.pearson.com/us/higher-education/program/Tanenbau...

The author famously had interactions with Linus Torvalds when he was starting Linux!

And, Minix is inside every modern Intel CPU (https://www.zdnet.com/article/minix-intels-hidden-in-chip-op...).

non-entity5• 7 years ago

Oh yes, I've been curious about breaking into the Minix in intel chips.

darkr• 7 years ago

If you’re looking for something specific to Linux, that is less about the kernel, and more about how the whole operating system is strung together, I can highly recommend: http://www.linuxfromscratch.org/

hd4• 7 years ago

https://github.com/0xAX/linux-insides

alfiedotwtf2• 7 years ago

For the Linux kernel, there is an excellent book called "Linux Core Kernel Commentary" (which is based off the SVR4 Lion's Commentary book), which has the source of the kernel (an old version) printed out and then a commentary on all the bits (code and data structures). It was an eye opener for me.

As for OS theory, "Operating Systems Vade Macum" and "Operating Systems Design and Implementation" are my favorites.

no-dr-onboard2• 7 years ago

In school we used Tannenbaum's MINIX book (Operating Systems Design and Implementation). It was generally a pretty good introduction to "OS Theory", directly references the source code in the back. Useful for a desk book if you're constantly elbows deep in writing syscalls.

linguae2• 7 years ago

For learning the basics about how an operating system works, I recommend Remzi and Andrea Arpaci-Dusseau's "Three Easy Pieces", which is freely available at http://pages.cs.wisc.edu/~remzi/OSTEP/. Both Remzi and Andrea Arpaci-Dusseau are great writers and are prolific researchers in the areas of operating systems and file systems, and their work is top-notch.

For learning the basics about how the kernels of Unix-like systems are designed, I recommend the following books:

- The Design of the UNIX Operating System (1986) by Maurice Bach. You could also compare this book to the source code of Version 7 Unix, which is freely available under a BSD-style license. Many of the basic design principles of Unix-like operating systems have not changed dramatically over the years, and so studying the design and implementation of older, smaller implementations of Unix-like operating systems can aid with figuring out the code base of Linux, which is huge.

- UNIX Internals: The New Frontier (1996) by Uresh Vahalia. This discusses some more modern features that weren't part of AT&T Unix.

- The Design and Implementation of the FreeBSD Operating System (2014) by Marshall Kirk McKusick et al. I've met "Kirk" in person a few times at USENIX FAST conferences; he's not only a great writer, but he is also very friendly and approachable. This is a well-written text describing FreeBSD 11, including modern features such as ZFS.

There are some good Linux kernel books (I'm partial to "Linux Kernel Development" by Robert Love, which helped me out nine years ago when I was writing a device-mapper device), but what I've found during my experience working with the Linux kernel is that the Linux kernel changes much faster than the books describing it get updated. Nevertheless, the best thing to do is to get a solid understanding of how Unix works, then get a basic understanding of the Linux kernel, then start diving into the source code, which is ultimately the definitive documentation of the Linux kernel.

When I was working on that device-mapper device, I found that perusing the Linux kernel mailing list was excellent for learning about how relevant portions of the kernel changed over time.

Good luck and have fun! I haven't done much kernel-level work lately, but I miss working at that level!

master_yoda_1• 7 years ago

http://inst.eecs.berkeley.edu/~cs162/sp19/

moron4hire• 7 years ago

I've semi-seriously been considering starting my own OS project to circumvent some of the IO latency issues with VR and AR applications.

The big metric is motion-to-photon latency, which is a measure (in milliseconds) of how quickly a user's head motion is reflected on the display. Anything greater than ~10ms is considered insufficient, as you've missed a whole frame of rendering for VR headsets that run at 90hz refresh rate.

Current XR applications need to wait for the USB or Bluetooth stack to return the head pose before modifying the view, then wait for the graphics stack to reflect that change. There are many ring transitions along the way. This leads to an absolute minimum motion-to-photon latency that can't be broken just because of OS overhead. This eats into your frame budget, as every ms of context-switching out of the IO drivers leaves you fewer ms to render a frame in time to get it back into the kernel for context-switching back to the graphics driver.

Ask HN: Recommended resources to learn the Linux kernel and OS theory?

Related Stories