Roll Your Own UNIX Clone

http://www.jamesmolloy.co.uk/tutorial_html/index.html

by blacky • 17 years ago

119 43 17 years ago

jamesmolloy.co.uk

Writing a kernel is fun... I started writing one a few weeks ago. It's not a real one, just something to play with. But it's very interesting, probably because I'd never worked at such a low level before.

Great resource: http://www.osdev.org/

daeken8• 17 years ago

OSDev and OSDever ( http://osdever.net/ ) are both great resources in general, but I can't help but mention #osdev on Freenode. Incredibly useful resource if you're in need of assistance.

Locke16897• 17 years ago

I'll just leave these here...

http://www.intel.com/products/processor/manuals/index.htm

[disclaimer: I'm an OS/kernel dev]

mahmud• 17 years ago

But you can bootstrap a cheap Unix with chapters 5-10 of this:

http://www.logix.cz/michal/doc/i386/

Getting into protected mode and exec'ing processes in their on private space has been done with ~4 pages of asm.

Locke16897• 17 years ago

Don't run in v8086 mode unless you have to (BIOS boot in Intel VMX). Just go directly to PE or LME as the new docs tell you. It may be interesting to look at the history though.

Also, while devs have to write portable code -- don't bother. It'll save you a bunch of time if you just write the IA32e specific asm code.

If anyone has any interest, I can also post my adaptation of an asmx86 vim syntax file I hacked up from something I found on Stack Overflow. It's not that great, but it does at least fix some of the register highlighting.

sown2• 17 years ago

I gotta ask...if I go do this sort of thing on my own, I might produce something that won't segfault or oops()...if I were to walk into a job interview for an os/kernel dev job (the kind that rand say I did write a crappy primitive kernel would that look good or bad?

I've been working for almost 3 years now and this is an area that I would like to move into.

Locke16897• 17 years ago

I'm not in industry but in academia so I'm hesitant to say anything concerning commercial development. I can't ever see it being a negative thing, but the difference between a toy kernel like he is proposing and a real system is larger than many people think. For example, knowing how to use the GDT and the LDT is definitely required, but knowledge of SMP design and general hardware architecture (think chip design, memory hierarchy, etc) are equally important. If you were looking for a career in systems design, definitely read the Intel Manuals cover to cover -- then go look at the Linux/BSD kernel to see where things were explained so poorly that they may as well be wrong. I would also highly encourage you to take part in a systems open source project (since I work on virtual machines, I would of course suggest Xen or QEMU).

If you like, a much simpler version of the Linux kernel, Kitten OS[1], is used to run at Sandia and in sequence with the Palacios academic virtual machine[2], which is what I'm working on. It may be worth looking at, although I still highly recommend that you look into actual production code.

[1] https://software.sandia.gov/trac/kitten

[2] http://v3vee.org/

dkersten2• 17 years ago

The System Programming manual and the Instruction Set references (I do not have a copy of the Application Programming manual, unfortunately, and don't really want to read it in PDF form) are exceptionally good and have helped me back when I used to mess with toy kernels and also helped with Uni assignments. I'd definitely recommend them to anyone who is interested in kernel or assembly development.

jacquesm6• 17 years ago

Do it! By all means, really, seriously. You'll learn more about coding complex systems than any other project you've ever done.

It's absolutely doable.

If you want to make your life easier go for a micro kernel with message passing. That will partition the tasks to the point where a single determined person can go all the way from MBR to prompt.

Runner up in the experience category: make a game from scratch that has multiple threads of execution, but use a single threaded process. That's almost the same experience but completely in userland, which should make your debugging life a lot easier.

But if you want to learn all there is to know about memory management, interrupts and interfacing with hardware as well as process scheduling build a kernel.

Older dr. Dobbs issues contain tons of useful information.

Here is some food for thought: http://ww.com/task.cc.html and http://ww.com/task.h.html

Other sources of inspiration: the early minix kernels and the early work on what is now known as FreeBSD by Bill and Lynne Jolitz (also described in dr Dobbs iirc)

Locke16897• 17 years ago

If you want to make your life easier go for a micro kernel with message passing. That will partition the tasks to the point where a single determined person can go all the way from MBR to prompt.

I don't know if I'd say that... For one, unless you are actually going to build vdevs and stuff any kernel you build is going to be a microkernel. In my experience, however, I would not say that message passing makes your life that much easier. In addition, it's probably a good idea to write an ELF compatible OS if you want a prompt. Writing a shell isn't that hard, but I wouldn't want to replicate the basic userland tools if I can just statically compile a busybox library. You should also plan to implement some sort of ramdisk if you want a useable environment (or virtual devs with drivers, but that would probably be harder). Actually, I'd recommend just building off QEMU, since that'll give you rudimentary device framework and keep you from having to debug on physical hardware (like I do).

jacquesm6• 17 years ago

Message passing will make the kernel dead simple, after that it is all userland. This makes it much much harder for one device driver you're writing to crash another (or the kernel) by data corruption. A simple post mortem crash inspection will tell you what went wrong. Instead of doing the same on a whole kernel you'll just have that one little process that goes under, and the rest of the machine is still working.

I don't think I could have ever gotten my little OS to the self hosting stage without that.

It still took more time than I care to remember though :)

A virtual machine is a great tool as well, especially if it gives you access to the cpu contents of the running VM.

That will save you days if not weeks trying to find out why your switch from 'real' to 'protected' mode during the boot didn't work...

sown2• 17 years ago

http://www.ipdatacorp.com/mmurtl.html

This is a neat book. I read the first six chapters a while ago and it seemed interesting.

michael_dorfman3• 17 years ago

I'd think that Tanenbaum's book Operating Systems: Design and Implementation (OSDI) and the included MINIX3 source would be high on the list of reading for any kernel-curious developers.

Locke16897• 17 years ago

Good, but a little outdated. For general systems overview, as well as introduction to UNIX architecture, I'd actually recommend my textbook, which is the CMU intro systems book, Computer Systems: A Programmer's Perspective[1]. For OS theory, our OS/advanced OS textbooks are fine, but for actual implementation my coworkers recommended The Design of the UNIX Operating System[2], Linux System Programming[3], and Understanding the Linux Kernel[4].

[1] http://www.amazon.com/Computer-Systems-Programmers-Randal-Br...

[2] http://www.amazon.com/Design-Operating-System-Prentice-Softw...

[3] http://oreilly.com/catalog/9780596009588/

[4] http://oreilly.com/catalog/9780596005658/

ericmc• 17 years ago

I also recommend Computer Systems: A Programmer's Perspective to anyone who wants an intro to systems. One of the best textbooks I've read.

michael_dorfman3• 17 years ago

Outdated? I thought the Linux Kernel was outdated in comparison. The Minix kernel was rewritten from scratch in 2006 for version 3.

GeneralMaximus• 17 years ago

Whenever I think about doing something like this, I always get stuck up on x86 ASM. There don't seem to be many books on ASM out there, and the ones I could find in the bookstores were all written around MASM which, of course, does not work on Mac OS X.

Any ideas?

limmeau• 17 years ago

The difference between Intel/MASM syntax and AT&T/GNU syntax is rather superficial[1], so with a little squinting you can use a MASM book with the GNU assembler. Newer versions of the GNU assembler also come with a directive .intel_syntax so you can write your learning programs in MASM syntax.

[1]http://www.ibm.com/developerworks/linux/library/l-gas-nasm.h...

jacquesm6• 17 years ago

Here's a trick to help get you started:

Write a relatively simple C program that does something useful that you understand thoroughly, then generate intermediate assembly code from your C compiler, with optimization turned off.

That way you get a problem that you already know how to solve in a .s listing that you can inspect and modify to your hearts content.

Then try to optimize it, make it run quicker by rearranging stuff.

You'll learn lots that way and the barrier to entry is low.

klipt• 17 years ago

Also, check out http://webster.cs.ucr.edu/AoA/

JamesM3• 17 years ago

The Netwide Assembler (NASM) which I mention in the article frontpage has the same syntax as MASM without the nasty licence. Some of the assembler directives may differ, but 90% of your program will be identical.

dkersten2• 17 years ago

Thats great and all, but hes forgetting two very important aspects of a real OS: drivers and software.

Ok, hes not building a real OS, but rather a toy OS (presumably for the learning experience), so these things are not so much of an issue. Its actually a good article, IMHO, and I wish it had been available a few years back when I tinkered with writing my own toy kernel. The "Roll your own unix clone" title given here is pretty misleading though.

JamesM3• 17 years ago

I wouldn't say it was misleading - the front page of the article clearly states that a kernel is being developed. Drivers and software belong in the rest of the operating system. Given the availability of GNU software, a UNIX userspace is fairly easy given a smallish set of system calls.

Yes, I'm the article's author. Yes, I'm biased.

asciilifeform6• 17 years ago

Please don't! Build something original. It really is possible!

Once you've built a UNIX clone, you have polluted your mind as an OS designer. I did so (as a standard college homework assignment) and it took me years to shake the crud out of my head.

As for the tutorial, it is very well written, but x86-32-centric and therefore worthless. There are nontrivial differences between x86-32 and x86-64 from the standpoint of an OS author who wants to make full use of the latter's capabilities.

daeken8• 17 years ago

Building an 'original' OS when you don't know how to build an OS is like asking a kindergartener to design a car.

As for the x86/x64 differences, they're by no means insurmountable once you have the concepts down, which is exactly what this tutorial provides.

jacquesm6• 17 years ago

Who knows, sometimes being versed in the status-quo is a hindrance.

asciilifeform6• 17 years ago

What makes you assume that blindly copying Unix is necessarily the easiest way to learn OS building?

JamesM3• 17 years ago

As to your previous point, the tutorial series is aimed to be one thing and one thing only: A set of tutorials to help bridge the gap between the practical and the theoretical as far as OS implementation is concerned.

There are plenty of tutorials on ASM coding, and plenty on low-level C coding. There are plenty of books about OS internals. The books and articles I've found before tend to stay firmly in the theoretical, in the abstract. The aim of this series is to show how the theory can be implemented. Then, the reader has the knowledge to (possibly) implement their own algorithms and know how they link to the CPU's internals.

As to your second point - do you think that the memory manager in this OS is optimal? Do you think that the linked-lists everywhere are optimal? Of course not. Everything was chosen for its simplicity - I failed in some areas I know, and I'm rewriting them as I speak (they're stalled at the moment due to a lack of time); the heap for example is a mess in that series. The new one is much easier to understand.

The series does not create an optimal OS for IA32. So your point about transitioning to amd64 to "make full use of the latter's capabilities" is moot - you have to do more research to optimise for IA32 anyway, let alone amd64!

(And let's not forget that all amd64 CPUs are backwards compatible with IA32).

Roll Your Own UNIX Clone

Related Stories