A reimplementation of NetBSD using a Microkernel [video]

https://www.youtube.com/watch?v=0pebP891V0c

by agumonkey • 11 years ago

153 83 11 years ago

agumonkey• 11 years ago

Youtube video description:

Based on the MINIX 3 microkernel, we have constructed a system that to the user looks a great deal like NetBSD. It uses pkgsrc, NetBSD headers and libraries, and passes over 80% of the KYUA tests). However, inside, the system is completely different. At the bottom is a small (about 13,000 lines of code) microkernel that handles interrupts, message passing, low-level scheduling, and hardware related details. Nearly all of the actual operating system, including memory management, the file system(s), paging, and all the device drivers run as user-mode processes protected by the MMU. As a consequence, failures or security issues in one component cannot spread to other ones. In some cases a failed component can be replaced automatically and on the fly, while the system is running, and without user processes noticing it. The talk will discuss the history, goals, technology, and status of the project.

Research at the Vrije Universiteit has resulted in a reimplementation of NetBSD using a microkernel instead of the traditional monolithic kernel. To the user, the system looks a great deal like NetBSD (it passes over 80% of the KYUA tests). However, inside, the system is completely different. At the bottom is a small (about 13,000 lines of code) microkernel that handles interrupts, message passing, low-level scheduling, and hardware related details. Nearly all of the actual operating system, including memory management, the file system(s), paging, and all the device drivers run as user-mode processes protected by the MMU. As a consequence, failures or security issues in one component cannot spread to other ones. In some cases a failed component can be replaced automatically and on the fly, while the system is running.

The latest work has been adding live update, making it possible to upgrade to a new version of the operating system WITHOUT a reboot and without running processes even noticing. No other operating system can do this.

The system is built on MINIX 3, a derivative of the original MINIX system, which was intended for education. However, after the original author, Andrew Tanenbaum, received a 2 million euro grant from the Royal Netherlands Academy of Arts and Sciences and a 2.5 million euro grant from the European Research Council, the focus changed to building a highly reliable, secure, fault tolerant operating system, with an emphasis on embedded systems. The code is open source and can be downloaded from www.minix3.org. It runs on the x86 and ARM Cortex V8 (e.g., BeagleBones). Since 2007, the Website has been visited over 3 million times and the bootable image file has been downloaded over 600,000 times. The talk will discuss the history, goals, technology, and status of the project.

fmstephe2• 11 years ago

That is very exciting. I am glad to hear about fairly substantial amounts of money being granted for this kind of project. I wish them well, but I won't be jumping on board this bus for a while.

carussell4• 11 years ago

I took a serious look at MINIX over the winter, and digested several of Tanenbaum's talks around that time. (For anyone wondering if this talk contains anything substantially different from past ones, the answer is no.)

Here are some things to add:

- Nowadays x86 is built with LLVM by default, and ARM is using GCC.

- X11 is mentioned in the video, but the 3.3.0 release from last fall didn't ship with a working X server, although past releases have. There is a message on the mailing list from someone who writes that they've got it working on a subsequent snapshot release.

- You may have heard something in the past about 10 minute build times. That info is out of date as of the switch to NetBSD userspace for 3.3.0. MINIX itself (i.e., all the interesting parts) still only takes about 10-15 minutes to build, but there's no way to just slurp down the sources for its kernel/drivers/servers to build a MINIX "core" and then supplement it with the prebuilt binaries for userspace (at least, not without doing some significant work on your end to allow for that). The initial build for x86 on my modest machine takes about 3 hours, almost all of it spent building LLVM twice.

- For anyone looking for serious collaborators, MINIX is seriously lacking in infrastructure from a project/community standpoint. E.g., a fair bit of documentation is missing and much of what you will find on the wiki is out of date. Development processes are neither documented nor easily discoverable because there are effectively no development processes in place. Until about six months or so ago, MINIX was without a bugtracker. Organizationally/project-wise, the whole thing is pretty sparse.

- If you have watched previous talks on MINIX, e.g. FOSDEM 2010, you will be familiar with the open calls for those interested in working for pay, using the money from the two grants mentioned in this video. That money is now gone. During that time, MINIX was basically a research project run by grad students who were working ~full time on MINIX, with the funding from those grants. It's the same now as far as the student-run aspect goes, but with drastically fewer contributions. Not much of the paid man hours seem to have gone towards scaffolding out project infrastructure as I mentioned before, or the sorts of drudge work that volunteers are unlikely to take up.

- The code quality is... I dunno, fair? As I mentioned before, there are/were effectively no development processes in place; no code review, etc. So there's a fair bit of nastiness that got checked in directly, like copy and paste, especially among the arch-specific boot time stuff; the comments are fairly sparse; and you can find dead code and references to routines/fields that either no longer exist or have been renamed, even in the relatively short (~500 line) main.c.

For anyone looking in to maybe starting to work with MINIX, I'd suggest assessing whether or not you would be comfortable striking out and doing things on your own, and then being prepared to do so. With MINIX, you aren't going to find a thriving community that you can just add your piece to, so as to contribute to the effort. You might run into a certain level of that sort of old-guard, paralyzing stop energy, so in a way it's got a lot of the downsides of a greenfield project except with few, if any, of the upsides.

fmstephe2• 11 years ago

Thanks for this post. That is very interesting. It seems like a shame, because a project like this is a long road. If they haven't built a place where work can continue it will likely fade away.

I would love to see a sustainable, non-vapourware, micro-kernel. I've _heard_ such great things about them :)

vezzy-fnord5• 11 years ago

MINIX 3 has been based on the NetBSD userland since the beginning, I think. That said, always interesting to hear Tanenbaum talk and the dynamic upgrade/checkpointing features sound interesting.

istvan__4• 11 years ago

Weren't they using FreeBSD?

carussell4• 11 years ago

No.

psgbg• 11 years ago

Indeed. I really love the part of the reverse check. It would never cross my mind.

mahmud• 11 years ago

Glad you clarified and said Minix 3; I remember the 90s version had its own userland, and its own C compiler (also written AST), I know this because I hacked on them extensively. Probably the first time I have seen extensive source code, more than a snippet or a *.c file.

codezero• 11 years ago

  The latest work has been adding live update, making it possible to upgrade 
  to a new version of the operating system WITHOUT a reboot and without running
  processes even noticing. No other operating system can
  do this.

Can anyone correct me if I'm wrong – but can't Linux do this with Ksplice, and the more recent live kernel patching by Red Hat?

istvan__4• 11 years ago

Yes, and there are some cases when it works. I would not run our production system with Ksplice. The microkernel architecture makes it possible to restart kernel functionality seamlessly. Emphasis on restart. Ksplice just binary patches the running kernel and overwriting the changed parts. Very different scenario.

vezzy-fnord5• 11 years ago

The talk addresses Ksplice specifically, noting that unlike the MINIX mechanism, it doesn't handle significant changes to data structures well.

glibgil• 11 years ago

The video can correct you. Watch it.

MUCH BETTER THAN KSPLICE
KSPLICE can handle only small security patches
KSPLICE patches the running process
Over time, crud accumulates in the process
If the update fails, there is no recovery

https://youtu.be/0pebP891V0c?t=47m56s

nickpsecurity7• 11 years ago

An AIX admin claimed it could do a live update of even kernel code without a reboot. The solution for most others is clustering, needed to deal with hardware risk anyway. OpenVMS clusters have reportedly gone 20+ years without system downtime. Individual instances needed reboots so little that admins occasionally forgot how to do that. So, yes, there's precedents in other operating systems according to their users.

Note: And as far as OpenVMS, I believe it because those designers built it like their job depended on it not failing. A cluster of that OS shouldn't experience any significant downtime given about everything I've ever read on it.

wumbernang2• 11 years ago

We had 11 years of uptime on a VAX cluster at a company I worked for in the late 1990s.

They took it down in 2001 to replace it with something that took up 2U of rack space, about 2kw less power and ran windows 2000.

I turned up in 2012 to replace it again with something cloudy and it had 11 years of uptime (well done NT!) again[1] so YMMV.

The cloud based version has gone down about 10 times (thanks Azure!).

[1] not a great position but this was on an isolated network with locked down everything so less of a problem than a normally networked system.

cyber• 11 years ago

This is pretty cool. It would be neat to seem some of this technology folded back into NetBSD (potentially with the already existent modules infrastructure.)

Animats2• 11 years ago

That's nice, but late. QNX had that 10-15 years ago. With hard real time scheduling, too.

All you really need in a practical microkernel is process management, memory management, timer management, and message passing. (It's possible to have even less in the kernel; L4 moved the copying of messages out of the kernel. Then you have to have shared memory between processes to pass messages, which means the kernel is safe but processes aren't.)

The amusing thing is that Linux, after several decades, now has support for all that. But it also has all the legacy stuff which doesn't use those features. That's why the Linux kernel is insanely huge. The big advantage of a microkernel is that, if you do it right, you don't change it much, if at all. It can even be in ROM. That's quite common with QNX embedded systems.

(If QNX, the company, weren't such a pain... They went from closed source to partially open source (not free, but you could look at some code) to closed source to open source (you could look at the kernel) to closed source. Most of the developers got fed up and quit using it. It's still used; Boston Dynamics' robots use it. If you need hard real time and the problem is too big for something like VxWorks, QNX is still the way to go.)

unethical_ban• 11 years ago

I'm watching the video now, but are you suggesting that QNX, which is not Free and Open Source, has already accomplished MINIX's stated goals of OS reliability?

I would like to hear Mr. Tanenbaum's answer to the less provocative form of the sentiment: "What design decisions were made with MINIX3 that other RTOS with microkernels didn't consider?"

jacquesm4• 11 years ago

QnX achieved Minix's stated goals of OS reliability 20 years ago.

And Minix isn't a micro kernel in the same way that QnX is.

nickpsecurity7• 11 years ago

Tanenbaum cited QNX in Round 2 of the microkernel debate between he and Linus. It's had all sorts of great traits for a long time. It also had plenty of development time and a rip-off open source model to give it capabilities. Like Tanenbaum said in his paper, Minix 3 has had a small amount of core developers working on it for a relatively short amount of time. There's no way Minix 3 will trump QNX with such small resources and I doubt they planned to. It's more a start on building something using better engineering principles that might eventually become a great alternative to other UNIX's and Linux.

stox• 11 years ago

I think UNIX-RTR has met those goals.

vezzy-fnord5• 11 years ago

QNX is fascinating on its own, but MINIX 3 is still a different project in that its full adoption of a NetBSD userland will probably make it more useful for generic servers and workstations as well. They also seem to be going much deeper with checkpointing and dynamic upgrades/hot code reloading.

If you need hard real time and the problem is too big for something like VxWorks, QNX is still the way to go.

There's all sorts of much tinier RTOS like FreeRTOS, MicroC/OS and Contiki that are used out there for particularly critical and/or constrained environments.

david-given• 11 years ago

QNX has a big advantage in that from userland it's basically a Unix. You can develop on it completely self-hosted on a desktop PC. The GUI's pretty good; it even comes with Java and Eclipse.

They've downplayed that recently, alas, but I believe that if you hunt around on their website you can find a bootable CD. I think platform support has slipped a bit so you might have trouble making it boot.

Way back when, there was a QNX demo floppy, which was a bootable 1.44 MB floppy disk which contained a full GUI, web browser, dialup modem support, etc. It'd run on a 386 with 8MB of RAM.

http://toastytech.com/guis/qnxdemo.html

QNX is pretty awesome.

Edit: Here's a similar writeup of the bootable CD, using a more recent version of the GUI. http://toastytech.com/guis/qnx621.html

bch3• 11 years ago

If QNX, the company, weren't such a pain...

Well, regardless of how late, Minix is bringing it fuss-free to the rest of us now.

pjmlp3• 11 years ago

Not really free from EU tax payers point of view.

I am not complaining as I find Minix great, just making the point that nothing is really free.

lambdaelite• 11 years ago

Funny you mentioning L4... based on the HN title, my first thought/hope was someone got seL4 running with the NetBSD userland.

justincormack• 11 years ago

Are you interested in this? I am planning to get the NetBSD rump kernel running on sel4 at some point soonish, not quite the same but you would get much of userland. (Email in profile).

rbobby• 11 years ago

RIM bought QNX a while back... which probably has tossed some spanners in the works in various ways.

soperj• 11 years ago

Based on what, you not being a fan?

Animats2• 11 years ago

That's when it suddenly went from open source to closed source. One day, RIM removed the sources from the web and FTP servers.

QNX used to be a standalone company. They declined a buyout by Microsoft. Then one of their key people died, and they sold out to Harmon, the car audio company. So then they were heavily into automotive dashboard applications, which continues. Then RIM bought them, which gave the Blackberry a better underlying OS but didn't fix Blackberry's problem of an obsolete UI and business model.

Meanwhile, QNX still sells to real-time and industrial automation customers, but those customers feel kind of neglected. Incidentally, the Boston Dynamics robots all have QNX managing the balance and the hydraulic servovalves. You need reliable hard real time for that.

nchelluri• 11 years ago

When is VxWorks inappropriate, but QNX appropriate?

EDIT: http://www.embeddedrelated.com/showthread/comp.arch.embedded... says:

the most fundamental difference between VxWorks and QNX is as you have described, QNX lends itself to a message passing architecture while VxWorks lends itself to a shared memory architecture.
My personal opinion is that a message passing architecture is easier to get to grips with and as such is potentially easier to understand and debug.
However, the majority of software engineers with experience of an embedded RTOS will be very well informed about the Shared Memory architecture.

gte525u2• 11 years ago

I think it's less of an issue now than say 10 years ago. VxWorks 6.x added support for protection domains (MPU/MMU support) and RTPs (real time processes). VxWorks 5 everything operated in the kernel. Even with 6.x very little typically runs by default in user space on a VxWorks setup.

With respect to the message passing - both support messaging. VxWorks has several types of message queues - vxworks proprietary msgQLib API, POSIX api etc. QNX has much the same MsgSend/MsgRecv which is the microkernel API and POSIX. QNX has an add-on PubSub middleware that the OP of the usenet group may be thinking of.

luckydude• 11 years ago

He lost me when he said "start a new window" as the work around to not having job control.

Neat idea but seems nowhere near done.

And as others have said, this was nicely handled by QNX way more than 15 years ago, I was running multiple users on a 80286 around 1986 or so. Really neat system.

sigzero• 11 years ago

I loved QNX.

mzs• 11 years ago

Oh wow, mklinux is still around... http://www.mklinux.org/

Sorry this really has nothing to do with the video, just that the tangental thought of linux on mach made me wonder and I was pleasantly surprised.

boardwaalk• 11 years ago

I was trying to install minix 3.3 for fun and ran into a bug in the e1000 driver that caused VirtualBox to throw up. It's fixed already, but not in 3.3:

https://github.com/Stichting-MINIX-Research-Foundation/minix...

virtio seems to be working.

nickysielicki2• 11 years ago

I am 100% on the side of Linus Torvalds when it comes to microkernels.[0]

I will concede that in some instances a microkernel may outperform a monolithic kernel in stability or performance or both. I am not the least bit excited about any progress made in microkernels, I feel that it can only result in much more closed systems that are easier to implement in ways that make them harder to modify. This is why I wish for Hurd to continue to fail.

[0] http://www.oreilly.com/openbook/opensources/book/appa.html

PuercoPop• 11 years ago

I'm confused, I understand you think the Hurd is making a technical mistake but why do you want to fail?

Is it because you know that usage decisions of software are not based on technical merits? Or do you not want to be proved wrong? Or something else?

nickysielicki2• 11 years ago

Let's go to an alternative universe where Hurd was successful in the 90's and it reached common usage to the extent that Linux has today.

You're Western Digital in 2008 and you're making a TV set-top box called the WDTV-Live. I own one of these in real life universe. It runs linux, which is awesome, because that means that I can SSH into it. It runs an apache server in my home. It can download from usenet or torrents. I can control it via SSH instead of using the remote control.[0]

In this alternative universe, WDLX going to use Hurd instead of linux, because for this small device it will certainly have better performance on their underpowered MIPS chip. And they're not going to ship anything besides what they have to, becasue this is a small embedded computer.

What happens to that homebrew community when they ship a microkernel with proprietary servers for everything, and nothing else? It's going to be profoundly difficult to develop on this. You might already see this if you own a chromebook or a WDTV-- missing kernel modules means that you simply can't do anything without compiling your kernel. Couple this with secureboot and you're locked in.

I'm no expert on these things, most of this is based on brief research from years ago. If you think that I'm wrong, please tell me why, I'd love to be proven wrong. But for the time being, I believe widespread implementations of microkernels would be very anti-general-purpose computing.

[0] http://wdlxtv.com/

vezzy-fnord5• 11 years ago

Microkernels have been running the world behind the scenes for a while now, but most people don't seem to have gotten the memo and are still stuck with associating the Mach server as representative of u-kernels in general.

JoshTriplett3• 11 years ago

There have been wildly successful microkernels. One of Xen's greatest successes was demonstrating that to encourage widespread adoption of a microkernel, you rebrand it as a hypervisor. More recently, some people have started running software directly under Xen without a full OS, including language runtimes, all without ever calling it a microkernel.

pjmlp3• 11 years ago

Yeah, for example each Symbian phone has one.

Also Mach OS X and Windows are hybrid in design and not the monolithic traditional UNIX way.

cbd19843• 11 years ago

Microkernels are the wave of the future and always will be.

rodgerd• 11 years ago

L4 is used in the secure element of every modern iPhone. OS X/XNU is based on microkernel designs. Windows is a hybrid.

regularfry• 11 years ago

The live update and reincarnation server in general remind me a lot of the ideas in Erlang.

fithisux• 11 years ago

I believe microkernels are a very good abstraction/concept.

For me it is very interesting. I would like to see some day darwin running on a real micro kernel (e.g. an updated mach).

I also like the fact that Hurd makes some progress. when ready, I will definitely switch.

In any case hardware vendors must release more documentation on their hardware to revive the OS scene. Blobs do not do much good.

brian_smith• 11 years ago

I actually just bought the 3rd edition of Operating Systems: Design and Implementation (and Design of the Unix Operating System) when I watched this. Now I'm quite excited. Any recommendations for other books on this topic?

nickpsecurity7• 11 years ago

The thing I like best about Tannenbaum's videos is that the dude is a great presenter. Funny too. You just don't get bored easily. We might get more outreach on some of these topics if presenters in those areas dress it up better.

A reimplementation of NetBSD using a Microkernel [video]

Related Stories