Examining the Legendary Hurd Kernel (2008)

http://www.informit.com/articles/printerfriendly/1180992

by vezzy-fnord • 11 years ago

86 63 11 years ago

informit.com

vander_elst2• 11 years ago

It would be nice seeing Hurd on L4se now that its source is freely available

lelf• 11 years ago

https://www.gnu.org/software/hurd/history/port_to_another_mi...

vander_elst2• 11 years ago

I meant the mathematically proved fork of L4

InTheArena2• 11 years ago

I sit on our company's architecture review board, and we do a pass on every product that the company does. There are three links that I regularly have grounds to give: The "Linux is obsolete" Debate - http://www.oreilly.com/openbook/opensources/book/appa.html Taligents Guide to Designing Object Oriented Software: http://www.amazon.com/Taligents-Guide-Designing-Programs-Obj... And "The Critique" http://walfield.org/papers/200707-walfield-critique-of-the-G...

Cymen• 11 years ago

The Taligent book is online too: https://root.cern.ch/TaligentDocs/TaligentOnline/DocumentRoo...

nickpsecurity15• 11 years ago

And yet, the monolithic systems (esp Linux) proved to have all the problems predicted. Shapiro did a nice job showing how ridiculous Linus's claim is in Round 2 of the debate, which you left off:

http://www.coyotos.org/docs/misc/linus-rebuttal.html

Can't help but repeat a main point: there are many high assurance (reliability or security) microkernel-based systems that have been fielded but not a single one has achieved this based on the monolithic model Linus loves. QED.

Animats3• 11 years ago

It's sad. Neither Hurd nor Mach reached the performance level of QNX. L4 finally did, although L4 now uses shared memory too much. (While it simplifies the kernel to do message passing via shared memory, it makes it easier for one process to mess up another. Queues in shared memory have to be managed very carefully.)

The real issues in microkernel performance are subtle. Message passing and CPU dispatching have to be well-integrated. Otherwise, when you pass a message from one process to another in a subroutine/service like way and get a response, you go to the end of the line for CPU time twice. A good test of a message passing system is to write a client/server which does some trivial computation on the server and then returns a reply. Benchmark it. Then repeat the benchmark with the system CPU-bound. If performance drops by orders of magnitude under load, the message passing / CPU dispatch relation has been botched.

On the security front, the "critique" says "programs are assumed to represent the interests of the user and, as such, are run with the user’s authority." This, of course, comes from the UNIX/Linux model and still gives us trouble. We still have trouble running something like Flash or a browser on a desktop without giving it far too much authority.

(Of course, if you do have a more granular authority model, apps ask for too much, as the mobile world has learned. Grumble of the week: Firefox on Android now demands access to my phone book. Not because the browser needs it, but because their ancillary products, Sync or Pocket, might.)

betaby• 11 years ago

I keep hearing those stories of "performance level of QNX" since at least 2000. Perhaps it's even true (while I didn't see that during 4.x QNX era, but I blame lack of experience back then). I've seen even various synthetic tests QNX vs L4 couple of years ago. Again QNX source code has been released about 5 years ago if I'm not mistaken. So what was that super performing sauce of QNX which other micro kernels lack. What are those concepts/ideas? Why they weren't re-implemented on L4/Hurd (you can't just borrow, QNS is NOT free software)?

nickpsecurity15• 11 years ago

I'd be interested in hearing Animats' opinion. My guess is that it stayed simpler than Mach but not too simple. Right tradeoffs in the kernel along with fitting into the tiniest of cache and [typical of RTOS's] smart choice of instructions for predictable timing. Here's a paper on an older version of it:

http://cseweb.ucsd.edu/~voelker/cse221/papers/qnx-paper92.pd...

In 1992. it could execute in 8KB of cache with the microkernel and interrupt handler. That by itself should buy it some performance. :)

krylon• 11 years ago

This makes me sad in three ways.

1. Hurd is still not here, yet.

2. With Duke Nukem Forever actually finished, and with Perl 6 looking like it might get finished this year, we kind of run out of good jokes to crack about the Hurd. What will be the next killer-app that will run on Hurd out of the box, once it's finished?

3. Unless I've been misinformed, there are multi-server microkernel systems out there that do work. So it's not like the Hurd was a totally misguided idea. And yet, it's still out there, like cold fusion, artificial intelligence and hypo-allergenic kittens, it's tantalizingly out of reach...

Well, one day

InTheArena2• 11 years ago

Meanwhile, Linux plays a dominate role in the world with a Kernel that is in absolutely everything.

So yeah, Micro-kernels work. But monolithic and simple has been carrying the day for a decade plus.

(and before anyone complains about NT, they lost their Microkernel card when the moved graphics into the kernel. They like XNU on the mac are Hybrid Kernels).

vezzy-fnord4• 11 years ago

Actually if we're going purely by adoption rates, the most common kernels are probably some variant of L4 (used in baseband processors among other places) and ITRON in miscellaneous electronics, both of which are microkernels and the latter also an RTOS.

vezzy-fnord4• 11 years ago

As tired as it may sound, I think there just might be a convenient time for the Hurd to rise, albeit in an unconventional way.

You already have a de facto GNU OS at the moment in the form of the Guix System Distribution (GuixSD), what with it letting you configure the entire system in Scheme, perform transactional upgrades, system state rollbacks, it has its own init daemon and service scripts in Scheme, etc.

At the same time, we've been seeing some efforts in creating tiny builds of the Linux kernel [1] that go as far as to reduce syscalls, VMM algorithms, capabilities, character devices and so forth. There's also the recent Linux libOS effort to create a network stack in userspace as a shared library, though that seems about the extent of it. NetBSD then gives you a userland driver framework with rump kernels.

So what one can do is build a really stripped down Linux kernel, write a Mach IPC compatibility layer, run the Hurd servers on top of it and plug in the resulting product with GuixSD. And you now have the complete GNU system. As far as you'll ever get, anyway.

This has already been discussed before [2], but it's not something that has ever been given any priority. With the current climate, it might be worth a shot.

As to why anyone would want to do this... well, for one you get a full OS that is entirely configurable in Scheme with all of its services running as userland file servers (translators) in a sort of Plan 9-ish way, all the while supporting the Linux API. That's still pretty ahead of GNU/Linux. The Hurd also has features like running multiple instances of itself in user mode (subhurds) that can serve a similar purpose to containers.

It's obvious no one's going to port the Hurd servers away from GNU Mach, so this is the best shot anyone has. Any Hurd or Guix devs in here to comment on this?

[1] https://tiny.wiki.kernel.org/

[2] https://www.gnu.org/software/hurd/open_issues/linux_as_the_k...

nickpsecurity15• 11 years ago

I'm not Hurd supporter but that's an interesting idea. I'd say replace Scheme with Python or something mainstream that's similarly powerful. Adding LISP to stuff usually kills it off unfortunately, with Clojure a freak occurrence. The transactional upgrades and rollbacks feature by itself would be compelling to some audiences.

teythoon• 11 years ago

Hurd developer here.

I also have high hopes for Guix, and in fact there is a gsoc project to port Guix to the Hurd. Guix tries to empower the user, a goal shared by the Hurd. They are a perfect match.

I do not believe that it is easy to provide the Mach IPC interface in the Linux kernel. The IPC mechanism is tightly integrated into the thread switching and scheduling systems.

Furthermore, I do not see much benefits in doing that, the main point would be drivers, which we want to run as a userspace process anyway. And we are doing that using the DDE framework. Rump is interesting, and it wouldn't even be that hard. It just lacks someone actually doing it.

What people don't realize is how compatible the Hurd is. We are shipping the glibc. Debian/Hurd contains ~80% of the software that is found in Debian/Linux. We have Firefox. The other day I deployed two Django sites on my Hurd development server to work out some issues I had with the Apache configuration I had on the production system. I was curious how it would go. All the Python stuff worked out of the box, I merely had to select a different Apache mpm module, because the default didn't work. Need to rescue a box? Append "init=/bin/bash" to the kernel command line to circumvent sysvinit.

Other microkernel based systems might have more ambitious goals, but many of them sacrifice compatibility to reach them. Some aspects in POSIX are hard to implement in a distributed fashion, for example fork(2). But what good is a fancy system if it requires a huge effort to port applications to it?

pjmlp• 11 years ago

Maybe if the likes of Intel and IBM had decided to put their employees on Hurd instead of Linux, the story would be different.

Frondo• 11 years ago

As an aside, you can have a hypo-allergenic kitten today!

Behold, the Siberian Forest Cat: https://en.wikipedia.org/wiki/Siberian_cat

I had one of these, and she was a remarkable cat. Very active, very engaged, loved to play fetch, and unbelievable amounts of fur. My allergic-to-cats friends were totally fine around her, too!

nickpsecurity15• 11 years ago

Duke Nukem Forever shipped before Hurd. Duke Nukem Forever itself had some... delays [1]. Yet, it got done before Hurd and had 376,000+ installs. You've put Hurd's situation into perspective in a way I didn't think possible. Just wow.

[1] http://news.softpedia.com/news/What-Happened-Since-Duke-Nuke...

agumonkey2• 11 years ago

The fun part is that DNF was delayed because of too much money. The HURD didn't need money for that. GNU owns.

tomcam• 11 years ago

Firefox..fills up available RAM with page caches. Often, swapping these pages back in from disk can be slower than re-fetching from the Internet.

WTF? I'm not a kernel guy and would like to know why this is. Very, very counterintuitive to me.

icebraining2• 11 years ago

Non-SSD disks are really slow for random accesses, like you might need if the cached files are not contiguous, and often congested due to background processes competing for their use. Meanwhile, the server most likely has those files in memory, ready to push them.

MichaelGG• 11 years ago

Disks are what, 10ms random seek? You'd need a good, nearby server to service in under that time. And that assumes you've got a connection already established. If not, then add another roundtrip. I'm not sure "often" is appropriate here.

the_why_of_y• 11 years ago

Yes, and it gets even worse if it's not the browser's disk cache, but the browser's RAM cache paged by the kernel to the swap.

Paging out RAM is fast, you just write pages to the disk in chunks that are contiguous on the disk. But the contiguous disk chunks don't correspond to contiguous virtual addresses, and when reading them back in, the order is just however the application chooses to access the pages, so it's quite random and hard to predict. And you can only buffer writes, not reads - you have to block reads until you get the data.

25 years ago you might have 4MB of swap, at 4KB page size that's 1000 pages, or 1000 potential individual points you may have to seek to.

Today 4GB of swap is sort of a minimum size that's not entirely useless, and that's 1 million pages, or 1 million potential seek points.

So IMHO putting swap on hard disks with 5-10 ms seek latency was very useful back in the day, but is pretty much obsolete now - if you need swap put it on flash based storage.

pronoiac• 11 years ago

It's plausible; I've been at a talk where they suggested checking the local cache and requesting it across the network in parallel. I'd thought maybe they were talking about, say, both people on a fast network, but digging into it, that's not the whole story.

Wikipedia [1] has 5,400rpm hard drives having average rotational latency of 5.56ms.

AT&T's real-time latency charts have a few spots with under 5ms latency, and that chart's just the average, I think, so these might be beatable.

[1] https://en.wikipedia.org/wiki/Hard_disk_drive_performance_ch...

[2] http://ipnetwork.bgtmo.ip.att.net/pws/network_delay.html

agumonkey2• 11 years ago

There was a xkcd or something similar saying this, network round trip time was below hard drive on average.

sergiolp5• 11 years ago

Some years ago, I've spent a lot of time studying GNU Mach and Hurd (I've also made some small contributions). I think I can say that I now both pretty well. I even started a project to preserve OSF Mach + MkLinux source code (https://github.com/slp/mkunity), a very cool project for its time (circa 1998).

These days I prefer to do my kernel hacking on monolithic kernels, mainly NetBSD. I've stopped working on Mach, Hurd and other experimental microkernels (there're a bunch out there) because it was becoming increasingly frustrating.

If you'd ask me to define the problem with microkernels with one word, that would be "complexity". And its a kind of complexity that impacts everything:

- Debugging is hard: On monolithic kernels, you have a single image, with both code and state. Hunting a bug is just a matter of jumping into the internal debugger (or attaching an external one, or generating a dump, or...) and looking around. On Hurd, the state is spread among Mach and the servers, so you'll have to look at each one trying to follow the trail left by the bug.

- Managing resources is hard: Mach knows everything about the machine, but nothing the user. The server knows everything about the user, but nothing about the machine. And keeping them in sync is too much expensive. Go figure.

- Obtaining a reasonalbe performance is har... imposible: You want to read() a pair of bytes from disk? Good, prepare a message, call to Mach, yield a little while the server is scheduled, copy the message, unmarshall it, process the request, prepare another message to Mach to read from disk, call to Mach, yield waiting for rescheduling, obtain the data, prepare the answer, call to Mach, yield waiting for rescheduling, obtain your 2 bytes. Easy!

In the end, Torvalds was right. The user doesn't want to work with the OS, he wants to work with his application. This means the OS should be as invisble as possible, and fulfill userland requests following the shortest path. Microkernels doesn't comply with this requirement, so from a user's perspective, they fail natural selection.

That said, if you're into kernels, microkernels are different and fun! Don't miss the oportunity of doing some hacking with one of them. Just don't be a fool like me, and avoid become obsessed trying to achieve the imposible.

vezzy-fnord4• 11 years ago

If you'd ask me to define the problem with microkernels with one word, that would be "complexity".

The problem with Mach, you mean. All the examples you listed are specific to it.

sergiolp5• 11 years ago

I don't know about other implementations, but I remember the original design of l4hurd (based on L4Ka), was even more complex. I'd same this applies to all "pure" multiserver designs.

userbinator• 11 years ago

It's like the argument about excessive modularity in software design in general: you can split a system into so many little pieces that each one of them becomes very (deceptively) simple, but in doing so you've also introduced a significant amount of extra complexity in the communication between those pieces.

Personally, I think modularity is good up to the extent that it reduces complexity by removing duplication, but beyond that it's an unnecessary abstraction that obfuscates more than simplifies.

sergiolp5• 11 years ago

You've just hit the nail right on the head.

nickpsecurity15• 11 years ago

The communication would've happened anyway. Now it just happens through a common mechanism with strong isolation. That all the most resilient systems, especially in safety-critical space, are microkernels speaks for itself. For instance, MINIX 3 is already quite robust for a system that's had hardly any work at all on it. Windows and UNIX systems each took around a decade to approach that. Just the driver isolation by itself goes a long way.

Now, I'd prefer an architecture where we can use regular programming languages and function calls. A number of past and present hardware architectures are designed to protect things such as pointers or control flow. Those in production are not, but have MMU's & at least two rings. So, apps on them will both get breached due to inherently broken architecture and can be isolated through microkernel architecture with interface protections, too. So, it's really a kludgey solution to a problem caused by stupid hardware.

Still hasn't been a single monolithic system to match their reliability, security, and maintenance without clustering, though.

copsarebastards• 11 years ago

I think this is a good example of the law of conservation of complexity[1]. You can't reduce complexity, you can only change what's complex. In the case of monolithic kernels versus microkernels, it sounds like going to a microkernel moves the complexity from the overall design into the nuts and bolts of interprocess communication.

[1] https://en.wikipedia.org/wiki/Law_of_conservation_of_complex...

twoodfin• 11 years ago

http://c2.com/cgi/wiki?RavioliCode

octatoan• 11 years ago

Since those days, HURD has acquired the same reputation in operating system circles that Duke Nukem Forever has among gamers.

What reputation does Duke Nukem Forever have among gamers?

detaro• 11 years ago

Vaporware. Announced 1997. Restarted/largely rebuild multiple times, released finally in 2011.

chatman• 11 years ago

It’s possible to have a complete and usable system running nothing other than GNU code.

Examining the Legendary Hurd Kernel (2008)

Related Stories