Linus Torvalds on userspace filesystems

http://www.spinics.net/lists/linux-fsdevel/msg46078.html

by sasvari • 15 years ago

135 97 15 years ago

barrkel5• 15 years ago

An approach for this kind of problem: suppose you have a very simple language composed of simple, verifiable instructions, like: block read, block write, read-offset, write-offset, simple arithmetic, simple branching; you could compose programs for various file-system operations and pass them over to the kernel for execution en masse, rather than needing a chatty interface.

This basic idea is something I've applied to a file format that stored objects, but where the costs of serializing and deserializing a whole object graph was prohibitive for a request/response round-trip. For any given object path, foo.bar[3].baz, a "program" could be compiled that could be interpreted over the file contents and the answer retrieved. All object paths were available for compilation ahead of time (it was a custom language, long story), so this approach could be far more efficient than any serialization story.

pjscott• 15 years ago

That reminds me of Singularity OS, an experimental OS from Microsoft Research, which has no distinction between kernel and user space, thanks to writing everything in a variant of .NET bytecode.

http://en.wikipedia.org/wiki/Singularity_(operating_system)

barrkel5• 15 years ago

There have been Java OSes too, with the idea that verifiable bytecode does away with the need for "expensive" memory protection etc. Running everything in one memory space is error-prone, but it's the default way of the monolithic kernel. Having services delegated out to processes with separate memory spaces is architecturally more stable; a single bug in an obscure driver in a less actively maintained corner of the kernel shouldn't lead to a system-wide vulnerability. The approach I mention preserves the ability to keep things separate (with appropriate controls and verification at the boundary); the Singularity / Java OS approach seems rather different.

Snoracle says JavaOS itself is obsolete, FWIW.

derleth• 15 years ago

I'm pretty sure Lisp OSes (you know, on the bytecoded Lisp Machines from companies such as Symbolics and TI) worked like that, and I know the CISC variants of the AS/400 architecture worked like that as well. (I don't know the RISC AS/400 or iSeries that well.)

An interesting aspect of that is what it does to your security model: In the AS/400 world, where applications are compiled to bytecode and then to machine code, the software that compiles to machine code is trusted to never emit dangerous machine code, as there are no further checks on application behavior. In the Lisp Machine world, anyone who can load microcode is God. In Singularity OS, the .Net runtime is effectively the microcode and the same remarks about Lisp Machines apply.

DrJokepu• 15 years ago

I wonder how they really achieve that considering that the x86 CPU in protected mode definitely makes the distinction (rings 0 and 3, respectively)?

sparky2• 15 years ago

Sounds similar to the Untrusted Deterministic Functions used in exokernels to let userspace do god-knows-what to underlying hardware resources in an efficient but provably safe way, using a correct-by-construction language [1,2].

Another tool in this space is Native Client; any chance it could be adapted to allow an even smaller, safer x86 subset, so that you could give the kernel pre-compiled 'callbacks' that it could execute on, e.g., receiving a network packet, with 0 context switches? Probably not without a lot of work, since NaCl relies a lot on virtual memory for memory isolation, but it's an idea.

[1] http://portal.acm.org/citation.cfm?id=266644 [2] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.120...

barrkel5• 15 years ago

Nice links. The UDFs look like a very similar idea to mine, taken further and generalized; I make no claim to originality, it seemed obvious to my compiler-oriented eye.

michael_dorfman• 15 years ago

It's a bit amusing to see Linus, all these years after his famous dispute with Tanenbaum, still fighting microkernels. Personally, I think that the Minix source code shows that microkernels are quite viable, and that Linus's disdain is misguided.

j_baker2• 15 years ago

I'm not an OS expert, so perhaps I'm just being an idiot, but doesn't that argument go both ways (and probably more to Linux)? I mean, couldn't one say that the Linux source code proves that microkernels aren't really necessary?

__rkaup__• 15 years ago

I hear Hurd is making some progress...

anant• 15 years ago

Microkernels and Linux style kernels solve different problems. If reliability is your concern, then Microkernels are the way to go because a buggy driver (which will exist no matter how careful you are) can't take down your entire system. For desktop operating systems, where the user really is fine with pushing the reset button when something goes wrong, it is not worth the development effort to write a microkernel.

In the end, the answer is always to pick the right tool for the job. Comparing Minix and Linux is about as meaningful as comparing a phillips screwdriver with a flathead. They're used for different purposes.

hristov• 15 years ago

How does the Minix source code prove this? Minix has existed long before Linux and yet Linux made enormous gains in popularity while Minix is still minor and very rarely used.

Has the Minix kernel shown significant performance results? (I am not being sarcastic btw, I am honestly asking.)

I know OSX uses something like a micro-kernel and it is a perfectly fine desktop OS, but performance suffers once you get to thousands of threads.

tedunangst• 15 years ago

minix was not free for many years. that may also have affected its popularity.

msbarnett2• 15 years ago

I don't think XNU is a microkernel in any real sense. Certainly not in the sense that the Minix kernel is or was.

The filesystem, device drivers, IPC, network stack, etc is all in kernel space. It's true that it's built around the Mach microkernel, but like most systems that built around it, Mach is just a core layer in the kernel onion, rather than a microkernel in and of itself with the other major components implemented as user-space servers.

uriel5• 15 years ago

> I know OSX uses something like a micro-kernel and it is a perfectly fine desktop OS,

OSX doesn't use anything like a micro-kernel, it runs Mach (which even microkernel fans ridicule, it is bigger than many monolithic kernels like the Plan 9 kernel), with a monolithic BSD-style kernel directly on top, when your microkernel has (being generous) two components, both huge, I don't think it is like what most people consider a microkernel at all.

It is also fun to remember that people also claimed Windows NT was a microkernel, and then they added the graphics system right into the kernel.

perfunctory• 15 years ago

> Has the Minix kernel shown significant performance results?

"If the automobile had followed the same development as the computer, a Rolls-Royce would today cost $100, get a million miles per gallon, and explode once a year killing everyone inside."

grandinj• 15 years ago

Minix is still just a toy. It doesn't implement virtual memory, which is one of the hardest things modern operating systems do.

gyom• 15 years ago

I'm beginning to suspect that people report cases of Linus insulting people regardless of whether the discussion was interesting or not.

Linus insults someone on a forum -> first page on HN !!

Just a thought.

dkarl• 15 years ago

Yes, if the discussion has some fascinating relevance for the general HN readership, I can't figure it out. Linus doesn't come off very well, trying to settle a discussion with common sense and sarcasm while the other people on the thread use facts and examples to paint a more complicated picture.

People who think that userspace filesystems are realistic for anything but toys are just misguided.

I use sshfs all the time. Much of the software I use every day only needs to meet "toy" standards to be useful to me. What is Linus on about here?

icebraining• 15 years ago

Yes, but would you rely on it as the filesystem for a public server?

His point isn't that userspace filesystems are useless, but they are much slower, so the existence of a Fuse filesystem is not an argument for not including a kernel space filesystem in the kernel.

swdunlop2• 15 years ago

Linus wasn't very clear, but the issue has more to do with the cost of context switches between the kernel and a userland process to support a userland filesystem. These switches are relatively expensive; a good example is the relative throughput of a LUKS/DM-CRYPT encrypted volume compared to a Truecrypt volume with equivalent cipher selections.

LUKS' throughput is much higher, simply because it does not add at least two more kernel<->userspace transitions on every read and write.

phillmv• 15 years ago

Oh, I remember back in the good-ole days when it'd be hitting the front page of Slashdot. Not a terribly new phenomena.

larrik• 15 years ago

"That's like saying you should do a microkernel - it may sound nice on paper, but it's a damn stupid idea for people who care more about some idea than they care about reality."

He just had to sneak in a dig at Tanenbaum.

yuvadam• 15 years ago

Context: http://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_deba...

And the original thread circa 1992: http://groups.google.com/group/comp.os.minix/browse_thread/t...

shubber• 15 years ago

And, intended or not, at Stallman

btilly3• 15 years ago

I have it on good authority that Stallman now agrees that going with a microkernel was a mistake.

sp3322• 15 years ago

GNU HURD was hamstrung mainly by politics, not so much by technical issues. There was a ton of code written, but it kept getting ripped up and thrown out because someone didn't like it.

uriel5• 15 years ago

This is amusing given the topic at hand, and what the Plan 9 people thought of Tanenbaum:

http://harmful.cat-v.org/software/andy_tanenbaum

KeithMajhor• 15 years ago

Personally, I love how confrontational he is. You don't see very many people who are that passionately practical. It's oddly inspiring...

rwmj2• 15 years ago

.. if performance is your only criterion. Some people (probably a minority) might prefer software that is highly resilient, even if that means it's 10-20% slower.

monopede• 15 years ago

Actually, there are microkernels running in a vast amount of mobile phones nowadays. They typically don't run on the main CPU, though, but on various controller chips.

Microkernels are practical -- they just have different trade-offs. Microkernels are preferable if (a) the whole Linux kernel would be too large, or (b) you consider security more important than performance. It's probably also easier to provide realtime services with Microkernels, but I don't know much about Linux' realtime API to know if Microkernels would be much better here.

Locke16893• 15 years ago

Given that most phones doing more than just providing calling and texting are running on monolithic or hybrid kernels, I wonder where you get this statistic. In addition, the most important part of secure and reliable systems is not what distributed safeguards you build into them, it's the amount of attack surface. If the monolithic kernel + drivers is smaller than the microkernel, it's probably more likely that the monolithic kernel is more secure.

sharmajai• 15 years ago

Snarky as this sentence might be, it is also my biggest takeaway from the thread, his relentless emphasis on practicality and not on ideas might be one of the reason he gets things done. I wish I was this practical.

Legion• 15 years ago

He was replying to someone named Andrew. I'm pretty sure it's a reflex by now.

tzs• 15 years ago

The funny thing is that at the time Linus made his famous remarks about microkernels, QNX was thoroughly kicking Linux's butt in nearly every area Linus thought was a problem for microkernels.

TheIronYuppie4• 15 years ago

One of the smartest guys of all time, but this was unnecessary. Plus, Windows is _generally_ a modified microkernel (some violations, but mostly right) - and it runs > a billion machines.

william422• 15 years ago

In fact, I doubt the kernel is responsible for any real performance issues anymore. A bloated userspace(especially on Windows, but most Linux distributions apply here too) is responsible.

BananaRepublic3• 15 years ago

Lolwut? Windows is at best a 'hybrid' kernel, and with that it's probably 95% monolithic and 5% microkernel-istic in it's design. How does that translate to '_generally_ a modified microkernel'?? You need to seriously read up on Monolithic/Hybrid/Microkernels and the differences between kernel and user-space. I'll be waiting.

j_baker2• 15 years ago

Correct me if I'm wrong, but this is nothing new. We've known about these limitations on FUSE for quite some time, haven't we?

walexander• 15 years ago

Anecdotally, a project I worked on involved evaluating some encrypted file systems on linux and encFS (on FUSE) was at an order of magnitude worse performance than kernel solutions.

Not to say it could never happen, but the tradeoff on performance is just no where near worth it right now.

marshray• 15 years ago

I'd suspected performance might not be the primary purpose when I came across the Python packages for implementing filesystems.

Still, it seems like a lot of the interesting filesystem ideas have to do with non-root users authenticating to something else over some network connection. This type of thing is simply a more natural fit for user space.

Keeping the high-level stateful protocol stuff out of the kernel is usually a good idea except when performance is all that matters (i.e. there will be full-time developers tuning it and cleaning up the inevitable security and crash bugs).

lukeschlather• 15 years ago

The main thing I use fuse for is sshfs, where trying to optimize inside the kernel is unlikely to give noticeable gains relative to network latency.

And Linus acknowledges:

> fuse works fine if the thing being exported is some random low-use

> interface to a fundamentally slow device

leif4• 15 years ago

I don't care what he says, I'm still waiting for an inclusion of some kind of union mount functionality in mainline. I don't need anything fancy, just one ro dir and one rw dir. If he can't even get me that in-kernel, I'll stick with unionfs in fuse, thank you very much.

Don't tell me about aufs, UnionMount, or overlayfs: aufs isn't up to date because it's not mainlined (for no good reason I can find), and UnionMount and overlayfs are too far out of mainline and too much in development to find reasonable packages.

Linus: when you can back up your claims with working code, I'll start listening. ;-)

jamespo• 15 years ago

Isn't it down to you to prove him wrong rather than vice versa?

leif4• 15 years ago

you misunderstand me. I have a working solution, using fuse. if he thinks that's bad, it's on him to provide a correct in-kernel solution, then show me it's faster

rwmj2• 15 years ago

The current status of this seems to be summed up in this LWN article: https://lwn.net/Articles/447650/

ComputerGuru• 15 years ago

Hey, question: What's a convincing use case for a union filesystem?

Also, what I don't think union filesystems are a valid argument. They're not real filesystems in the way NTFS/FAT/extfs/reiserfs or whatever are — there's a lot less work to be done, and I could see the falling under the "toy" category that Linus was talking about...

leif4• 15 years ago

squashfs plus unionmount

commonly used for livecds, I compress /usr, saves me about 5GB and some I/O and therefore some battery

regularfry• 15 years ago

I'd love to use them for VM image building, something I'm doing quite a lot of at the moment. Having an editable FS overlay on top of a pristine root image without having to futz with qcow2 and nbd would make the process a lot simpler.

uriel5• 15 years ago

Union mount functionality is not something that needs to be done in userspace, Plan 9 has union mounts, but that is one of the few things it does in kernel, for details see: http://man.cat-v.org/plan_9/2/bind

The key thing is to have simple, clean and well defined semantics, this is much harder when your VFS is polluted by stuff like symlinks and other weird kinds of pseudofiles.

I know Al Viro has wanted to have proper union mounts in Linux for many years, but getting proper private namespaces was hard enough, and now nobody uses them, which is sad (but more the fault of the suid-centric userspace environment than of the kernel, if you have to be root to create a new namespace, it is rather pointless).

andfarm• 15 years ago

aufs isn't mainline, and will probably never be mainline, because it's ~27,000 lines of poorly documented and often buggy code, written by a single maintainer in a style that doesn't really mesh with the rest of the kernel.

Speaking from experience, there are a lot of edge-case bugs remaining in aufs. It's usable enough for livecds and whatnot, but trying to use it on a server is a recipe for disaster.

andrewcooke• 15 years ago

this reply is a bit more sane - http://www.spinics.net/lists/linux-fsdevel/msg46080.html

glad i don't work with torvlads.

pragmatic• 15 years ago

Is there a website or book with the collected ideas/opinions of Linus and the design of linux?

"that's like saying you should do a microkernel - it may sound nice on paper, but it's a damn stupid idea for people who care more about some idea than they care about reality."

So I look up this microkernel thing and find this old debate: http://www.dina.dk/~abraham/Linus_vs_Tanenbaum.html

It would be great to have a resource that explains the what and the why behind the linux architecture. Also arguments, errr I mean debates, between high caliber smart people would be cool.

There is a book "Just for Fun: The Story of an Accidental Revolutionary" that I ordered, but I have a feeling it doesn't go into the technical depth I would like. I could be wrong, just ordered it, will find out.

uriel5• 15 years ago

cough Plan 9[1] cough

[1] http://doc.cat-v.org/plan_9/4th_edition/papers/names

borism• 15 years ago

It's sad Linus doesn't want to learn.

sambeau• 15 years ago

Interesting in light of Go:

  - New languages are needed for writing distributed/parallel applications
  `Needed', no.  `Helpful', perhaps.  The jury's still out.

uriel5• 15 years ago

Note that the Plan 9 team at the time were already working on Alef ( http://doc.cat-v.org/plan_9/2nd_edition/papers/alef/ ) and would later develop Limbo ( http://doc.cat-v.org/inferno/4th_edition/limbo_language/ ) as the language for the Inferno distributed system ( http://doc.cat-v.org/inferno/ )

treetrouble• 15 years ago

sshfs isn't a toy

16s• 15 years ago

If you've ever implemented real-world code that solves a problem you'll find that there are some people (mostly who have not implemented much practical code but read a lot about it) who want to tell you how you should have done it. I think Linus got sick of hearing these suggestions and began defending his implementation decisions based on his experience and the running code.

api• 15 years ago

http://www.opendedup.org/

Not a toy.

masto• 15 years ago

Linus is increasingly becoming a crazy old man. I find his diatribes largely irrelevant.

Linus Torvalds on userspace filesystems

Related Stories