Lua support in the NetBSD kernel

http://mail-index.netbsd.org/source-changes/2013/10/16/msg048283.html

by jboynyc • 13 years ago

110 50 13 years ago

mail-index.netbsd.org

Holy crapola - they are putting a Lua interpreter in the kernel, and it has access to some limited calls.

This is partly driven by "we need device drivers and no one understands C anymore", partly because it's cool, but mostly because Moores law is still alive and well. They are putting a scripting language in a kernel ! Forget write your app in python / perl / Ruby then optimise in C - the raw power argument is going to overwhelm us all.

I heard a stat the other day - that a greetings card, the kind that plays a silly tune, has more processing power than all computers on the planet in 1960. And we should expect a similar growth in the next 50 years. Even if that's out by two orders of magnitude just let it sink in.

I think we are at a Cambrian explosion period - where the goal is to try out every possible new body shape, as fast as possible and see which ones get the Darwin seal of approval.

New organisation forms are possible, some for the first time in human history, new ways of thinking and communicating - it's stuff like this that makes one realise the water around the frog is getting hotter.

(Still it's worth remembering that most train companies in 19C England failed and the average return for stockholders was 10% - just because the world will change beyond recognition does not mean industry stocks is a great return)

recuter• 13 years ago

Lua can be sandboxed quite effectively, can you elaborate on your concerns?

vor_• 13 years ago

What concerns?

tonyplee• 13 years ago

Haven't they been keep saying that about JAVA - Just Another Vulnerability Announcement?

Mac just turn off JAVA completely in user space and NetBSD is putting Lua in kernel?

NSA: "I have a dream! Someday, a script language will be in BSD kernel, so I can ping an script into the kernel"

:-)

ketralnis3• 13 years ago

Kernel programming is an eternal struggle between people trying to pull more into the kernel for performance, and people trying to pull stuff back out for safety and modularity. We've all seen the arguments about the in-kernel HTTP daemons and whatnot. I tend to be on the side that says that more things should be pulled out.

This is partly driven by "we need device drivers and no one understands C anymore"

I don't buy this argument. Who are the people

* Using NetBSD

* having cutting-edge hardware for which drivers don't exist

* deciding to write device drivers

* that have device specs/docs needed to write them

* that don't know/want to use C

* that would do it if only they could write it in a scripting language?

Do these people exist? I assert that the far more common case is that the device specs, documentation, BIOS, etc. aren't accessible. If that's true, they couldn't write the driver regardless of the language, C is hardly the limiting factor here

gwu78• 13 years ago

"Do these people exist?"

This is spot on. When you hear NetBSD developers arguing about proposals (or commits, ex post facto, like here), there is often mention of "users". But these are mostly fictional.

I think it's safe to say most NetBSD users are comfortable working with C.

I reckon it's not much different with OpenBSD: most changes they make are for the benefit of the developers themselves, to suit personal preferences, than for the benefit of other "users". (And that's perfectly acceptable, I think.) Of course, these changes can benefit users beyond simply those who make them and those who approve them, but those users are not the ones driving the changes.

tedunangst3• 13 years ago

A far better rationale is that it makes fast prototyping easier. You write three lines of Lua, stuff into the kernel, and run. Did the blinkenlights blink? Yes, good, now translate that code into the C driver.

chubot• 13 years ago

We've all seen the arguments about the in-kernel HTTP daemons and whatnot. I tend to be on the side that says that more things should be pulled out.

I always wondered if a lot of components like this could be either sped up, or moved out of the kernel, using a zero-copy messaging system based on splice() / vmsplice() in the Linux kernel (which works by page table remapping as far as I understand). I guess the prime candidate would be device drivers.

TheCondor• 13 years ago

I've used lua in kernel space elsewhere. It links easily, it can invoke your C functions easily, and it's great for testing stuff. I could see some clever hooks being out in so you could custom tune some stuff. I bet typically they use it for testing though.

fafner• 13 years ago

Are they using regular Lua? Because LuaJIT can actually be quite fast. Snabb Switch is written mostly in Lua and it features Linux userland ethernet device drivers. http://blog.lukego.com/blog/2013/01/03/snabb-switchs-luajit-... (edit: link to the code https://github.com/SnabbCo/snabbswitch/tree/master/src/apps/...)

btw. there once was a project which added a Scheme interpreter to the Linux kernel. Sadly it's dead now: http://www.abstractnonsense.com/schemix/

maximilianburke• 13 years ago

I'm not sure they would be well suited together, NetBSD has a huge list of old hardware they support; I don't think Mike Pall will be porting LuaJIT to m68k anytime soon.

baldfat• 13 years ago

Moore's law will be over by 2025 (We have 12 years or so). Semi-conductor will allow a separation of just 3 - 5 atoms to the conducting part. The belief is that quantum physics will allow the electrons to go through "walls" at that point and the switch will always be closed.

I think we will go back to the days of 640k where there is a ceiling and we will just have to refine for "large" projects. Software and not hardware will be Moore's law BUT only for a extreme circumstances. The Golden Age is coming to an end in our life time.

chipsy• 13 years ago

Some discussion: http://mail-index.netbsd.org/tech-kern/2013/10/06/msg015632....

jarjoura• 13 years ago

Isn't this similar to Windows wanting to rebuild their entire userland on top of .NET?

tedunangst3• 13 years ago

I would say, no, not at all. Rewriting your entire userland is not similar to adding a script interpreter to be used for writing kernel extensions.

pjmlp5• 13 years ago

Actually the best example is OS/400 mainframe system from IBM.

The kernel is a mixture of Assembly, C++ and Modula-2, with a JIT.

All user space code compiles to bytecode.

The kernel JIT compiles the applications on first execution. Afterwards the application is validated against generated native code, and re-JITed again if the bytecode has changed between executions.

So Microsoft's attempt was actually nothing new.

anttiok• 13 years ago

It's awesome to see this work finally hit the tree. Now, I cannot say I agree with all of the motivations. For example, I do not believe that prototyping should be done in kernel mode at all, nor do I believe that using Lua will shield from incorrect operation or crashes. However, there is already proof that device drivers can be written in Lua (https://github.com/SnabbCo/snabbswitch), and it's great to see more push in that direction.

After all, why use C if you have a working alternative? We just need real world experiments to see if Lua is a working alternative for the kernel.

morenoh149• 13 years ago

yay. I was just looking at learning lua to do corona development

ape4• 13 years ago

The note about the copyright on the diffs seems a bit paranoid or something.

almk2772• 13 years ago

It's good somebody even tries to break the old myth about C as an essential kernel part. C is _not_ good for drivers: it lacks most of the appropriate abstractions, just as facilities to create them. Once we switch from asm to C - why not go further?

Speed matters, of course, but I suppose that even script speed is enough for many drivers (and, just in case, we have luaJIT).

Don't sure if lua is a right choice, but it's an interesting experiment - and this is what NetBSD is good for.

simias3• 13 years ago

C is _not_good for drivers

You'll have to back up that assertion with some facts here, there's a whole industry and millions of lines of code that contradict you.

almk2772• 13 years ago

It's just legacy. C as infrastructure is good, C as a language is bad. There's a whole industry and millions of lines of code in visual basic, but I never met somebody who says that VB is good.

Is there a way to say "do this under spinlock"?

Is there a way to handle errors not mixing it with main logic? Not wrapping nearly every line with if/goto?

Is there a way to create a generic collection? No, linux list.h is not that: it requires manual initialization, it does not work with elements, but with their special parts, it's macroses requires logically unneeded information (such as 'member' in list_for_each_entry), and so on. And this is the best you can do in C.

C is too unobvious, too. Google for "wait4 backdoor".

C works, of course. It just requires too much boilerplate code and too much programmer resources.

pjmlp5• 13 years ago

C is designed for security exploits from day one.

The industry of million lines of code exists, because UNIX and its system programming language, C, spread into the enterprise when the university students brought it from their labs into the enterprise.

Thankfully the industry is slowly moving to other languages.

simias3• 13 years ago

Anybody has a link that explains their motivations? There's a lot of speculation in this thread but surely NetBSD being an open community there must have been a public discussion before they merged that in?

I write linux drivers for a living and I must say it puzzles be a bit. There are times C falls a bit short (if only by its syntax and idiosyncracies). There are times I wish I could use something like a subset of C++ (for templates and destructors mostly) or maybe Rust these days. But a scripting language?

You can already write drivers in userland (using /dev/mem for instance). You can already write device drivers any way you like that way. One of the main limitations is IRQ handling, but do they propose handling IRQs in a scripting language? Maybe I'm too close minded (or just brainwashed by years of C) but it sounds like a recipe for madness.

I actually used lua to write a small userland test driver (it was convenient because the lua interpreter is very small and easy to compile and cross-compile).

It was quite painful really. There's no builtin bitwise operator (although it's now in the standard library), all integers are stored as double floating point values (floating point in the kernel is hardly ever needed and usually very poorly supported). Using integers and bitwise operator is like all you do in a kernel driver (ok, I'm exaggerating, but not by a lot).

I guess they might have changed the syntax a little bit (and use ints instead of doubles for storing integers) but then it means their lua is not compatible with the "real" lua so you can't prototype your code in userland with any stock lua build.

I dug up my userland test driver code, here's what the naughty bits look like:

     function avi_unset(addr, val)
       local v = assert(devmem.read32(addr))

       v = bit32.band(v, bit32.bnot(val))

       avi_write(addr, v)
    end

For reference here's what it'd look like in C and its bitwise operators:

    void avi_unset(unsigned addr, unsigned val)
    {
      unsigned v = __raw_readl(addr);

      v &= ~val;

      __raw_writel(addr, v);
    }

pjmlp5• 13 years ago

Symbian, BeOS, Windows and Mac OS X have C++ support for kernel drivers, just as info.

adamnemecek2• 13 years ago

Does anyone still write drivers for BeOS?

pygy_2• 13 years ago

I won't address your general point, because, I never wrote any drivers, but the following may appease some of your concerns.

Changing the number type in Lua is only a few #DEFINEs away. See http://www.lua.org/source/5.2/luaconf.h.html#LUA_NUMBER and the comment above it.

Regarding the lack of infix bitwise operators: usually, the functions are cached in locals. It is not as readable as C, but less bulky than what you wrote:

     local band, bnot = bit32.band, bit32.bnot

     function avi_unset(addr, val)
       local v = assert(devmem.read32(addr))

       v = band(bnot(v), val)

       avi_write(addr, v)
     end

If it became apparent that infix operators were needed, adding them to the VM is trivial (There's a power patch for Lua 5.1, and a patch for Lua 5.2 was recently posted in the mailing list). If you want to use the stock VM, you can still use a preprocessor (which makes things a bit more cumbersome).

simias3• 13 years ago

I remember looking at this LUA_NUMBER but I wondered if changing it to an integer type would break something, it's not explicitly mentioned that you can use an integer type there. I'm sure it can be done but it still means that your custom lua will behave a bit differently than the "stock" lua (especially when it comes to things like roundings and overflows). That might be a bit problematic if you prototype your code on one and then attempt to run on the other.

As for your improved code I didn't know I could do that, it does look much better indeed.

Again, maybe I'm just nitpicking but my main point was that lua, scripting language or not, just didn't seem better suited than C at writing device drivers.

That being said, having a good in-kernel REPL would revolutionize kernel hacking, but it seems very hard to do right. Lots of kernel code is very timing-sensitive and you can't afford to wait for user input when you're in a critical path. Just look at the difficulty to implement proper kernel debugging to get a taste of it.

The thing is, the NetBSD people know that and more and yet they decided to go with it. I would be very interested to know their objective/motivation. I googled a bit around and found this:

http://netbsd-soc.sourceforge.net/projects/luakern/

The page mentions some use-cases (like power management and packet filtering) but those can already easily be handed to the userland through the sysfs and tun/tap interfaces for instance. I don't get what problem it's trying to solve.

Lua support in the NetBSD kernel

Related Stories