Live patching for Linux 3.20

http://lkml.iu.edu/hypermail/linux/kernel/1502.1/00753.html

by meskio • 11 years ago

200 30 11 years ago

lkml.iu.edu

matteotom• 11 years ago

Here's some more discussion from LWN: https://lwn.net/Articles/597407/

(Note: from May 2014)

SEJeff3• 11 years ago

And for those who prefer the official lkml.org link:

https://lkml.org/lkml/2015/2/9/534

sigjuice• 11 years ago

lkml.org explicitly states that they are unofficial.

From https://lkml.org , "In case you haven't read the titlebar of your web browser's window: this site is the (unofficial) Linux Kernel Mailing List archive."

agumonkey• 11 years ago

Firefox Nightly doesn't display that neither in the tab name nor the Xwindow title. I'd never have seen this ...

resc1440• 11 years ago

And lkml discussion of the follow-up patches: http://lkml.iu.edu/hypermail/linux/kernel/1502.1/00694.html

peterwwillis• 11 years ago

An interesting case of the Cathedral causing siloed duplication, and the Bazaar creating a better solution without duplication.

rockdoe• 11 years ago

Ignoring the fact that both SuSE and Red Hat are paying the involved engineers their paychecks in order to get the feature in their silo first.

bjackman• 11 years ago

This is cool as fuck, I didn't know about kpatch or kGraft.

Does anyone know if any other OSs have live kernel patching?

feld2• 11 years ago

FreeBSD has not gone quite that far, but there was a PoC for loading new kernels without rebooting.

https://www.bsdcan.org/2012/schedule/events/325.en.html

SEJeff3• 11 years ago

Yup Linux has had a similar feature, kexec, for several years.

crististm• 11 years ago

Not that it matters too much for the current ecosystem, but Lisp Machines had this 25 years+ ago.

lemonade3• 11 years ago

I was told by Andrew Tanenbaum some time ago that Minix has had this kind of feature for a number of years now.

kasabali• 11 years ago

Windows Server 2003 had a live patching feature [1] and it was also applicable to kernel patching according to [2]. For some reason it was quickly abandoned, though.

[1] https://technet.microsoft.com/en-us/library/cc781109%28v=ws....

[2] http://jpassing.com/2011/05/01/windows-hotpatching/

mkonecny• 11 years ago

Does anyone have a simple example of how this would work? I can't wrap my head around code evolving during runtime for any arbitrary binary change.

detaro• 11 years ago

(just what I gathered from a bit of mailing-list reading, if any of this is wrong please correct me)

I don't think it can do arbitrary changes, it "only" applies specially prepared changes to the running system by replacing function call targets while tasks are sleeping. The biggest limitation to changes probably is that old and new code runs in parallel, so you can only do changes to data structures that won't confuse the old code. "Simplest" use case might be adding guards against exploits to syscalls.

I can't tell how viable it would be for the new code to build an entire parallel structure and to only switch to this after everything is migrated, or how deep into the kernel these changes could go. Could one fix a file system driver while using the file system?

AlyssaRowan4• 11 years ago

Yes, it's just that the fix won't happen for you when you're halfway through a patched function.

DSMan195276• 11 years ago

I took a quick look at the accepted patch. while I can't guarantee I know what's actually going on, my understanding is that patching individual functions works by sticking the replacement functions code somewhere new in memory, getting a pointer to it, and then over-writing the code in the old function to jump to the new one. (Kinda like short-circuiting the old function - all the old code still calls the old function location, but that location simply says 'jump to this new location over here').

It looks like, however, because kernel modules seem to be in elf format (Don't quote me on that, just going from the code), elf format includes a 'relocation table', which is basically a table that says "this function is located here, and this next function is located here, and ..." for every function in the module. Ignoring why that is actually there, they can take advantage of the relocation table and replace a functions location with the location of the replacement function, effectively overwriting the old one. Even if it's still in memory (I can't tell if it gets removed or not) the code will never be called again.

From there, the discussion mostly seems to be around how to 'stop' the kernel enough to be able to replace the function without resulting in a mess because something was trying to use that function at the same time that you replaced it.

donavanm• 11 years ago

Basically yes. Check out the kprobes docs for a nice description of how these frameworks work ,https://www.kernel.org/doc/Documentation/kprobes.txt. Being able to intercept (and mangle) kernel function calls is awesome. With uprobes the same techniques work in userland as well.

lemonade3• 11 years ago

The original academic paper from MIT describing how it works:

http://www.ksplice.com/doc/ksplice.pdf

bchallenor• 11 years ago

This paper is very readable - thanks for the link.

poims• 11 years ago

And it was officially merged into Linux this evening:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....

AlyssaRowan4• 11 years ago

I am very happy about this. It's been years in coming!

We might finally get live updates on distros. (No thanks to Oracle, of course.)

Wonder if we can also do it that well in userspace? (How does systemd behave with patching, actually?)

lemonade3• 11 years ago

The original KSplice project was actually fully open source, funded by a Dutch charity called NLnet that funds lots of interesting projects like that. I've been running it ever since.

After this was all done the team got acquihired by Oracle. I was actually amazed that the team was allowed to keep the service up for some non-Oracle distro's.

But very happy to see a broader adoption of this kind of technology, it is essential for all these unmanned systems out there in the cloud that they can be patched whilst running.

Live patching for Linux 3.20

Related Stories