Unikernels

by seeker89 • 4 years ago

129 64 4 years ago

seeker894• 4 years ago

It looks like around 2015 there was a lot of hype around the topic, but as I tried to document the state of the art, I noticed there hasn't been that much pick up yet. What's your take?

gjvc• 4 years ago

I suspect it was because containers sufficed and using and creating the tooling around them consumed the attention of those who might have otherwise looked at unikernels.

walterbell• 4 years ago

Some marketing material on docker vs unikernels, https://nanovms.com/learn/docker-vs-unikernels

seeker894• 4 years ago

I suspect it was because containers sufficed and using and creating the tooling around them consumed the attention of those who might have otherwise looked at unikernels.

I think you might be right. Right place at the right time for containers.

gwd• 4 years ago

The two main advantages of unikernels are performance (reduced CPU and/or memory requirements) and security (hypervisor rather than container boundary).

It turns out that basically nobody cares about either of those. I know someone who for a while worked at a company that was trying to make money off unikernels. They ended up re-implementing a database in a unikernel, in a way that gave them a 2.5x performance improvement over the original project -- or to put it a different way, switching should allow companies to cut their hosting costs in half. Even with such a clear "win", it was still a difficult sell.

Think about how much backend code is written in interpreted languages like Python, or PHP, or Javascript, rather than in compiled languages like Go or Rust. It's just simpler to start with the simple solution and then throw money at the problem as you scale. And while performance may be one of the reasons that people are choosing Go or Rust for backends, if it were the only advantage, it's unlikely that would be compelling enough.

bluepizza• 4 years ago

Because they are a silly idea. Why would you spin up a full kernel to run a single application on top of a hypervisor that is balancing resources?

Operating systems already solve this problem relatively well, without the overhead, via processes and containers.

igorkraw• 4 years ago

What is the functional difference of os=>hypervisor=>unikernel vm vs. os=>capabilities and pledges or containers? I would get if we use a unikernel approach running on bare metal for high security, specialised applications but this doesn't seem to exist?

eyberg10• 4 years ago

The difference is that the vast majority of people are deploying to the cloud so they are already deploying to a hypervisor. Every single cloud is built on top of virtualization. AWS used to use Xen, now they use KVM. Google Cloud is entirely built on KVM. Azure uses Hyper-V. The cloud is just an API for virtualization.

Instead of AWS (hypervisor) => linux => k8s => containers unikernels advocate for AWS (hypervisor) => unikernel and that makes them run much faster in general (we've clocked upwards of 300% req/sec for go/rust webservers on AWS for instance) and a lot safer.

jasode• 4 years ago

Why would you spin up a full kernel to run a single application

You're misunderstanding the levels of abstraction probably because the word "kernel" within "unikernel" is throwing you off. The idea is to use a partial kernel (only the minimum services one needs). The so-called "kernel" is a library of code where you compile the minimum bits into the single-process image.

Operating systems already solve this problem relatively well, without the overhead, via processes

A full operating system like Linux is expending extra overhead to schedule/prioritize/monitor processes (plural) -- because Linux is designed to be more general purpose and open-ended than a specialized unikernel. In contrast, a unikernel with only 1 singular process (say a specialized db engine) doesn't need to expend extra cpu on processe(s) scheduling.

All that said, it doesn't seem like unikernels have enough advantages to attract widespread adoption like containers.

hypertele-Xii• 4 years ago

So it's less of an operating system and more of a single app that runs on metal?

eyberg10• 4 years ago

This is very very common misconception. Just yesterday I was helping someone out with a networking issue they were having on AWS stemming from this concept.

Coming from k8s/firecracker it is common to think that you need to orchestrate your unikernels with a framework of some kind. In our case (Nanos/OPS) a lot of people think that means spinning up an ec2 linux, sshing in and using 'ops run' on top of that but that is never suggested for prod deploys. Instead we suggest doing an 'image create' followed by an 'instance create'.

What does this mean? Essentially every time you hit the deploy button a new ami is made and a brand new ec2 instance spins up without any linux inside. So instead of adding layers through containers we actually subtract them. That means you can still configure the instance to your hearts content but you don't have to manage it - the cloud does for you and this is a huge win for many teams that don't want to deal with all the ops/SRE work that something like k8s brings (or even normal vanilla linux does).

It is important to realize that containers extract heavy performance penalties when running on top of existing infrastructure (like the cloud) since they duplicate storage and networking layers. They also have severe security issues - the shared kernel being the main one.

einpoklum• 4 years ago

Why would you spin up a full kernel to run a single application on top of a hypervisor that is balancing resources?

What if your application involves multiple processes and threads?

eyberg10• 4 years ago

Nanos supports multiple threads but not multiple processes so you can have as much performance as you have underlying hardware but if you are using something like an interpreted language where is normal to spin up X app-workers behind a reverse proxy those become vms. (I should point out that those languages are single-thread/single-process to begin with.)

amouat• 4 years ago

It's worth looking at https://kontain.app/ - seems to address a lot of the mentioned pain points in the article.

kalmi10• 4 years ago

Took some clicking around to find a good description of what this actually is: https://kontainapp.github.io/guide/overview/#a-new-approach-...

kitd• 4 years ago

I think micro VMs solved a lot of the issues that people had with regular containers and that unikernels were going to fix. There is still probably a performance improvement to be had with unikernels, but not enough to throw away all the investment companies made into containers.

That said, there are options for running unikernels as K8s workloads if you want, eg NanoVMs: https://docs.ops.city/ops/k8s

goodpoint4• 4 years ago

Counterpoint: the absence of logging, monitoring, host-based IDS, and all the system engineering tools you have on Linux is a big negative for security.

wyuenho• 4 years ago

You can compile in these metric collection facilities into the unikernel if you need them. The whole point of unikernel is to allow you to mix and match only the things you need.

dimitar• 4 years ago

You can log and send metrics from your app over a network (which you should probably be doing instead of writing on disk), monitor a VM, IDS makes no sense when you don't have logins and such. And the tools are rarely installed on "cattle" anyway.

goodpoint4• 4 years ago

You can log and send metrics from your app over a network

That's far from enough. It's conceptually and practically wrong to rely on the application to monitor itself.

IDS makes no sense when you don't have logins and such

On the contrary, there is plenty that IDS can do for webapps.

goodpoint4• 4 years ago

IDS makes no sense when you don't have logins and such

citation needed

And the tools are rarely installed on "cattle" anyway

citation needed

kristianpaul• 4 years ago

You have Alpine Linux container image that is around 2 Mb in size.

And its a shame AWS doesn't allow volumes with sizes smaller than 1 GiB that i understand you can get really small images with Unikernels

eyberg10• 4 years ago

There are benefits to this if you are deploying something like a 5G microservice or some other single-purpose, small binary that takes few resources but at the end of the day if you are deploying a JVM application that is gigabytes in size it doesn't really matter how small the base image is. Same thing applies in unikernel-land.

The 1 gig limitation on clouds is not such a huge deal though as we can upload your image, however small it is as that size and then tell the cloud to provision the disk the size you need which acts kind of like a sparse file (but technically not one).

marmarama4• 4 years ago

https://github.com/google/gvisor gives you essentially the same benefits as a unikernel without having to compromise on compatibility or recompile your apps, and integrates nicely with Kubernetes already. It also doesn't require a hypervisor at all.

bitcharmer• 4 years ago

Since this looks like an intermediary layer between userspace and the host kernel (at least if I'm reading it correctly), does anyone know what its performance impact is?

marmarama4• 4 years ago

The gVisor documentation has performance comparisons vs. cgroup-style 'traditional' containers at https://gvisor.dev/docs/architecture_guide/performance/

There is definitely some performance overhead, but in most cases it is less than hypervisor-based approaches.

zamadatix3• 4 years ago

I always thought of gVisor as being in the opposite direction of the main point of unikernels in that it's another layer between the app and the kernel rather than removing the separation completely.

marmarama4• 4 years ago

Unikernels never really removed the layer between the app and the kernel, they just made the hypervisor the kernel and invented a layer to handle IO to/from the virtual devices presented by the hypervisor, inside the same memory space as the app.

If the hypervisor is KVM, which they are if running on modern AWS EC2 instances or GCP, unikernel apps are literally just Linux processes; the underlying Linux host is doing all the heavy lifting. Conceptually, they're essentially the same as a sandboxed ordinary Linux process with an in-process IO stack, but without the ability to monitor or debug them as if they were an ordinary Linux process.

seeker894• 4 years ago

The lightweight-ness argument is enticing, but I'm wondering if the fact that now you have to give these VMs enough RAM to run the app won't mean that you end up with worse flatpacking in terms of RAM than if you used containers?

eyberg10• 4 years ago

It depends. A t2.nano has 512mb of memory. If you are using Go or something like that you could go much lower but any runtime environment such as ruby/python/node are going to want a minimum of a few hundred meg. If you are using the JVM you most definitely are going to want an instance with much more memory.

At the end of the day your application decides how much memory it wants and the sysadmin/SRE/devops person just ensures it has enough so it doesn't crash.

If you are hosting your own workloads and those workloads only need tens of megs of ram than you can pack as much as your hypervisor can handle.

Alfred talks about booting 110,000 vms on one host before memory exhaustion:

https://ieeexplore.ieee.org/document/6753801

convolvatron2• 4 years ago

kinda. if the guest uses something like virtio balloon the sharing isn't any worse. handling exhaustion isn't any better - although I think transparent migration of vms is more of a standard function than for containers, so that's some up I guess.

seeker894• 4 years ago

Also, some interesting perspectives on Linux unikernels: https://www.bu.edu/rhcollab/files/2019/04/unikernel.pdf

api2• 4 years ago

I want a unikernel that runs as a process with no special privileges. Huge bonus points if its portable to many common operating systems.

Since recompilation is necessary anyway for unikernels, syscalls could be replaced by function calls or some other user mode thing instead of trapped. It would allow entire containers to run as processes. Not that interesting for cloud, but very interesting for distribution to endpoints or self-hostable apps.

convolvatron2• 4 years ago

like user mode linux?

api2• 4 years ago

Yes, but like portable UML. I've seen a few people try to port UML to other OSes but nothing is mature.

mwcampbell• 4 years ago

Honest question: How is this different from an ordinary program written in a portable language with minimal reliance on shared libraries?

fwsgonzo• 4 years ago

There seems to be a lot of guessing by the author. VMs that run on the CPU have the same performance as everyone else, except when you trap into the hypervisor, and especially on buggy CPUs. Now, there are a lot of buggy CPUs out there right now, so I won't say any more. But just imagine a world where we have a fully working io_uring (all system calls that make sense), and less buggy CPUs.

UltraViolence2• 4 years ago

Microkernels are a much more viable way of solving security problems in an operating system. Windows and Linux could both be rewritten as microkernels within a couple of months or years.

muricula• 4 years ago

Windows and Darwin (MacOS) were originally designed to be hybrid kernels, but compromised by allowing more and more stuff into kernel space until they were the monolithic kernels we know today. Changing code built up over 20-30 years while maintaining compatibility, security, and performance guarantees is not something which could be accomplished in a couple of months.

eyberg10• 4 years ago

The Hurd is actually older than Linux now at a ripe age of 32, although I think it was Mendel Rosenblaum that said "hypervisors/machine monitors were microkernels done right".

https://www.usenix.org/legacy/events/hotos05/final_papers/fu...

lazyier• 4 years ago

Early versions of Windows NT WAS a 100% honest microkernel OS. Microsoft abandoned that approach when they realized they had zero chance of being competitive with Unix with a microkernel architecture.

Darwin was never intended to be anything except what it is, which is a monolithic kernel. The XNU kernel was based on FreeBSD kernel and Mach kernels. Some versions of the Mach kernel were microkernel, but many were not.

Both NT and XNU incorporate message passing features from microkernels, but they are monolithic in that they are essentially a single large process.

"Hybrid kernel" is more of a marketing thing than an engineering term.

Microkernels are a dead-end and never stopped being a dead end. It's a lovely idea that didn't work out. They had limited commercial success in embedded systems, but only because those embedded systems didn't actually do very much and what they did was largely not performance critical.

fsflover• 4 years ago

Another approach is to use virtualization and there will be no need to rewrite anything. See: https://qubes-os.org.

Unikernels

Related Stories