IncludeOS – A minimal, resource efficient unikernel for cloud services

It's built on V8 JavaScript engine and uses event-driven and non-blocking I/O model inspired by Node.js. At the moment KVM is the only supported hypervisor.

http://runtimejs.org/

perbu7• 8 years ago

Lua and Micropython have been ported - mostly as PoC. I expect us to do some proper ports next years, likely Node and Python. C support is OK, but it should be fairly simple to expand it as MUSL provides IncludeOS with the relevant bits of POSIX as long as IncludeOS can provide the underlying syscalls.

sgt• 8 years ago

And how about Rust?

kunthar2• 8 years ago

finally the end of lx* containers on the way. not the actual end of course, who knows docker guys could buy this one as they did to some other unikernel companies.

tamentis• 8 years ago

What other unikernel companies did they buy?

kuwze3• 8 years ago

Unikernel Systems, which is cool because it is OCaml based. Their stuff is open source too:

https://mirage.io

jmspring• 8 years ago

Jails and Zones did their jobs before LXC was a thing. The better technology doesn’t always win out — see Windows history.

Thaxll• 8 years ago

Jails and Zones are inferior to Docker overall, the advantage of Docker is not the isolation technology but the whole workflow of using those containers.

ckocagil• 8 years ago

I don't understand all the hatred towards containers. It's simply an OS abstraction to facilitate isolation. What's wrong with that? In fact I'd love to see more containers, especially on the Linux desktop: for example a container aware compositor that could render windows from container processes and color code their borders, like Qubes does.

I have nothing against VMs either. But even when they're tiny (as in the case of unikernels), do VMs scale as much as processes? I'm skeptical. Maybe someone has data.

en4bz• 8 years ago

People have turned containers in to a packaging, deployment, and orchestration tool rather than the original intent of lightweight virtualization.

Containers kinda suck for those 3 things but no other tool does all 3. It's a jack of all trades master of none situation.

I hate the trend of "download this container cause packaging a .deb is hard" etc.

0xFACEFEED• 8 years ago

I don't understand all the hatred towards containers. It's simply an OS abstraction to facilitate isolation.

Containers on their own aren't enough to deploy a reliable production system. Even on a small scale. To use them for reals you need a lot of domain knowledge and the support of a large ecosystem like Docker. And even then it's easy to get stuck in a rats nest of brittle tooling.

I don't have anything against containers. I'm just tired the exaggerated "ease of use" claims (eg: I talk about container management being a non-trivial problem and people telling me "just user Kubernetes!").

tannhaeuser4• 8 years ago

Docker-style containers only shield you from mixed-library situations eg. where one "service" needs version A and another needs version B. It's painful because this situation is entirely accidental and could be trivially solved by linking everything statically. Then once you have two or more containers up and running, these libraries still need security updates and other fixes which is the entire point of shared libs in the first place. Basically, Docker is not a solution but part of the problem. But with Docker comes a ton of constraints; for example, basic file access and anything related to permissions is a PITA. And you need overreaching orchestration software such as mesos/marathon, k8s, or openshift. It's not that these tools are bad, but rather that they're nuclear weapons for relative little benefit, requiring expensive devops experts. All so you can pack multiple services on a single physical host (which would be much easier using classical service runtimes), and so your HTTP services (because everything has to be HTTP because of the firewalls) can all listen on virtual port 80/8080.

zaarn• 8 years ago

Docker ruined containers. I've been using LXC exclusively on my server and it's much better than Docker, especially since I don't have to retool and it behaves like any other server while still being a container. If all else fails, I can easily bootstrap it into a virtual machine (basically, install the same distro on a virtual machine, tar up the container and dump the contents at the root, adjust networking configuration, done)

ArtWomb• 8 years ago

Worth watching is Justin Cormack's recent presentation on unikernels, LinuxKit, eBPF and databases such as SeaStar:

The Modern Operating System In 2018

https://www.youtube.com/watch?v=dR2FH8z7L04

NuclearFishin• 8 years ago

Really enjoying this talk. Thanks for sharing!

javajosh• 8 years ago

Yeah, that's a really good overview. I think it's interesting how progress in the DevOps field is characterized by removing things more than adding them, and I think it's great. In a way I think that's the highest and hardest aim of computing technology, not deciding what to build, but deciding what to leave out.

ksec• 8 years ago

How could Windows + Linux be 99.9% of the marketshare?

I know not a lot of people are using FreeBSD, but combined with OpenBSD I am pretty sure there are more than 1%.

Edit: Turns out it wasn't a lie.... at least in terms of Web Server usage... FreeBSD has less than 1%.

Sigh.

2trill2spill• 8 years ago

FreeBSD may have less then 1% of server market share but FreeBSD drives a large amount of network traffic. For example FreeBSD runs the open connect CDN that Netflix uses and that accounts for almost 32% of North American downstream traffic [1]. Limelights edge CDN nodes run FreeBSD and Limelight is one of the Bigger CDN companies out there [2]. Juniper routers are FreeBSD based and are the second most used routers after Cisco. There's plenty of other examples but FreeBSD may not be widely deployed but it serves a significant amount of the worlds internet traffic.

[1] http://testinternetspeed.org/blog/half-of-all-internet-traff...

[2] https://www.freebsdfoundation.org/testimonial/limelight-netw...

[3] https://en.wikipedia.org/wiki/Junos_OS

pjmlp2• 8 years ago

Great talk! Thanks for sharing it.

bigbrooklyn• 8 years ago

OSv is another, compatible with some unix apps http://osv.io

ofrzeta• 8 years ago

It seemed interesting back then but it's kind of dead since the core developers (Avi Kivity, Nadav Har'El and others) have moved on to another startup called ScyllaDB that is a re-implementation of Apache Cassandra in C++.

nyh2• 8 years ago

The OSv project is indeed now moving much slower than it used to when we had an entire startup company devoted to it, but is not dead - it still has three committers from three companies - myself (Nadav Har'El), Waldek Kozaczuk and Timmons Player. It still works, and you are welcome to try it. OSv is not as minimal as includeOS, which of course depending on your viewpoint and use case, is either a pro or a con. OSv supports more types of hypervisors, and has support for a larger subset of the Linux ABI, can run pre-compiled, unmodified, Linux shared libraries, and supports multi-core VMs.

a012• 8 years ago

Do these unikernel OSs tend to be language specific? Like if this app is written in x language, use y OS?

chubot• 8 years ago

Yes, because they rely on the compiler (for a specific language) to build and deploy only what the application depends on.

The main examples I know of are Mirage (OCaml compiler) and IncludeOS (C++ compiler).

Though, I think there are experimental / research systems that try to understand a program composed of multiple languages that each compile to LLVM, or anything that compiles to JVM bytecode. So those might not be limited to a single language.

smadge• 8 years ago

Here is a list of Unikernel projects, with their associated languages: http://unikernel.org/projects/

nyh2• 8 years ago

Some do, and some don't. OSv (https://github.com/cloudius-systems/osv), for example, is also a unikernel in the sense that it can only run a single application at a time (it has no kernel-user boundary, nor support for multiple isolated processes). But is designed to run any Linux executables (using a dynamic linker which is part of the kernel) so it is not limited to a particular programming language.

jazzyjackson• 8 years ago

Would love to dig into this and try some small C servers.

FYI if anybody @ IncludeOS is here, the page at http://www.includeos.org/get-started.html automatically made a link out of the file path `./seed/service` as well as `your_service/service.cpp` and 404'd on me, which was confusing for a moment. Looks like putting them in code blocks would prevent it.

emersonrsantos• 8 years ago

Can I use gdb to debug my applications?

fwsgonzo8• 8 years ago

Yes, build for userspace linux and run it as a normal program. Then use all your favorite tools. You can also attach GDB to a qemu process that is waiting for a connection (abit harder). In a production environment on a public cloud it gets even harder. For that I can only recommend UBsan (undefined-sanitizer), logging and memory dumps.

exabrial• 8 years ago

It seems that hypervisors have developed a whole new level of OS API/ABI, sort of like the POSIX specification.

tannhaeuser4• 8 years ago

Except there is no common specification, and there's not going to be one either because "standards are for loosers". No meaningful standard has been developed in this decade. It's almost as if the situation with two or more super-stable, compatible POSIX O/Ss, compiler suites, databases, etc. we had was too good to be true.

perbu7• 8 years ago

IncludeOS can support most of POSIX. It uses Musl, which provides a full POSIX implementation for the system calls that are implemented.

Obviously things like fork() won't work.

pjmlp2• 8 years ago

Well, type 1 hypervisors are the OS.

tinktank4• 8 years ago

I have never really got unikernels, maybe someone could educate me?

- There is the argument of reduced attack surface/typechecked safety. I get that, but I don't see how this is different from compiling custom Linux/NetBSD kernels per VM?

- There is the argument for increased performance. I don't get that at all -- I've done experiments on unikernel system for MirageOS vs IncludeOS vs Kernel on a web and memcache server and the performance results were worse (15%), 0.4% better respectively for MirageOS respectively but for a tonne more effort and really painful debugability.

- There is the argument for isolation but I don't see how it's different because the biggest cause of churn is the hypervisor? My results with Xen 4.x didn't show any difference in isolation performance with both Include and Mirage.

- There is the argument that OSes don't map to existing hardware as well, e.g. in the InfoQ talk in the comments the speaker talks about PCIe VFs for example. All of these VFs map to physical resources, and you end up doing a much worse job when you can't centrally manage these -- that's one of the main reasons SR-IOV hasn't been adopted (including the fact that migration is non-existent). Similarly with CPU hardware threads or NVMe device drivers -- they are abstractions provided for convenience and reducing the burden of porting existing drivers to work in hypervisor environments but surely the same issues that appear with VM contexts (hardware scheduling and isolation) still exist in unikernel environments? I feel I'm missing something in particular about this point as I see it made often and it doesn't make sense to me.

- How do unikernels work in a world of containers? This has always puzzled me as I thought people moved away from VMs due to the heavyweight nature but of course the flip side is the underlying isolation abstractions are more lightweight -- so in essence are unikernels in containers user-level processes or are they replacing the kernel bits too?

fwsgonzo8• 8 years ago

I am a kernel developer working on IncludeOS. Take everything I say with a grain of salt as this is just my thoughts at 9 in the morning in written form.

1. I can build a bootable binary without the full TCP and UDP, shaving off 60% of the whole IP stack. LTO removes a large amount of code as well. It makes all the difference.

2. Performance comes from building a fixed binary (all memory addresses are constant), as well as using LTO and C++. IncludeOS is not at its most performant because we haven't been working on performance. Project is still young. We should be faster overall compared to just about anything that isn't compiling to a fixed binary layout.

3. Isolation can come in many forms. On traditional hypervisors you have the same old same old. While on a newer type of hypervisor you can do just about anything from avoiding VMEXITs altogether to removing the ability to even talk to the hypervisor after configuration stage. If the hypervisor is just a thin configuration program...

4. Yes doubly so if your OS uses threads as well, and the schedulers are almost always beating differently on the hypervisor and the guest. This causes even more trashing than normal. In any case, IncludeOS is not using threads at all. It is fully async and matches the hardware really well. The bottleneck is the multiplexing ability of the cloud platform itself. We can live with that.

5. I don't know much about containers other than that I avoid them. Containers share kernel. Sorry, I can't answer this.

bizzleDawg• 8 years ago

I was reading this thinking that security would be mentioned at some point in your response! Presumably a big benefit of smaller codebases is a more realistic possibility of auditing the security?

chrisper• 8 years ago

Any reasons you are avoiding containers?

progval• 8 years ago

Performance comes from building a fixed binary (all memory addresses are constant),

So, no (K)ASLR?

bullen• 8 years ago

So how can you build a parallel application on top of this that can share all cores of the CPU?

tinktank4• 8 years ago

Thank you very much for your response. Some follow-up questions, in the same order as your responses:

1. So, in production, we do this all the time with the kernel. We have specific .config's that we use on a per-app basis so I'm not sure why this point is made against the kernel? Is it because in general people run what they're provided by the distro? I don't really understand the LTO argument as well, is it that the kernel is smaller? When I compared IncludeOS against the production kernels we run for memcache they were 1.5x bigger. In general, how much of an issue is kernel code size? a 50M kernel, while big pales in comparison to the amount of memory required for runtime operation. Have you done any measurements on runtime costs?

2. Do you have any benchmarks/evidence you could point me to? My benchmarks showed the exact opposite at a huge development cost (debugging unikernels is very painful - at least for me)

3. What do you mean by "same old same old"? The primitives for isolation are provided by the hardware with VT-x and VT-d and your point about "removing the ability to even talk to the hypervisor after configuration stage" is referring to VT-x from what I gather. All modern hypervisors use these primitives and in combination with VT-x and PCIe VFs provide the illusion of complete access to hardware so I think I'm missing your point.

4. I'm sorry but I find the argument that "threads automatically mean worse" hard to understand. The cost of a context switch is the cost of a context switch. The cost of a context switch between threads/processes on the same CPU is still much cheaper than the cost of a context switch between VMs on the same CPU. If a program requires threading then, with the Unikernel either I need an abstraction for threading (e.g. co-operative scheduling) that bypasses the cost of the switch _or_ I need to write the program differently. In your answer, you make the point you don't support threads which means that either I move to an event-driven model (which I can do in Linux too) until I run out of CPU or I switch between VM contexts, which in my understanding is much more costly. Do you mind elaborating further?

5. Do you have an official line on this? I would assume the people you're selling against will be deploying containers so it would be good to get a feel for unikernels on containers.

Thanks again for your response. I found it most helpful.

asien• 8 years ago

This is potentially a big step forward for the cloud, this is also the missing component to push Serverless.

Currently if you want to use containers and Serverless you'll need to spin up 200+MB of linux bistro + tons of dependancies that have nothing to do with what you originally planned, that's to say running a container with a virtual machine on it.

This kernel would allow to replace Unix for a much more lightweight (1MB) and efficient kernel to run programs on it with an hypervisor.

This is promising.

perbu7• 8 years ago

You'd need a small and efficient hypervisor. Ukvm and Solo5 provides such a system and if I remember correctly IncludeOS can boot and shutdown in less than 10ms on those systems.

So yeah, it would be interesting to see this on a serverless platform. Hit me up if you have a serverless platform. :-)

justicezyx• 8 years ago

Had the same impression. I was surprised for a long time there hasn't been any such project under the k8s umbrella.

perbu7• 8 years ago

FWIW the most interesting tidbit I've learned after working with IncludeOS for couple of years is this; The operating system cannot reconfigure itself.

All Unix and Windows-derived systems are meant to have the ability to reconfigure themselves. For a lot of systems there isn't really a good reason why the system itself should have this capability.

My Wifi access point is a good example. The fact that it runs Linux means that a security flaw gives the attacker the tools to fundamentally modify the devices' behavior. The attacker can install sniffers, blockchain miners or spambots, without vendor approval. This makes any RCE bug critical and in general means the device is dangerous to use if the vendor stops patching it.

tinktank4• 8 years ago

Your response confuses me so I must be misunderstanding it. Code is code, all I need is an indirect jump to a pointer somewhere in memory to run the arbitrary code I've installed to compromise the system, so why are unikernels not susceptible to this attack?

nutjob22• 8 years ago

"arbitrary code I've installed"

How do you do that if there is no facility to load and run code? That's what he's getting at, the binary installed has fixed functionality and has no facilities to do anything but what it's programmed to do. It may have no file system nor almost any other feature that an OS has. It likely supports network ports, but they are unlikely to allow arbitrary code execution becuase again, they will have a fixed function.

The only way to do what you're talking about is replacing or modifying the binary, which means you have physical access or there is a serious breakdown in security which nothing could withstand.

close04• 8 years ago

I guess he means that the OS cannot do more than it was intended to do. A software with a very limited set of features and functionality that you can't expand even if you wanted to. Which would mean an attacker would have to get by with only that instead of the "endless possibilities" an OS usually provides.

presscast• 8 years ago

A few silly questions from someone who comes from containers (Docker, specifically).

1. Can I "run" and "manage" unikernels on my development machine like I would a Docker image? (Use case: keeping build and deploy environments identical)

2. What's the workflow for inspecting running containers?

3. Is it possible to work in languages other than C++?

tobiaswk• 8 years ago

Every time I see this project pop up I think about how awesome an idea it is. Not sure about the practical side... but the project is very cool indeed.

jokoon• 8 years ago

I just found about OpenRISC https://fr.wikipedia.org/wiki/OpenRISC

I'm wondering how this is performing against the raspberry SOC chip. I'm not sure what is the most expensive part, the licensing or fabricating the chip.

IncludeOS – A minimal, resource efficient unikernel for cloud services

Related Stories