Hypernetes: Bringing Security and Multi-Tenancy to Kubernetes

http://blog.kubernetes.io/2016/05/hypernetes-security-and-multi-tenancy-in-kubernetes.html

by scprodigy • 10 years ago

109 29 10 years ago

blog.kubernetes.io

markbnj2• 10 years ago

Can someone clarify how this compares to LXD, released by Canonical in Ubuntu 16.04? A lot of the keywords and concepts seem similar.

mjg592• 10 years ago

They're pretty much the exact opposite of each other. LXD is something that runs things that look like VM images when they're actually running the same kernel - this is creating full (albeit lightweight) VMs to run container images.

The broad approach isn't new (https://clearlinux.org/features/clear-containers describes Intel's Clear Containers work which allows you to use rkt to run containers in individual lightweight VMs), but the trick in this case is to perform the isolation at the pod level (ie, a collection of functionally related containers) rather than the individual container level.

(Edit: Apparently the Docker support for Clear Containers hasn't gone upstream, so dropped the reference to it)

philips4• 10 years ago

The idea is to integrate LXD into OpenStack as a "lightweight VM". So, LXD runs a system with a full init system and executes them inside of a Linux Container. Another term for this is a "system container".

Kubernetes is designed for running Application Container images (e.g. docker, oci, appc) and in this case those containers are running not as Linux containers on a shared Kernel but inside of a VM with Xen/Hypernetes. Another implementation of this "app container in a VM" concept is Clear Containers which is available in rkt[1] and can, like Hyper, run the application container image formats (docker/appc).

This is all a bit confusing so let me know if this helps :)

[1] https://coreos.com/blog/rkt-0.8-with-new-vm-support/

markbnj2• 10 years ago

Awesome guys, that helps a lot. Thanks for the info and links.

resouer3• 10 years ago

And another different side of HyperContainer is that it follows OCI spec, check the runv project here: https://github.com/hyperhq/runv/, so technically speaking, it's a hypervisor version of OCI, just like docker is a linux container version of OCI. Seems rkt/clear linux or LXD does not.

philips4• 10 years ago

The OCI Runtime spec is still a work in progress. Docker and HyperV are implementing parts of the current pre-release versions of the OCI Runtime spec.

You can use OCI Runtime bundles with rkt if you use the oci2aci[1] tool and rkt will be a full OCI runtime once the OCI Runtime and Image spec mature. We can use help to get these OCI projects[2][3] to v1.0 if you can spare cycles!

[1] https://github.com/huawei-openlab/oci2aci

[2] https://github.com/opencontainers/runtime-spec

[3] https://github.com/opencontainers/image-spec

shykes• 10 years ago

Actually it's runc that is the linux container version of OCI. Docker is a higher-level abstraction which calls runc by default but can call any OCI-compliant runtime, including runv.

See https://blog.docker.com/2016/04/docker-engine-1-11-runc/

scprodigy6• 10 years ago

LXD = VM-like Linux container HyperContaineer = (Docker) Container-like ultra-light VM

Make sense?

resouer3• 10 years ago

Aha, the chancellor(https://www.youtube.com/watch?v=PivpCKEiQOQ) will feel great to see that finaly he can eliminate IaaS/vms and use docker in production env.

andrewstuart2• 10 years ago

This really does not appeal to me at all. The major point of docker containers is not the image format, it's that the kernel can allocate resources more intelligently. VM images work just fine for "shippable images."

What I'd rather see is an allocation layer for physical resources that just cordons off the whole machine (physical or virtual) by tenant as soon as previous tenant resources have been fully consumed, then reclaims hosts after usage subsides. So as a provider I still only have one cluster to manage, but as a consumer I still don't worry about another layer of abstraction slowing things down or pre-allocating resources.

chatmasta2• 10 years ago

I'm interested in the economics of (docker) containers vs. virtual machines. Containers can run within a VM, but a VM can only run within a hypervisor.

Currently, if you want to resell computing resources, you need to rent or buy a dedicated server, and run a hypervisor on it.

Containers enable a new class of reselling computing resources. Because you can run a container within a VM, you can resell computing capacity on a VM.

I think we are going to see another abstraction on top of "the cloud," due to this additional layer of reselling (new russian doll on the inside, new doll on the outside).

The physical abstraction is:

Datacenter > Floor Space > Server Rack > Server

The virtual abstraction is:

Server > VM > Container > Kubernetes|MESOS|...

Virtual is a 1:1 inverse of physical. Next step is datacenter interconnections (i.e. multihost kubernetes or whatever flavor of the month IaaS software people use).

derefr• 10 years ago

People have been reselling containers without needing an intermediary VM abstraction forever; look at any cheap VPS host offering OpenVZ-based "virtual machines"—which are actually [resource-quota'ed] containers.

scprodigy6• 10 years ago

So we end up with:

HW -> EC2 scheduler -> VM -> You -> Container scheduler -> Container -> App

There should not be two schedulers if container is secure for multi-tenancy.

jldugger• 10 years ago

At a brief glance, this looks comparable to Magnum:

- containers - openstack - multitenancy

scprodigy6• 10 years ago

Totally not!

sarnowski• 10 years ago

That is what rkt with kvm is essentially doing as well, correct?

https://coreos.com/rkt/docs/latest/running-lkvm-stage1.html

scprodigy6• 10 years ago

No, it is HyperContainer.

https://github.com/hyperhq/hyperd

mjg592• 10 years ago

Not quite - rkt will bring up a new VM for each container, this approach only brings up a VM per pod (ie, a set of functionally related containers).

philips4• 10 years ago

This isn't correct. rkt does a VM per pod.

philips4• 10 years ago

Yes, this is a similar concept. A pod (1 or more containers) is wrapped in a VM. I believe Hyper uses Xen and rkt uses lkvm.

Veratyr• 10 years ago

Speaking of security on Kubernetes, it's worth noting that most of the "Getting Started" guides (e.g. [0]) to help you set up a cluster result in completely unauthenticated API servers.

This means that by default, anyone can do anything they want with your cluster.

There are no warnings, no suggestions that turning on the much better TLS based authentication would be a good idea (or even how to do it), no nothing.

Be very careful with Kubernetes.

[0] http://kubernetes.io/docs/getting-started-guides/ubuntu/

robszumski• 10 years ago

Maintaining community-led documentation is a hard and time consuming process, especially with a young project. I encourage you to get involved if you have a few free cycles.

Sometimes you have to take a stance on these types of things, as we have done with the CoreOS + Kubernetes community guides [0]. The guides are open source, but full TLS, passing conformance tests, etc is required for contribution.

(I work at CoreOS)

[0] https://coreos.com/kubernetes/docs/latest/#installation

praneshp• 10 years ago

The CoreOS documentation is a lifesaver (even when setting up k8s on a non-coreOS system. Thanks a lot (and agree with the comments on documentation/involvement).

xbeta• 10 years ago

Anyone knows whether there's similar thing for Mesos ?

scprodigy6• 10 years ago

A patch is available: https://issues.apache.org/jira/browse/MESOS-3435

switchbak• 10 years ago

They mention that this helps cure some issues with regards to resource sharing / memory usage, etc. But does each VM still have a static allocation of memory?

One of the main benefits I have now is that if I run a number of containers that all take various amounts of memory, I can just throw them on and they share memory amongst each other quite efficiently. If I have to make a static allocation of memory for a VM, I'll typically choose a conservative memory number, and usually under-utilize the machine, wasting a lot of memory per-instance. Not so bad since I chose per-pod, but still an issue.

As it happens, this same issue is why I'm leaning towards lightweight native applications these days instead of an aggressive greedy virtual machine that grabs a bunch of heap. Golang/Rust in particular.

scprodigy6• 10 years ago

Actually, scaleup is pretty easy for both VM and Linux container. But scale-down is very troublesome for both.

And the scheduler will need the mem size (not right now, but inevitable)

Hypernetes: Bringing Security and Multi-Tenancy to Kubernetes

Related Stories