How We Built SELinux Support for Kubernetes

https://gravitational.com/blog/how-we-built-selinux-support-for-kubernetes-in-gravity-7-dot-0/

by twakefield • 6 years ago

70 32 6 years ago

rwmj2• 6 years ago

I've written a bunch of SELinux policy, also without being especially expert. The tooling is definitely getting better, but still there are many annoying things from the obscure file layout, the giant all-in-one everything-is-a-global policy[1], to the small annoyances like m4 in the policy files.

Of course you need to buy into the whole premise of SELinux - that in addition to simple Unix permissions you can use mandatory access control to fine-grain control everything that a process can do and get access to. Obviously this is a giant boil-the-ocean project that has consumed many good people for decades. For SVirt (secure virtualization) this was pretty successful, allowing many qemu instances to run on a single server, and confining them with SELinux so each can only access its own files even when they are all running as the same Unix user.

The linked article is a good intro.

[1] But they're fixing this, yay! I can't find a link now but there's a project to decentralise SELinux policy. Perhaps it's still an internal-only Red Hat project.

als0• 6 years ago

[1] But they're fixing this, yay! I can't find a link now but there's a project to decentralise SELinux policy. Perhaps it's still an internal-only Red Hat project.

+1 this is sorely needed and I wasn't sure if the design would ever evolve to include this

yrro2• 6 years ago

Worth mentioning that SVirt is also used for running containers on RHEL/Fedora/CentOS etc.

totony3• 6 years ago

I don't think that's the case, svirt is specific to libvirt + selinux-virt, whereas containers use selinux-container which are integrated into docker & al. Same principle tho

kapilvt2• 6 years ago

Tools write selinux policies, humans write apparmor policies .

VWWHFSfQ2• 6 years ago

apparmor is dead at this point right. selinux is the standard

nisa• 6 years ago

could you explain why you think so? We are are Debian/Ubuntu shop and it looks like it's all AppArmor.

LinuxBender2• 6 years ago

Both tools can be managed by both tools and humans. SELinux is for Redhat distros and operates on inodes. AppArmor is for the Ubuntu distros and operates on files/dirs. Each have pros and cons, too long to cover here at HN.

marbu• 6 years ago

While the article shows how they build SELinux policy from scrath, they also mention udica tool[1] which generates SELinux profile based on container inspection:

# podman inspect my_container_id | udica my_container

I wonder how much would the result be different in their case or how much time would they save by having the first version of the policy being generated this way instead of writing it from scratch.

[1] https://github.com/containers/udica

lima• 6 years ago

What's the point of SELinux support in a k8s environment, where services are already sandboxed by the container runtime?

It's unsafe to run untrusted workloads in the host kernel using only namespaces - SELinux does not change this, no matter how much Red Hat wants it to be true. The kernel has a massive attack surface. For untrusted workloads, you need virtualization like gVisor or Kata/Firecracker which isolates workloads from the host kernel.

SELinux can mitigate some logic bugs in the container runtime, yes, but it doesn't make your containers secure.

Also, the title of this submission on HN is inaccurate - Gravitational did not build SELinux support for k8s. Red Hat did, a long time ago. They added support for SELinux to their own product, Gravity.

blitmap• 6 years ago

My naive understanding is that containers do not provide complete security.

Also, I could see giving a user shell access within a container and still restricting what they do with SELinux.

VWWHFSfQ2• 6 years ago

Defense in depth. Running untrusted code in a kubernetes cluster is risky even with the sandbox. SElinux adds another layer of protection

Spivak2• 6 years ago

I would go further and say that SELinux (and AppArmor) aren't defense in depth but actually a necessary part of the tooling to implement sandboxing of containers.

cpuguy83• 6 years ago

SELinux would have prevented a number of container breakout issues.

That said, as someone one time worked on an SELinux issue in Docker and then became the defacto "expert"... it's a terrible mess, often coming with incompatible policy changes tied to kernel changes and policies which are stacked on top of each other like a house of cards that are difficult to tease apart for other distros.

Spivak2• 6 years ago

Because MAC (ie. SELinux and AppArmor) is part of the tooling the container runtime needs to sandbox services. Without SELinux you end up with the awkward situation where services are running as real users on your system and there's no real way to confine them beyond that.

robszumski• 6 years ago

OpenShift is the only Kubernetes distribution with SELinux in enforcing mode out of the box (that I know of) and this has stopped attacks in the past.

markandrewj• 6 years ago

The point is that containers are actually just running PIDS, if you were to escape the process, you would have access to the host without security measures like selinux. You can also use selinux stop things like interprocess communication.

Something I notice in regards to selinux, is many people don't understand that you are not just applying context to files, but also PID's. You can see the context of a PID by doing 'ps -eZ', similar to how you would with files. This allows you to create a finer degree of process seperation.

You are right though, RedHat has been doing this with OpenShift since 2011, and it is a major part of their security model. RedHat's current push is the deployment of rootless containers, which addresses similar concerncs.

tedk-42• 6 years ago

SELinux was a good way of securing linux machines with many services sharing things like file systems and binaries.

k8s solves this in a different way. You can write applications that are completely stateless when spun up on a k8s node and can be limited to a read only file system. Your container can be a scratch container with no other executables binaries.

I don't know what their product is, but i seriously doubt mixing these two concepts is the right thing to do.

rwmj2• 6 years ago

I think you need to look at how it really works under the hood if you believe that. RHEL runs containers with a few layers around them, one of which is SELinux (example of an exploit being blocked: https://www.redhat.com/en/blog/latest-container-exploit-runc...)

kevin_nisbet2• 6 years ago

I'm not sure it's fair to say the advantages of selinux are solely around sharing services and filesystems on a host. I believe selinux also allows a more expressive policy on how a running piece of software can interact with the kernel than say seccomp can, which is probably what you're container runtime is using to limit kernel access. And unless you're configuring seccomp, you're probably picking up some default policy designed to work on as many systems as possible, which sort of violated least privilege.

I'm also not sure I would consider this a totally solved problem within kubernetes. In supporting gravity, our kubernetes distribution, we used to hard-disable all privileged containers. This become a common point of friction, since many pieces of common kubernetes software seemed to just use privileged containers instead of properly listing and setting needed capabilities. Although this is a separate issue from selinux, I'm just pointing out that many kubernetes clusters may not be as isolated between running processes as expected.

And to answer what the product is, gravity is a kubernetes distribution, that in this use case allows for packaging a kubernetes based application together with the runtime, for a unified installer for the application + runtime. Think something like packaging up a SAAS application, and being able to install a copy within a banks network who doesn't want their systems talking to the internet while abstracting away the kubernetes portions.

The problem we were running into, is that end user would keep trying to install the application onto selinux enabled hosts, which would break the install or runtime. And whether you believe in selinux or not, it causes significant friction to try and tell your customers we got this, just turn off that security feature in production.

pm90• 6 years ago

Given the additional context “we need out k8s deployment to work for customers who use selinux on their nodes”, the motivation to do this makes more sense. The majority of k8s users likely don’t share their clusters with untrusted/low trust users so it seems unlikely to help them much.

Not to dunk on the work you’ve done: it’s still awesome. Just want to point out the disconnect from users wondering how this would help them.

jonwalch• 6 years ago

Their product is (or at least was, when I used it back in late 2018) bootstrapping a K8S cluster on bare metal with "one command". Obviously this is an extremely aspirational claim.

ape4• 6 years ago

NSA made SELinux. My theory is that they made it so hard to use so nobody would use it /s

LinuxBender2• 6 years ago

Redhat helped the NSA make SELinux substantially easier for more people to use. Disclaimer: It's still not super easy to use. The implementation at the NSA did not have the concept of "unconfined" anything and they use MLS policies. At least, that was the intent. This may have changed with time and perhaps by now they are using Redhat's version of SELinux.

heywire• 6 years ago

An off-topic but somewhat related question. What is a good way to whitelist which applications can access the network on my desktop linux system? (Currently running Arch, but I distro-hop quite often). Years ago when I had a MacBook, I used a program called Little Snitch. I see that there is a similar application for Linux called OpenSnitch, but I’m hoping there is a better way to utilize the built in features of Linux rather than introducing a 3rd party kernel module. Are SELinux / AppArmor the answer? Cgroups? Something else?

totony3• 6 years ago

Selinux can add labels to network packets which can be matched with nftables & al. Its a bit complicated tho. Since you need to add and create labels.

You can also match by owner in iptables. So you can have a network restricted user and sudo to it.

How We Built SELinux Support for Kubernetes

Related Stories