Docker 0-Day Stopped Cold by SELinux

http://rhelblog.redhat.com/2017/01/13/docker-0-day-stopped-cold-by-selinux

by jwildeboer • 10 years ago

275 158 10 years ago

rhelblog.redhat.com

sofaofthedamned• 10 years ago

Does AppArmor also block this I wonder?

placchetti• 10 years ago

Depends on how well it's configured. My guess is that an out-of-the-box profile probably wouldn't.

w8rbt3• 10 years ago

I think it could.

Here's a write-up showing how AppArmor can protect Docker containers and the underlying host... quote from the article, "So without even patching the container we have prevented rouge pid from spawning using a correct security profile with AppArmor."

    https://scottydoesntknow.io/container-secure-right/

wlamartin5• 10 years ago

AFAIK as I know, the answer is yes. In fact if you look at the Docker apparmor documentation you can see an example where ptrace was blocked https://docs.docker.com/engine/security/apparmor/

wlamartin5• 10 years ago

I just want to update this to clarify that it blocks ptrace, but this is only part of the issue and you shouldn't rely on AppArmor to mitigate this CVE entirely.

ajross• 10 years ago

AppArmor and SELinux share the same set of LSM hooks underneath, so there's comparatively little difference in capability. The big fight is over the philosophy of how they are configured (very broadly: AppArmor tends to tolerate ad hoc configuration by doing what you want instead of what you say, and SELinux like to break everything it doesn't understand perfectly).

I really wish SELinux would evolve some better configuration tools/culture, but after a decade and a half I despair of this ever happening. Every single Fedora release I leave it enabled on my personal machine thinking "THIS will be the moment where I really puzzle it out". Then I disable it a week later when I realize how much annoying work configuring rules for my rando backup and device management scripting will be, and when I see that it still lacks rules for a bunch of in-distro tools I need to use.

brians• 10 years ago

Last I looked, Apparmor still applied rules based on file names, not on types assigned to the underlying objects. That's a big limit on writing correct policies: every hardlink is like a firewall-spanning device.

peterwwillis• 10 years ago

AppArmor is a DAC+MAC and SELinux is an RBAC+MAC. AppArmor is not nearly as powerful. Grsec is much more effective than AppArmor at preventing a wide range of attacks, but SELinux is more or less king.

jacques_chester2• 10 years ago

My understanding is: yes, it is blocked by AppArmor.

Based on seeing email chatter about this report in the context of Cloud Foundry[0]. Under the hood CF uses runC, partly to allow AppArmor to be applied.

Disclosure: I work for Pivotal, the majority contributor of engineering to Cloud Foundry.

[0] https://www.cloudfoundry.org/cve-2016-9962/

justincormack4• 10 years ago

I would recommend not giving that advice, and recommend that all your customers upgrade. There are a number of ways to exploit this, and as stated elsewhere on this thread Red Hat's advice that SeLinux entirely mitigates this is not correct, it is highly likely that it can be exploited on Cloud Foundry too. I would never advise people to not upgrade software when there is a CVE because someone says there is a mitigation, it is very high risk.

Disclosure: I worked on testing the fixes for this CVE.

CrLf• 10 years ago

SELinux used to be one of those things you'd disable immediately upon installing a new RHEL/CentOS box for all the troubles it would cause. But default policies have evolved a lot, making this the wrong thing to do, for a few years now. But people still do it out of habit.

If you ignore SELinux, it won't cause issues besides the ocasional need to run "restorecon" (which one gets into the habit of doing whenever an "access denied" error happens when permissions seem otherwise correct).

But one problem still remains. SELinux is (very) complex and people (myself included) have a very hard time groking its base concepts. This limits adoption greatly, and I'm still to find a decent document that starts from the simple stuff and lets one build a mental concept of how it works before jumping into the more complicated (real-world) use cases.

w8rbt3• 10 years ago

That's the issue with LSMs in general. They all work, but historically have been difficult to configure and impossible to maintain. That's changing.

There used to be only one (SELinux), however, there's competition now from other LSMs. Smack, AppArmor, Tomoyo, etc. In part, that's why SELinux is improving.

I've tried them all and settled on Tomoyo. The documentation is outstanding and it is (to me at least) the easiest LSM to reason about and configure.

http://tomoyo.osdn.jp/

jwildeboer6• 10 years ago

How up2date is tomoyo, though? The Changelog and updates on the front page seem to indicate that not much happened after 2015.

djsumdog2• 10 years ago

I've had to create apparmour profiles in the past. There is some good tooling, but it's still not trivial.

nailer2• 10 years ago

I worked at Red Hat when SELinux was made mandatory. Every single customer was hit with 'avc denied'. Having to explain that avc was 'Access Vector Cache' and that 'Access Vector Cache' was part of SELinux, and trying to convince somebody to patch the messages to just say 'selinux denied' was nightmarish.

That said: SELinux is one of the only things that can make shared Docker hosting (ie, where the containers are actually isolated from the host and each other) possible.

nailer2• 10 years ago

Correcting self: 'made mandatory' -> 'enabled by default'

gtirloni• 10 years ago

SELinux is (very) complex and people (myself included) have a very hard time groking its base concepts.

I wonder what a clean slate OS design would look like. One that satisfied the same requirements without any concerns about backward compatibility with POSIX history.

Does anyone know an OS with vaguely this goal?

nicoburns• 10 years ago

Microsoft's Midori project. It never saw the light of day, but there are some very interesting blog posts about it. The Redox project has some leanings in this direction.

thegeomaster• 10 years ago

How about Magenta [0] by Google, on top of Little Kernel (LK), a neat and modern microkernel design? (It's part of Google's work-in-progress complete OS named Fuchsia, which appeared briefly a while ago on tech news sites.)

[0] https://fuchsia.googlesource.com/magenta

nickpsecurity• 10 years ago

Try the first two to be certified to high-assurance security:

http://www.cse.psu.edu/~trj1/cse443-s12/docs/ch6.pdf

The MLS model was too difficult to adapt to commercial use. Biba was good for stopping malware from overwriting files. They still preferred something more flexible. SCC then invented type enforcement in another high-assurance system:

https://web.archive.org/web/20160311233659/http://www.cyberd...

Flask architecture was combining that tech with a microkernel. SCC, acquired by McAfee, added type enforcement to a BSD OS for their Sidewinder firewall. The next work by Mitre was proof-of-concept for OSS by adding it to Linux. That and a pile of incremental additions is called SELinux. I'm sure you'll find the LOCK design a lot cleaner as it was originally intended. ;)

Also worth noting are the KeyKOS system (esp with KeySAFE), the capability-security machines, and one language-based mechanism:

http://www.cis.upenn.edu/~KeyKOS/

http://www.cs.washington.edu/homes/levy/capabook/index.html

http://www4.cs.fau.de/Projects/JX/

These collectively should keep you busy for a while. They're the kind of thing worth imitating or building on.

rch• 10 years ago

Aside from well known options like Plan 9 and QNX:

- https://pdos.csail.mit.edu/archive/exo/

- https://atheos.syllable.org/

feld• 10 years ago

Yes, Trusted UNIX where you have a very complex MAC framework

binarycrusader• 10 years ago

I'm probably biased, but Solaris' role-based access control system has been very successful among its particular customer base:

https://docs.oracle.com/cd/E23824_01/html/821-1456/rbac-1.ht...

It wasn't necessary to throw away POSIX history or concerns either.

deckard1• 10 years ago

Not sure if you're only considering Unix/BSD-ish OSes here or not. But there are perhaps hundreds of such projects.

http://tunes.org/cliki/operating_20systems.html

You may want to dig around that entire site to get an idea of what people have tried to do (and frequently failed).

mastax• 10 years ago

Robigalia is interesting but not nearly ready: https://gitlab.com/robigalia (their website has a cert error right now. It seems like I've seen a lot of those these days)

It's a rust userland built upon SEL4. SEL4 is very simplified in order to meet their verification goals so robigalia has to implement some interesting resource sharing primitives on top of it to get things to work. It could be interesting.

s_kilk3• 10 years ago

The thing that kills it for me is that Fedora ships with broken default configuration.

Install Fedora, reboot and log in. Chances are within minutes you'll start seeing "selinux denied" messages popping up, complaining about services, files and policies you've never heard of. How is anyone but a seasoned RHEL admin supposed to know what to do with that?

cozzyd2• 10 years ago

This might have been true many years ago, but it certainly isn't true nowadays (at least... not on the Workstation spin, can't vouch for others).

seanwilson• 10 years ago

Is there a good reason why it's hard to make SELinux easy to use?

lisivka4• 10 years ago

Because parts of SELinux which are easy to use are already covered by POSIX.

reidrac• 10 years ago

You're absolutely right. The problem is that it really doesn't matter if it is amazing an works fine out of the box: people won't use it because SELinux is generally perceived as a technology that is too complicated and will break things if you don't disable it.

At this point, very few people will bother to learn the few bits you need to know to troubleshoot and fix any issues you might encounter; and any benefits you get from using it are not worth the effort.

I used Fedora from 12 to 21 (? I think) and always left SELinux enabled, and it just works. The few things that failed (I remember two issues, one with an experimental build of Chromium and another one with OpenVPN and certificates in $HOME), I submitted a bug report and created my own rule to workaround the issue.

If I managed to run a desktop system with SELinux on, it should be possible (and potentially easier) to use it properly on a server.

mhurron• 10 years ago

people won't use it because SELinux is generally perceived as a technology that is too complicated

It's amazing how bad things have got with the majority of developers and admins that they refuse to learn things that are difficult and instead simply turn it off. It's not like all of them are too young to remember when nothing in their system was easy and they actually had to learn about what they were doing.

pmoriarty• 10 years ago

"I used Fedora from 12 to 21 (? I think) and always left SELinux enabled, and it just works."

I've had all sorts of weird errors over the years when I set up new CentOS machines.. one thing or the other. And then I remember to turn off SELinux, and it suddenly works. Over and over again.

It's going to take a lot of convincing to get me to leave SELinux on when it's caused so many problems over the years.

noinsight• 10 years ago

I'm still to find a decent document that starts from the simple stuff and lets one build a mental concept of how it works before jumping into the more complicated (real-world) use cases.

Well, there is this:

https://people.redhat.com/duffy/selinux/selinux-coloring-boo...

You can link it to repeat offenders who disable SELinux. (That might not be a good idea.)

reitanqild2• 10 years ago

Read that a while ago and it didn't make it click for me.

If I had just stumbled across it without a recommendation and without recognising the bame of the author I'd easly dismissed it as some kind of trolling but thats maybe just me.

What I want is:

1. How to do tasks x, y and z (with explanations on why).

2. Complete documentation of all commands, settings files etc.

Usually 2. is somewhat covered in the official docs and 1. is available in blog posts etc. Last I checked 1 was not covered in any way when it comes to SELinux.

Making a coloring book out of it is nowhere on my list.

hobarrera3• 10 years ago

That sounds pretty cool, but where do rules come from?

Suppose some author wrote some daemon. Is the packager responsible for writing the rules? It sounds like having packagers understand SELinux rules is a lot of responsibility, and if upstream is cross-platform, they might not care about such specific needs so as to provide it.

Also, what happens if I write some small app? Do I need to write its rules? If it has no rules applied to it, then it's basically game over, because SELinux sounds like it works ONLY if it applies to all processes.

toredash2• 10 years ago

It doesn't help that cloud providers, e.g. DigitalOcean disable Selinux

_jcwu• 10 years ago

When I asked them about this and why they do not even tell the customer about this, they said it is so that they can reset the password when requested through the control panel.

hobarrera3• 10 years ago

But people still do it out of habit.

It's very hard to come back from such a popular image of "100% broken and must be disabled immediately". Even if SELinux evolves to absolute perfection, the damage to its image is done, and that will take a long time to change, regrettably. It should not have been shipped so green.

If you ignore SELinux, it won't cause issues besides the ocasional need to run "restorecon" (which one gets into the habit of doing whenever an "access denied" error happens when permissions seem otherwise correct).

Sound like it's still pretty broken, IMHO. I should never see an "access denied" error on a host I control, unless I misconfigured it.

The truth is, the defaults MUST work on all common scenarios all of the time for these things to be successful. Otherwise, people will only see the downsides (and the upsides are rarely visible, and rarely outweigh the downsides).

notriddle2• 10 years ago

I should never see an "access denied" error on a host I control, unless I misconfigured it.

What do you mean? Even stock UNIX will give a permission denied error if you try to run an executable without `chmod +x`-ing it or `rm -rf /boot` as a regular user.

cesarb• 10 years ago

If you ignore SELinux, it won't cause issues besides the ocasional need to run "restorecon" (which one gets into the habit of doing whenever an "access denied" error happens when permissions seem otherwise correct).

You also have to put things in the correct place. For instance, your VPN certificates should be in $HOME/.cert so restorecon knows they should have the home_cert_t label.

giis• 10 years ago

SELinux is one of our main protection against user abuse. Our project (webminal.org) provides free terminal access anyone. Thus we need protection. Behind the scenes we rely on things like SELinux/Quota/Pam.d/limits.conf/rootkits etc But SELinux has a learning curve than others, but its worth a ton.

jlhawn• 10 years ago

Since when is a security issue which is known to and patched by the vendor a "0-day"? Have there been any reported exploitations of this vulnerability in the wild before it was publicly disclosed with a patch made immediately available?

justincormack4• 10 years ago

No, there were not, and it was disclosed to related vendors some weeks earlier. Some people seem to thing "0 day" is a l33t term for "vulnerability" it seems.

jeremyjh• 10 years ago

I think all they are suggesting is that running SELinux can protect you from vulnerabilities that no one knows about yet. Obviously they can't give an example of an vulnerability that is still a 0 day.

notriddle2• 10 years ago

I think they mean "SELinux was stopping this vulnerability from being exploited before it was even discovered."

hobarrera3• 10 years ago

I'm not sure if the situation has changed, but I recall installing Fedora (and another distro I cannot recall) years ago, and SELinux would kill sshd when a connection was received, on a clean, out-of-the-box installation, making the host inaccessible.

These sort of super-critical bug make software go immediately into my blacklist, and it's very hard to come back from that - it basically meant that it had to be disabled immediately, because it's defaults were completely broken.

chucky_z• 10 years ago

I installed Fedora Server 25 a week or so ago on a small server, sshd was open (with an extremely strong random password) and fully functional at install time. SELinux has caused no issues, I actually chose to install docker in the installer at install-time and it came pre-configured to play nice with SELinux.

Firewalld on the other hand, I'm still figuring out (firewall-cmd is useful, but trying to translate iptables rules -> firewalld is proving harder than I expected)

ams6110• 10 years ago

One of the reasons I like OpenBSD. Linux has a habit of taking well established things and replacing them with incompatible things (firewalld, systemd for example).

I have enough to do, I don't need to throw away years of acquired experience every few releases. It's one thing if the replacements are clearly better, but for me they just seem to be new, different ways to do the same things.

__david__2• 10 years ago

Every time I mess with Fedora I have to screw around with selinux trying to figure out how to make ~/.ssh/authorized_keys work. Every time I need to google the stupid magic incantation that makes things work (that normally should "just work") because I can just never remember it.

bigmac2• 10 years ago

This post is incorrect. SELinux does not fully mitigate this issue. We recommend users update to 1.12.6.

I expect Red Hat to issue a retraction shortly. We notified them last night that this post was incorrect.

Source: Security at Docker.

saycheese2• 10 years ago

> "Source: Security at Docker"

In case it is not obvious, the comment above is by Nathan McCauley, who is the Director of Security for Docker.

Source: https://news.ycombinator.com/user?id=bigmac

philtar• 10 years ago

I don't care if Jesus, the director of security in heaven said that.

I'm going to take a look at both arguments and decide for myself. No need to name drop.

jwildeboer6• 10 years ago

Why not simply state that when posting? Bad style IMHO.

jwildeboer6• 10 years ago

AFAICS Red Hat explicitly mentions that updates are available and the benefits of having SELinux in no way means you shouldn't update.

Disclaimer: I work at Red Hat

runesoerensen4• 10 years ago

the benefits of having SELinux in no way means you shouldn't update

Where is this explicitly mentioned in the post?

I got the opposite impression reading this story titled "Docker 0-Day Stopped Cold by SELinux", with the closing statement "When we heard about this vulnerability we were glad to see that our customers were safe".

I'm sure your customers are glad to hear that as well, but it sounds like the Docker folks have reason to believe SELinux doesn't fully mitigate this vulnerability.

otterley4• 10 years ago

Can you please take some time here to explain why their post was incorrect, with a brief technical explanation of why SELinux enforcement failed to stop the attack that exploits the particular vulnerability?

I realize you're busy, but it would be much more helpful than a curt statement that simply claims they are wrong.

bigmac2• 10 years ago

We're working with Red Hat now. Folks can expect more technical details when everyone is on the same page.

That said, the solution is the same as with every other piece of software -- update to latest to get security fixes.

krakensden• 10 years ago

Especially given Docker Inc.'s history of counterproductive Red Hat hostility.

cyphar2• 10 years ago

Do you have a PoC? Even if you don't, please add the information to the security@opencontainers.org thread. I'm guessing that it would involve overwriting program code before the SELinux policy is set by runC?

throw2016• 10 years ago

Selinux is badly designed and it's not suprisingly people don't use it. Experts are supposed to simplify complexity. You can't design convulated and complex applications that are user hostile and break things without warning and then accuse people of laziness. Few who deploy apps take security casually and selinux is not the only way to gain security.

In a typical scenario when deploying software things can already get hairy and with selinux in the way you could end up going down multiple rabbit holes and squander hours only to discover selinux is somehow in the way disabling some functionality but not logging clearly what exactly it is disabling with proper messages. That's why most advice online is to disable it.

Given its connected to the NSA and Redhat tried its best to get it into the kernel at one time is all the more reason for anyone concerned about NSA to avoid it. Security experts like the author of Grsec also doesn't think too highly of it.

ungamed• 10 years ago

Spender/GRSec mainly doesn't like it because he has a competing RBAC which people pay for. It might be better, it might not.. but I don't have a GRSEC license.

astrostl• 10 years ago

Productivity is also stopped cold by SELinux :P

opk• 10 years ago

Just a pity that SELinux is widely disabled because it ends up being a pain.

Docker 0-Day Stopped Cold by SELinux

Related Stories