Don't expose the docker socket, even to a container

https://www.lvh.io/posts/dont-expose-the-docker-socket-not-even-to-a-container.html

by lvh • 11 years ago

133 101 11 years ago

benwilber0• 11 years ago

Docker is not a sandbox and was never intended to be. A comprehensive SELinux profile for untrusted Docker containers could be developed, but I've yet to see one. If you want to run untrusted Docker containers, then this is what you want.

0x400614• 11 years ago

I don't understand the point of Docker. It seems like a great product. For any serious production grade containerization , I'd use a real virtualization solution like KVM, or VMWare.

geofft8• 11 years ago

It's containerization between trustworthy apps; it's not security containerization. What it gets you is, if you have one application that's designed to run well on RHEL 5 with /usr/bin/python pointing to Python 2.4, and another one that's designed to run well on Debian testing with a manual /usr/bin/python symlink to Python 3, you can give both of them what they want. This has nothing to do with security.

If you want Docker + security isolation, I'm intrigued by Clear Containers, which is a lightweight KVM-based virtualization thing:

https://lists.clearlinux.org/pipermail/dev/2015-September/00...

https://lwn.net/Articles/644675/

takeda6• 11 years ago

It's containerization between trustworthy apps; it's not security containerization.

Isn't that what a process is? The containers in Linux are based on Jails from FreeBSD and zones from Solaris. They are absolutely there for security.

Regarding the remaining part of your post, I understand what you are trying to show but python is a really bad example. You absolutely can have python 2 and 3 side by side, or even different minor versions. And with virtualenv or pyvenv (that came with 3.4) you can even have multiple installation of the sane version. If you add setuptools to your application you can easily generate single file package (I personally like wheel) the deployment is as simple as writing pip install myawesomeapp-1.0.py2.py3.whl it downloads all dependencies. There is not much that Docker would help, it only makes things more complex.

philips3• 11 years ago

If you want to try out Clear Containers we merged it into the rkt container runtime a few weeks ago. And since rkt can run both docker and appc container images you can run any existing app container from this runtime. This blog posts details Clear Containers in rkt:

https://coreos.com/blog/rkt-0.8-with-new-vm-support/

LoSboccacc2• 11 years ago

so it's basically a complete set of workaround for enabling people to live with brain-damaged applications?

and we are encouraging this now, instead of fixing the damned apps?

sheesh this makes me feel old.

stephengillie2• 11 years ago

Any virtualization solution is going to require you manage an operating system. One of the goals of contanerization is for developers to only work with the application.

voltagex_2• 11 years ago

Unfortunately most docker containers seem to bring along 200+mb of operating system with them.

AgentME• 11 years ago

Think of it like a tool for packaging and deploying applications with everything they need. Its purpose is closer to package managers than being a secure sandbox for running untrusted users' VMs.

vezzy-fnord• 11 years ago

Funny, an article got posted here just an hour ago advocating against treating Docker like a package manager: http://vilkeliskis.com/blog/2015/09/23/docker_is_not_a_packa...

I'd be inclined to agree. The reductionist sum of mechanisms that make a Linux container have always been about detaching, multiplexing and partitioning kernel resource subsystems. Docker was the first program that really hyped it into the idea of being about application deployment, but I fear this gives people wrong impressions and makes the mistake of treating an emergent property as if it were a fundamental.

doug1001• 11 years ago

that was my initial impression i got from reading some posts on the Docker Site, and perhaps there is no consistent definition of "package manager"--to me though, the most difficult tasks a p/m must do are: reproducible builds, dependency management, conflict resolution among transitive dependencies, and the like. But Docker does none of these things as i understand it.

jkyle2• 11 years ago

KVM & VMWare are not containerization, they're full virtualization.

There are a lot of benefits to containers and they don't have to be insecure. More efficient resource utilization and orders of magnitude faster allocation and launching to name two.

Google runs a significant portion of its internal operations in a container infrastructure and has for quite a while.[1]

They're perfectly capable of deployment into production environments.

I won't comment on docker as I haven't spent the time to fully grok all its warts.

1. http://research.google.com/pubs/pub43438.html

_yy• 11 years ago

Google runs a significant portion of its internal operations in a container infrastructure and has for quite a while.[1]

They use containers inside virtual machines. Virtualization for security, containers for deployment.

Pyxl101• 11 years ago

Containerization and virtualization serve different purposes. VMs run actual operating systems within them. A single operating system runs many different containers, that each act something like processes running on that same OS, in a way where they're highly sandboxed and segmented from each other.

If your goal is strong isolation, then VMs are definitely better today. The purpose of Docker and similar container technologies is not that kind of isolation. It's to package up and distribute applications in a way that's more decoupled than simply installing them all on the same system.

takeda6• 11 years ago

I'm not hiding that after I learned about Docker I became skeptic. It seems like yet another thing that people observed what Google was doing, and then implementing it wrong.

Google is using containers instead of VMs. This still provides security isolation and allows them to use resources more efficiently (VM has overhead where you need a whole OS for every instance).

This approach does not make much sense in public cloud, where you already run inside of VM and the overhead is really for Amazon not you. So I see Docker is now pivoting to be a package manager, but there are already tools that do that. You can argue that Docker is simpler but so was rpm when it started. As Docker will grow it will become more complex in order to support all functionality package format already provides. There might be an argument that you can run multiple Docker containers on a single host, but that's what processes are for.

There is change happening, and looks like cloud companies want to create "cloud os", I guess Docker is step toward that direction, but at current state in don't see it offering anything valuable to the organization that uses it.

vacri• 11 years ago

If your VMs are single-purpose, you don't necessarily need VMs. Containers are single-process things - they're not running syslog or cron or any of that overhead, for example. Docker is also big on ensuring you are using the literal same artifact in dev as on prod (assuming you change your team's workflow, of course).

Which is the right thing to use is entirely dependent on your use case.

insaneirish2• 11 years ago

Containers are single-process things - they're not running syslog or cron or any of that overhead, for example.

You're either being entirely too prescriptive or interchanging containers and Docker freely when they are not equivalent, though one is a particular implementation of the other.

People have been running containers that emulate a full system for quite a while (see FreeBSD jails, illumos/Solaris zones, Linux OpenVZ, LXC, etc.).

tbronchain• 11 years ago

We[1] believe that the point of Docker is to provide "application packages" (called containers), which is a big step ahead to deliver applications (using their words: build, ship, run).

However, we also do believe the isolation containers provide isn't sufficient for multi-tenant usages. This is the main motivation behind Hyper, which run groups of container images (Pods) as Virtual Machines.

[1] https://hyper.sh

hosay123• 11 years ago

I don't really get this, the implication is the container becomes more secure without access to the socket, yet it has access to the hundreds of local kernel APIs with which on the average month it can easily gain higher privileges than root, especially on contemporary machines where half the admins around these days don't even know what a security update looks like

KirinDave6• 11 years ago

Why knowingly give a trivial breakout vector to code you don't trust?

notjack3• 11 years ago

This assumes that Docker containers are being used like VMs. They're not designed to allow running isolated arbitrary code in some sort of multi-tenancy setup, they're designed to isolate dependencies and configuration between your own deployed services. This is a "security vulnerability" in the same way that putting up a white picket fence around a jail is - a gross mis-application of a tool built for a completely different purpose.

insaneirish2• 11 years ago

They're not designed to allow running isolated arbitrary code in some sort of multi-tenancy setup

You mean Linux containers are not designed for this, nor are they designed to be secure. What a sad design failure.

Some [1] container technology was developed with security as a first principle.

[1] http://us-east.manta.joyent.com/jmc/public/opensolaris/ARChi...

notjack3• 11 years ago

I should clarify I mean Docker containers specifically, good point.

geofft8• 11 years ago

And you can run Docker containers on SmartOS, using "LX-branded zones", which do Linux syscall emulation:

https://www.joyent.com/blog/triton-docker-and-the-best-of-al...

I'd be curious if the Joyent folks believe Triton to have the security properties that everyone seems to want out of Docker. It doesn't involve the standard Docker daemon, so it might.

JoshTriplett3• 11 years ago

As far as I can tell, Solaris Zones are no more secure than Linux containers; both use kernel-level isolation mechanisms, while sharing the same kernel between zones/containers. And thus they both have the same security property: secure as long as you can't successfully exploit a syscall, but if you can, you get kernel-level access.

kentonv3• 11 years ago

Linux containers are certainly intended to be secure. The problem is that the Linux kernel API is huge and privilege escalation bugs are found all the time. In order to make a container secure, you need to disable 95% of this API, e.g. using seccomp, not mounting /proc or /sys, drastically limiting /dev, etc. Sandstorm.io hasn't seen a single working breakout since we started keeping track over a year ago, despite numerous Linux kernel exploits going by in that time, because all the bugs have been in features that we turned off.

https://docs.sandstorm.io/en/latest/developing/security-prac...

That said, there is a cost in compatibility. Most apps can be made to run just fine in this constrained environment, but it does sometimes require tweaks. Docker is more interested in compatibility than in security, so naturally they don't do this kind of attack surface reduction by default. (You can configure it manually, but realistically if it isn't mandatory then few people will bother.)

acconsta4• 11 years ago

Are linux containers really not securable? That's how Google runs all their stuff in production.

http://research.google.com/pubs/pub43438.html

stephenr• 11 years ago

Isn't this specifically a Docker failing? My understanding (from reading rather than use) is that LXC (the original Linux container project) provides a lot more in the way of security than Docker does?

pjmlp2• 11 years ago

Specially since HP-UX and Tru64 were doing it already in the late 90's.

zurn• 11 years ago

Listening to what the Docker people write about security, they sure do sound like they are designed to be secure, even though they admit to a couple of potential shortcomings versus VMs.

https://blog.docker.com/2013/08/containers-docker-how-secure... concludes,

"Docker containers are, by default, quite secure; especially if you take care of running your processes inside the containers as non-privileged users (i.e. non root)."

They also recommend using SELinux with Docker to beef up security (see https://blog.docker.com/2014/07/new-dockercon-video-docker-s...)

As long as people are using containers as a security boundary it makes sense to pay attention to things like this one about the control socket.

kentonv3• 11 years ago

There is a section in that link titled "Specific Attack Surface of the Docker Daemon". Meanwhile they are ignoring the attack surface of the Linux kernel. There is no mention of seccomp in the whole post.

People find privilege escalation bugs in the Linux kernel often enough that you can't really claim that running arbitrary native code is "secure" unless you're doing something to mitigate these. Things like:

http://www.openwall.com/lists/oss-security/2015/07/22/7

Note that SELinux won't do anything about this kind of bug. You have to use seccomp and similar to disable large swaths of the Linux kernel API, particularly exotic parts that aren't well-tested or reviewed.

DyslexicAtheist• 11 years ago

I second the idea of using SElinux to secure docker because most people really are not able to fully deal with the complexity of SElinux. If they would understand security they wouldn't be jumping onto the docker hype bandwagon in the first place.

To paraphrase Theo de Raadt:

‟You are absolutely deluded, if not stupid, if you think that a worldwide collection of software engineers who can’t write operating systems or applications without security holes, can then turn around and suddenly write virtualization layers without security holes.”

[1] http://blog.valbonne-consulting.com/2015/04/14/as-a-goat-im-...

lvh• 11 years ago

Absolutely; as I mention in the article. There's nothing new or exciting here; there's simply a big discrepancy between reality and how a lot of users understand it. This is partially true for the my-dev-user-is-part-of-the-docker-group case, but even more so for the container-with-docker-sock-access case.

notjack3• 11 years ago

First let me state that this article is interesting and well written and taught me something new, thank you for that.

However my main frustration is I'm not really sure what the article is advocating for here. Instead of not giving access to the docker daemon to containers (which is legitimately needed for complex deployments where one container needs to dynamically start up "sibling" containers, e.g. a CI service), wouldn't it make more sense to talk about not viewing Docker container security the same as VM security in the first place?

Sure if you're going to do that anyway it makes sense to disable access to the socket, but then there's a million other things you'll have to do because docker containers are currently not primarily intended as a replacement for the isolation security of VMs. Their security is more like a useful extra layer, rather than a full blown replacement.

nailer3• 11 years ago

Running docker inside VMs loses all the IO benefits. That's not to say I use docker at all: I don't for this reason. Someday it will be solved and docker will actually be as production ready as its proponents think it is.

nailer3• 11 years ago

Since folks seem to be new to docker and Xen: Hypervisors like Xen have very poor IO performance: one of the major benefits of docket is that it provides containerisation without the IO overhead of a hypervisor. Neither of those have been disputed by anyone, ever.

nogox• 11 years ago

Check www.hyper.sh and https://github.com/hyperhq/runv

They can boot a new VM with Docker images in 200ms, which is very close to LXC, and perfectly isolated by hypervisor.

The problem of "Virtual Machine" is not "Virtual"/Virtualization, the problem is the full blown guest OS, aka "Machine".

codemac3• 11 years ago

Docker is a rootkit over HTTP.

voltagex_2• 11 years ago

A rootkit is a collection of computer software, typically malicious, designed to enable access to a computer or areas of its software that would not otherwise be allowed (for example, to an unauthorized user) while at the same time masking its existence or the existence of other software.

Doesn't really describe Docker, does it?

codemac3• 11 years ago

It does if you're talking about `docker -d` and --privileged.

jayfk• 11 years ago

In other news, mounting / gives you access to the root filesystem.

KirinDave6• 11 years ago

Do... people really offer docker sockets to running containers without thoroughly vetting them first? Are people really that good at avoiding the warnings?

I mean, I know it's popular to pass the socket in for automatically re-configuring proxies... but I haven't seen any serious use outside of that.

raspasov• 11 years ago

Can someone describe a real-attack scenario, using Docker's default settings as of the latest version? I see a lot of people claiming that it's insecure but no concrete examples of exploits. I am not saying that it's as secure as VMs, or that it's inherently secure - I am genuinely* interested and really do care about this.

*We're in a limited beta of cloudmonkey.io, and we want to run unrelated/untrusted containers side by side securely.

brianshaler• 11 years ago

I assumed that's what the video in the article was about [0], but I haven't watched it to be sure. Is there anything in particular that isn't clear from the article? Or are you asking about docker security concerns beyond what this article is about?

It seems the main point is if there is any way to exploit code running within a container that has unfettered root access to the host system via the docker socket, an attacker would then have complete control over the host system.

Exploitation is often mitigated in layers, where if Service A is exploited, an attacker can only rwx what and where Service A has been granted priveleges to rwx. That should be as little as possible, the bare minimum access that service needs to operate. There's no reason your web server or database should be able to install new programs, create users, etc.

If Service B is running in a container and is given access to write to the docker socket, suddenly any exploitation of that service opens a door to immediately have full and unfettered root access to the host system.

[0] FTA "... ended up making a screencast to unambiguously demonstrate the flaw in their setup..."

alpb• 11 years ago

dockerd -- nope. It's `docker -d`

nailer3• 11 years ago

'daemonisedthingd' is standard UNIX terminology.

skarap2• 11 years ago

To be honest, exposing host's /var/run/libvirtd.sock to a guest VM will have exactly the same consistencies.

tobbyb• 11 years ago

A container is just running a process or chroot in its own namespace. If you run it as it is you have a single process container like Docker uses, if you run an init in the namespaced process you have LXC or OS containers that can support multiple processes like a lightweight VM. [1]

With LXC containers you start them as root and there is no lingering background LXC process running. Docker also starts containers as root but also has dockerd hanging around presumably so non root users can interface with it. But the container process is still running as root so dockerd seems a bit redundant and unnecesary.

This is because untill recently you couldn't run chroot as non root users and needed to run containers as root. But 'user namespaces' (> kernel 3.8) changes this and allows users to run processes in namespaces as a non root user. LXC has supported unprivileged containers for some time now [2] so you can run LXC containers as non root users, as in the entire container process is unprivileged. Docker and Rkt are working on this but its not simple to implement for container managers as non privileged users cannot access networking and mounts. But when it does presumably dockerd can run as an unprivileged process.

But Linux kernel namespaces have not been designed for multi-tenancy for instance cgroups are not namespace aware, and untill this changes in the kernel, containers will not provide the level of isolation or security required for multi-tenant workloads.

And containers managers like LXC or Docker that take these capabilities and merge them with networking and layered filesystems like aufs or overlayfs cannot work around this. Parallels OVZ is designed for multi tenancy but the kernel patch it appears is too large and invasive and doesn't look it will be merged.

So user namespaces is one level of security and isolation, you can also use seccomp, app armour, selinux or even grsec. But you have to find the middle ground between security and usability and given the relative confusion about containers, namespaces, and container managers it will take time to mature.

[1] https://www.flockport.com/how-linux-containers-work/

[2] https://www.flockport.com/lxc-using-unprivileged-containers/

Don't expose the docker socket, even to a container

Related Stories