Docker is vulnerable to a symlink-race attack

There is no happy medium. Due to its complexity, the whole ecosystem of modern software engineering is inherently dysfunctional. Being able to understand existing solutions and to create your own solutions that aren't broken toys is both essential.

jameskilton• 7 years ago

"Don't build your own" does not imply "just use what's there blindly". You should have an understanding of the tools you're using, but of course if you try to audit every line of code you include into your app (God help you if you're using node and npm/yarn) then you'll never get any work done.

At some point you simply have to trust other developers, but don't be afraid to build your own, especially if it's smaller functionality (looking at you left-pad ...).

Dependency management is a skill all developers should learn but it's not one that is actively taught or encouraged. Most (including myself) only really start learning once you've felt the bite of that one bad dependency, or fought through dependency upgrade hell.

In short, make sure you're asking yourself "Do I really need this as a dependency or can I build something myself?"

AlexB138• 7 years ago

It's not something to have a crisis about. We all stand on the shoulders of giants. There's too much out there to understand every facet yourself (for a fun example of this, go read I, Pencil) and have any time left over to actually make anything. Like most things in engineering, and life, it's a trade-off. The only problem here is the smirking know-it-all nature of comics like that.

thenewwazoo• 7 years ago

"A novice was trying to fix a broken Lisp machine by turning the power off and on.

Knight, seeing what the student was doing, spoke sternly: “You cannot fix a machine by just power-cycling it with no understanding of what is going wrong.”

Knight turned the machine off and on.

The machine worked."

worldsayshi• 7 years ago

If you'd like to venture further into this rabbit hole I can recommend "Preventing the Collapse of Civilization": https://www.youtube.com/watch?v=pW-SOdj4Kkk&list=LL6MdPYF0rD...

And "The Thirty Million Line Problem": https://www.youtube.com/watch?v=kZRE7HIO3vk

mrguyorama• 7 years ago

There is absolutely nothing wrong with building your own for personal projects, especially fun ones. The "don't reinvent the wheel" is more about how if you are ENGINEERING something, you should fully understand standards and best practices, how to use them, any configuration if needed, and composition of tools.

The professional will use and understand tools, it's the enthusiast who builds his own.

zzzeek• 7 years ago

The happy medium is build your own, do an amazing job, and don't care what some clueless middle manager who secretly hates that you are actually able to write your own is telling you.

Also forget about sleeping or having any social or family life for the next eight years.

munchbunny• 7 years ago

The "don't build your own" advice has a more specific piece of wisdom embedded.

When you build and use a non-trivial piece of software, you also create a large body of non-trivial design and reliability issues, which will take you many iterations and possibly rewrites to get right. Mature software has generally had time to address those issues, and what isn't addressed is better understood by the users and implementors.

When you roll you own, you have to ask: compared to the thing I'm replacing, do I understand the problem domain well enough to anticipate the issues?

PyroLagus• 7 years ago

I definitely understand the sentiment of "don't build your own" when it comes to cryptography, but do people get mocked for not using Docker and instead using a traditional containerless setup? (Well, I guess people get mocked for anything.) Of course you shouldn't reinvent Docker; if you really need containers, use what's there (whether that's Docker or something else) and try to learn how it works, but I think there's more choices than just "use docker containers" or "build your own containers".

drdaeman• 7 years ago

What's the happy medium?

Possibly, not building your own, but investing some time researching how others have built already existing tech?

That helps if what's out there doesn't work for you - either if it breaks (and you have to fix someone else's stuff that you use) or lacks feature-wise (and you _have_ to build your own, either by extending or re-designing).

MagicPropmaker3• 7 years ago

Why is MalwareBytes giving a warning for this site?

phoe-krk2• 7 years ago

MalwareBytes is wrong. seclists is a website containing security mailing lists, and CVE mails are a very common sight there.

smpetrey• 7 years ago

Because MalwareBytes is scareware?

MagicPropmaker3• 7 years ago

I'd still like to know why MalwareBytes is warning me about this page. If you can't answer that, why mock someone?

raesene9• 7 years ago

So from a reading of the advisory and pull request, this seems to affect a specific set of scenarios, where a malicious image is running (attacker has gained access either to the running container, or compromised the image before distribution) and then a process goes to copy data out of the running container (for example by using docker cp).

Not sure if there are other scenarios where this would hit as well.

One to be aware of, but as with most vulnerabilities, good to understand how it can be exploited, when you're assessing mitigations...

chatmasta2• 7 years ago

Many services allow uploading arbitrary images. This is certainly a threat they should mitigate against in their sandboxing strategies.

rtempaccount1• 7 years ago

Would an arbitrary image upload alone allow exploitation of this, or would it require an operation on the host along the lines of a 'docker cp' as well?

Bucephalus355• 7 years ago

This is a known attack for quite sometime. The Apache Web Server, which I love/hate, has for years never really been 100% secure because of issues with how the Linux kernel handles FollowSymLinks and SymLinksIfOwnersMatch. You can purchase special distributions of Linux that are patched against this vulnerability, and also I believe this gentleman [1] has released an OS patch that sort of fixes the issue.

Anyway I switched to Caddy [2] even though I deeply love Apache and it has an amazing history behind it.

[1] https://twitter.com/hanno

[2] https://caddyserver.com/

xrisk• 7 years ago

Caddy is brilliant. Configuration is so simple and easy compared to Apache / Nginx.

tdhz772• 7 years ago

Have you tried Traefik ?

mholt• 7 years ago

Thank you!! These comments made my day. Feel free to open an issue on GitHub if you have any ideas for how to improve it even more, since the sky's the limit for Caddy 2.

https://github.com/mholt/caddy/issues

jetpks• 7 years ago

The vulnerability is solved pretty easily by doing your ownership checks _after_ you've opened (but not served) the file. At least, that's how I did it [1]. Despite our pleas and submitted patches, Apache wasn't interested in fixing it upstream.

[1] https://github.com/bluehost/apache-symlink-patch

cyphar11• 7 years ago

Small correction -- you'd want to O_PATH, then verify, and then do a real open (by doing a re-open through /proc/self/fd). Sometimes, tricking you into opening a file is sufficient to cause problems.

kazinator3• 7 years ago

I think symlink attacks could, at least in some situations, be prevented by a Boolean function (perhaps in the kernel) which validates a path. It returns true if the path is immune to symlink attacks, false otherwise. Immune to symlink attacks means this: the path is walked, and the permissions of each directory are checked to make sure that it isn't open for anyone to create a symlink (anyone whose effective ID is not either root, or the caller). If the path is relative, then the current working directory is prepended to it for the purposes of checking. (A nuance here that has to be taken into account is sticky-bit directories like /tmp.)

If the path contains symlinks, then we have to validate that the real parent directory of the symlink target doesn't allow that target to be replaced. E.g. we are following /foo/bar and bar is a symlink to /xyzzy/pop. If /xyzzy is writable to adversaries, pop can be replaced by a malicious symlink even if pop itself isn't writable to other users.

Anyway, the idea is that if a path we would like to operate on has any components vulnerable to manipulation, we fail the entire operation.

Things can be simplified if we canonicalize the path (so it is free of symlinks) but that is often undesirable. Software should keep the paths it has been given as-is; the symlink abstraction belongs to the user and should be respected.

the84725• 7 years ago

Or you can use the open + O_PATH + *at syscall family which can be used to get a handle on a resolved directory relative to which you can manipulate with no re-traversal happening for different operations on that handle.

This combo exists exactly to avoid these kinds of issues.

Another way would be temporarily joining the container's mount namespace to obtain the source handle. But that can't really be done in go since goroutines don't play nicely with per-thread operations.

Edit: After looking through the go standard library it seems that there is an impedance mismatch. Go just does not expose the necessary pieces to do this properly. A dedicated docker-cp tool in C or Rust could probably handle this better. I could be wrong though, maybe it's just not part of the stdlib.

kazinator3• 7 years ago

But that starting point has to be initially checked somehow. Say we are user "bob" we open "/foo/bar/xyzzy" (for later use with openat), and, say, "bar" is writable to "mallory", then that isn't a path we can trust in the first place.

O_PATH is a Linux Kernel 4.x thing; can't find it in POSIX. Won't compile on non-Linux POSIX operating systems or older kernels. It seems to be an optimization; if we omit it, we just get a "fatter" file descriptor that takes more cycles to set up.

anaphor2• 7 years ago

https://github.com/google/capsicum-linux would basically render this entire class of bug impossible if used correctly.

It would probably require a redesign of how docker interacts with the host filesystem though, and obviously relies on something that isn't yet in the mainline Linux.

cyphar11• 7 years ago

Just using O_PATH won't completely save you. O_PATH can definitely help, especially if you actually check that the path is what you think through readlink(/proc/self/fd/$n), but doing it safely is quite a bit harder than just using *at(2). As mentioned in TFA, I am working on some kernel patches (which will probably result in an openat2(2)) which allow for restriction of path resolution so you can block symlink crossings entirely or force symlinks to resolve to the dirfd you passed.

paulfurtado4• 7 years ago

It's certainly possible to call nsenter in Go but even so, unlike other namespaces, mount namespaces affect the entire process rather than threads, and the thread must not have launched any threads yet at the time that nsenter is called.

That said, docker does support `docker exec` which gives you a shell inside the container's namespaces and cgroups. I'd imagine they could do something similar and just not call exec once they've entered the container. This would be similar to calling `docker exec $containerid cat /path/to/file`

cpuguy834• 7 years ago

The problem is checking for it and then operating on it. The check and the operation would need to prevent anything else from operating on it during that time.

This is why the the proposed patch to docker is pausing the container during the copy.

kazinator3• 7 years ago

The check and the operation would need to prevent anything else from operating on it during that time.

I don't think you understand. If we verify that the absolutized path has sane permissions from top to bottom, then nothing else can operate on it; nothing else that is not either the superuser, or our own user ID. I.e. no untrusted security context.

(If you think that the requirement is literally "nothing else", such as a different thread in the same application in exactly the same security context, then that's a whole different set of goalposts in another soccer field.)

ga-vu• 7 years ago

Not that big of a deal after you read it, though

bayareanative• 7 years ago

"Docker is vulnerable" is also a valid statement at present, hopefully it will improve by the forces of $$$ and tech fashionablism.

phoe-krk2• 7 years ago

It won't. Docker already has too much inertia to be stopped by a single CVE. The "it's just another bug of an actively supported piece of software, we'll just update tomorrow" behaviour will click.

Docker is vulnerable to a symlink-race attack

Related Stories