Simple Dockerfile examples are often broken by default

https://pythonspeed.com/articles/dockerizing-python-is-hard/

by itamarst • 7 years ago

461 157 7 years ago

Good points, but it's amusing that his solution to #1 didn't lock down the patch version, nor the distro around it. I think that also makes a decent point for Nix[0], which solves #1-#3 by default (since choosing a particular version of Nixpkgs locks down the whole environment, and considers the build as a DAG of dependencies rather than a linear history). It also supports exporting Docker images, while preserving Nix's richer build caching.[1]

[0] https://nixos.org/nix/

[1] https://grahamc.com/blog/nix-and-layered-docker-images

itamarst8• 7 years ago

Good point, will go fix that. My soon-to-be-ready attempt at a production-ready template (https://pythonspeed.com/products/pythoncontainer/) covers the tradeoff between point releases vs. not-point-releases, and it does pin the OS.

And yes, Nix fixes some of the problems of building a production-ready image, but only a subset.

j88439h843• 7 years ago

Could you elaborate on the remaining problems with Nix for building Python images?

X6S1x6Okd1st• 7 years ago

Nix does a pretty poor job of being able to specify the exact version that you need of something.

Nullabillity5• 7 years ago

It's not great at mix-and-match pinning, but if you pin a version of Nixpkgs then you will always build from the same environment (down to the libc). There's some boilerplate, sadly, but it's still fairly simple to understand.[0]

[0] https://github.com/Etimo/photo-garden/blob/f597b95c0c488abad...

jand2• 7 years ago

I do not intend to play down the importance of using docker carefully.

But the reproducible build aspect of the critic seems unnecessary to me: Isn't that more a concern of the packaging system? (no python scripter)

If your packaging systems supports version selection/locking, then use your packaging system right. If your packaging system cannot pin a version, how should docker solve this?

itamarst8• 7 years ago

This issue isn't Docker, Docker itself obviously can't deal with this. The issue is the examples and tutorial people provide on Docker packaging often don't talk about the requirements (pun intended) for reproducible builds at all.

Or they don't talk about need to run as not-root.

Or they suggest base images that are often broken in subtle ways (Alpine Linux).

Or they talk about multi-stage builds for small images, and neglect to explain that you've just destroyed your caching (this is fixable, but you need to know to expect it and how to fix it.)

Etc.

batbomb2• 7 years ago

The caching part is (mostly) generally fixable by making sure to always add your source code in two stages - dependency files first, then resolve dependencies, and the rest of the source code later.

Rarely is this done though. And definitely alpine+musl doesn't always do what you might expect, and it's often language dependent as to whether or not you'll encounter something strange (not to mention you forfeiting bash)

zokier• 7 years ago

Docker is a packaging system.

jand2• 7 years ago

Thank you for clarifying this.

Did you understand the point i tried to make nonetheless or do i need to detail it?

j88439h843• 7 years ago

Most Python packaging systems don't include the Python VM itself, though they can specify which version is required. In the post, Docker is used to provide a specific version of the Python VM.

AgentME2• 7 years ago

If your packaging systems supports version selection/locking, then use your packaging system right.

That's exactly what the article is recommending in point 2. The original Dockerfile author was using pip in a way that's only intended for in development. Having a requirements.txt file is the correct way to use pip when distributing a project.

derriz• 7 years ago

Docker can't escape all the blame here - its layer caching mechanism is IMHO flawed. It's fine to say that a packaging system should offer reproducibility but Docker's layer caching design assumes that every RUN command produces reproducible results.

You could of course blame users for not making sure that all the commands they use in their Dockerfiles are actually reproducible but many/most examples even in the official documention are clearly not reproducible.

Therefore you end up with what is in my opinion a semi-broken system - building images seems to be reproducible (and fast) until you lose your layer cache or you spin up a new CI build agent or a new dev joins the team and tries to build the same image.

Not that I can think of an clean and performant solution to this problem.

tln• 7 years ago

This is a great point -- you could potentially solve this using --cache-from which makes the layer cache explicit, and not something that varies between dev / CI / new devs.

tobbyb• 7 years ago

We have been working on a simplified container build system which does away with layers altogether. [1]

The use of layers at the build stage adds a lot of needless complexity with very little benefits and users really need to step back and question the value they are getting from the use of layers. [2]

Words like 'immutability', 'declarative' and 'reproduciblity' are often used in ways that can lead to user misunderstanding and can be accomplished with simpler workflows. For instance immutability, reuse, composition do not require layers. There needs to be a lot more technical scrutiny to avoid confusion.

[1] https://www.flockport.com/docs/containers#builds

[2] https://www.flockport.com/guides/say-yes-to-containers.html

btilly• 7 years ago

I have a mixed opinion about his first point.

There are two basic approaches to take with dependency management.

The first version is to lock down every dependency as tightly as you can to avoid accidentally breaking something. Which inevitably leads down the road to everything being locked to something archaic that can't be upgraded easily, and is incompatible with everything else. But with no idea what will break, or how to upgrade. I currently work at a company that went down that path and is now suffering for it.

The second version is upgrade early, upgrade often. This will occasionally lead to problems, but they tend to be temporary and easily fixed. And in the long run, your system will age better. Google is an excellent example of a company that does this.

The post assumes that the first version should be your model. But having seen both up close and personal, my sympathies actually lie with the second.

This is not to say that I'm against reproducible builds. I'm not. But if you want to lock down version numbers for a specific release, have an automated tool supply the right ones for you. And make it trivial to upgrade early, and upgrade often.

pdonis2• 7 years ago

The post assumes that the first version should be your model.

No, it doesn't. It just assumes that you want explicit control over when you upgrade. You can always change your Dockerfile or your requirements.txt and build again when you've tested your software against a new Python version or a new version of a package. You can do that as often as you like, so this is perfectly consistent with "upgrade early, upgrade often". But not specifying exact versions in those files means they can get upgraded automatically when you haven't tested your software with them, which can break something.

yongjik• 7 years ago

From what I've seen, explicit version control does not really work unless there's an organizational force toward timely upgrade. In every company, everyone's busy, nobody has time to look at a service that's been running fine for six months (and risk getting paged "Why did you change it? It suddenly stopped working for us!"). The path of the least resistance is to not upgrade anything that's running "fine", and then old versions and their dependencies accumulate, and when you actually have to upgrade it becomes much more painful.

It might work if there's a dedicated team whose mission is upgrade dependencies for everyone in time, but I haven't seen one in action so I'm not sure how well it might work out. (Well, unless you count Google as one such example. But Google does Google things.)

shawnz• 7 years ago

Isn't this dichotomy the whole point of dependency locking? Sometimes, you want to specify that your code requires a specific version. Sometimes, you just want to keep track of the most recent version that the code has been tested with. They are two totally different needs

jasonhansel• 7 years ago

I think the last option you mentioned is (effectively) the best of both worlds. Lock down dependencies explicitly for the sake of reproducibility, but make it very easy to upgrade (as automatically as possible).

awinder• 7 years ago

At my previous workplace we were using https://greenkeeper.io/ and locking dependencies, which i think may be the perfect compromise between those 2 systems of organization. You get pinned dependencies, resulting in stable builds. For every package update (automatically scanned), you get a branch spawned w/ tests run if you've set up CI. It makes staying up to date easy when it's an easy upgrade (just merge a green branch!), and you get isolated knowledge up-front when a dependency has upgraded and you're gonna need to budget some time on it.

itamarst8• 7 years ago

GitHub just bought Dependabot, so something like this is now available in beta and eventual general availability for all GitHub users.

tetha• 7 years ago

Something we have built for our stuff: There's a private repository and all applications run with minimum versions for their dependencies, so if there's a new available version, everything will update.

Beyond that, we have a daily job that runs the integration tests of all applications with the upstream repository, and if all integration tests end up green, the current set of upstream dependencies gets pushed into the private repository.

It is work to get good enough integration tests working, and at times it can be annoying if a flaky new test in the integration test suite breaks fetching new versions. But on the other hand, it's a pretty safe way to go fast. Usually, this will pull in daily updates and they get distributed over time.

And yes, sometimes it is necessary to set a maximum version constraint due to breaking changes in upstream dependencies. Our workflow requires the creation of a priority ticket when doing that.

oconnor6632• 7 years ago

The first version is to lock down every dependency as tightly as you can to avoid accidentally breaking something...The second version is upgrade early, upgrade often...Google is an excellent example of a company that does this.

This is misleading. My understanding of Google's internal build systems is that they ruthlessly lock down the version of every single dependency, up to and including the compiler binary itself. They then provide tooling on top of that to make it easier to upgrade those locked down versions regularly.

The core problem is that when your codebase gets to the kind of scale that Google's has, if you can't reproduce the entire universe of your dependencies, there is no way any historical commit of anything will ever build. That makes it difficult to do basic things like maintain release branches or bisect bugs.

if you want to lock down version numbers for a specific release, have an automated tool supply the right ones for you. And make it trivial to upgrade early, and upgrade often.

This part sounds like a more accurate description of what Google and others do, yes.

verroq2• 7 years ago

Yeah but only google is google. You are not google, your code doesn’t need to google scale and you don’t need to go to their extremes to manage dependencies. They do it because they are forced to, doesn’t mean it is right or their way is how it should work for everyone.

OJFord4• 7 years ago

For an easy open source example of such tooling, see Pyup.

We use it to do exactly that: pin down every dependency to an exact version, but automatically build and test with newly released versions of each one. (And then merge the upgrade, after fixing any issue.)

dalore• 7 years ago

Yes they have a huge mono repository and tooling to update projects in it to specific versions. You don't get a choice really. You can go home one night with your project on say Java 7 and then wake up and find someone has migrated it to Java 8 because they've decided it's Java 8 now.

eitland2• 7 years ago

If I remember correctly Angular comes with unpinned dependencies.

paulddraper4• 7 years ago

A larger problem is that Docker is nearly inherently unreproducible.

Downloading and installing system packages lists, etc.

For this reason, Google doesn't use Docker at all.

It writes the OCI images more or less directly. https://github.com/bazelbuild/rules_docker

xorcist2• 7 years ago

Google is an excellent example of a company that does this.

Which part of Google would that be? My impression is the complete opposite, dependencies are not only locked down and sometimes even maintained internally.

coding123• 7 years ago

Not sure about Python, but I think its language specific. In the JS world, we have "yarn upgrade" which bumps all non-major versions of your dependencies to the latest. It then locks them in until the next time you upgrade something. There are other actions that may also upgrade them, but it's always through a dependency change in some way.

I still think the overall advice is good. We depend on node in our Dockerfile like this:

FROM node:11

If we went further into the version, of course we'd be even better off probably, but there's a tiny point to make here. We don't build any docker images for deployments from dev to production. In fact the last time a docker build is run is for the development environment. After that it's just carrying the image from dev to qa to stg to prod, and we simply change the configuration file along the way.

This makes it so that we're not re-building again and possibly getting a different set of binaries that were not tested in any of those other environments.

AgentME2• 7 years ago

FROM node:11

Node follows semver and rarely has breaking changes within major versions, so this makes sense to do. The article recommends pinning a minor version of Python because it doesn't follow semver and sometimes has breaking changes within minor versions.

tathougies3• 7 years ago

The first version is to lock down every dependency as tightly as you can to avoid accidentally breaking something. Which inevitably leads down the road to everything being locked to something archaic that can't be upgraded easily, and is incompatible with everything else. But with no idea what will break, or how to upgrade. I currently work at a company that went down that path and is now suffering for it.

If you use a system like nix or guix, this concern is largely obviated.

djsumdog• 7 years ago

You have to have tests and you need a CI that will scan your requirements.txt regularly and throw a warning when they're out of date.

Test are ESSENTIAL. You should be able to bump all your versions, run your tests and fix the errors. If something gets through broken, then you know where to add a test (before you fix it).

You should pin versions for your sanity. You should also have a process (a weekly process) to deal with updates to dependencies. Dependency Rot will catch up with you!

efdee• 7 years ago

Maybe this is pedantry, but I have a problem with "if something gets through broken, then you know where to add a test". No. If something gets through broken, your tests have failed to fulfill their purpose.

abtinf• 7 years ago

This will occasionally lead to problems, but they tend to be temporary and easily fixed. And in the long run, your system will age better.

This is the reason I love archlinux. Most of the time, updates are no big deal. Sometimes, they break the system. Rolling release distros force you to deal with each change as it happens, usually with a warning that breakage is about to happen, and a guide for how to quickly deal with it. Once the system is up and running, basic periodic maintenance will keep it that way. In the past, I've used arch machines continuously for 5+ years and they work great and stay up to date.

Compare to intermittent release distros like Ubuntu. Every time I need to update an ubuntu machine, I end up reinstalling from scratch and configuring from the ground up. There are too many things that need tweaking or simply break between when releases are 6-24 months apart. And I'm not convinced that locking down dependencies actually solve anything. Wait six months after an LTS release, when you need to get the latest version of some package. Suddenly, you are rummaging through random blog posts and repos trying to find the updated package. PPAs, Flatpacks, Snaps, oh my! Intermittent distros offload a lot of their responsibility onto users by pretending like package update problems don't exist.

nurettin• 7 years ago

Why do you configure from ground up? Don't you have your home directory in a separate partition? I keep it around and reuse it in case I have to upgrade the whole system. When you install most of your configuration into user space (except things like ssl and third party repos) it becomes easier to recover from a full system upgrade. Worked for me past 10 years.

beat3• 7 years ago

I learned the hard way to lock down dependencies. Long ago, in a galaxy far far away, I was doing some Java/SQL Server stuff. We upgraded Java (which was badly needed), and immediately all the SQL stuff stopped working, which led to a few days of paralyzed bafflement.

Found out a few days later that the official release of the JVM broke the Microsoft SQL Server drivers, and Oracle had to ship a new version out asap. Meanwhile, we lost days of work.

Of course, that was also the bad old days of bad old configuration management. But I'd never do something like put an arbitrary version of a language driver in a Dockerfile, not for production.

edit: Of course, the main reason we get scared to upgrade is because we often can't easily back out the change. Docker fixes a lot of that.

milesvp• 7 years ago

Yeah, seriously, having been in the nightmare that is trying to rollback on systems not designed for it, it is my number one (background) priority to get systems to a place where rolling forward isn't the only answer for trying to rollback. I tell you virtualization really made my life easier, being able to at least take a snapshot prior to major upgrades was a game changer. After that, finally getting chef to the point where we could rebuild production in a mostly repeatable way (dependency chains can only be tamed so far without increasing infrastructure costs) really made dev work easier. Using chef kitchen to trivial build a new VM to know that you're close to production really helped reduce dev time by a lot (even if it seemed like the chef recipes would break in subtle ways every month or so). I've been watching Docker for years now, and am hoping it's hit the tipping point where the benefits outweigh the added complexity. I suspect my next gig that's lacking reproducibility I'll start with docker rather than chef and see how far that takes me.

xorcist2• 7 years ago

we often can't easily back out the change. Docker fixes a lot of that.

Software can be hard to downgrade. Sometimes dependencies change. Sometimes data models are migrated one way only. Nobody takes the time to properly test them. Among other things.

How Docker, or any other container packaging format for that matter, could possibly help with that I do not understand. It is not the first time I've heard something like this, but I have never been in a situation where the application packaging was part of this particular problem.

Surely starting the an old version of some software isn't neither harder nor easier with Docker than any other way.

donmcronald• 7 years ago

I used to be strongly in favor of using fixed versions everywhere, but now I also have a mixed opinion. I think a reasonable compromise is to continually update and to promote images. That way you can start with `dev --> prod` when you're small and add more QC layers as things grow.

Something that's even more difficult is dealing with upstream changes. What do you do when `ubuntu:18:04` updates? It's easiest if the upstream is released with a predictable cadence (ex: every Wednesday), but none are AFAIK. That way you could plan a routine where you regularly promote an update through QC.

I'm not sure what to think about event driven release engineering like auto-builds (repo links) on Docker Hub. I think that might be an ok solution for development builds or rebuilds of base containers, but it seems to be abused. I bet there are maintainers of popular images on Docker Hub that are effectively triggering new deployments for downstream projects every time they publish a new image.

Orphis• 7 years ago

You can always have both in parallel. The first one to test changes and deploy to production and the second one to try to upgrade your dependencies. Should all the test pass on the second one, you can then commit the new requirements.txt and other updated package versions.

You can then run the second continuously and warn when it fails and handle whatever happened manually, without having a broken prod.

LoSboccacc• 7 years ago

we do that but it's hard to CI a whole operating system, some week ago we got bit by a weird imagemagick bug that was triggered by very specific tiffs that weren't in out test suite but that one of our clients used extensively for their product images.

annoying and wouldn't have happened if we were running pinned versions, that said getting stuck on old software would be worse. however nothing can ever test something like that fully, just too many combinations :(

kdmccormick• 7 years ago

Your last option us exactly what the author recommends -- pip-tools.

ljm• 7 years ago

I don’t think your Dockerfile should be downloading your dependencies and building from scratch. Let your build pipeline pull them in, run tests, and pass them verbatim into your container.

Vendor your dependencies if you have to, or maintain a cache, but don’t make your Dockerfile redo all of that work.

underwater• 7 years ago

We have a Dockerfile for builds, and a Dockerfile that consumes the built artifacts for deployment.

Lx1oG-AWb6h_ZG0• 7 years ago

this assumes your build agent is running the same environment as the docker image. What happens when you’re running ubuntu and pull in packages that won’t work in alpine? (Or worse, if it’s windows/OSX and Linux)

matthewbauer• 7 years ago

There is a 3rd and IMO ideal approach. Instead of pinning to dependencies, you pin to an entire set of dependencies. You are guaranteed stability and that dependencies work together while you are pinned to one set. Non breaking changes like security fixes can still happen but major updates don’t. This is less reproducible than locking everything, but we get to reuse fixes among projects and backport them to the shared set. Updating your dependencies is just moving to a new package set. This approach is utilized by NixOS and Stack, but AFAICT no where else.

hvidgaard• 7 years ago

Until a simple bugfix breaks your system because you relied on the wrong behavior.

rgbrenner• 7 years ago

In your examples, you're comparing locking down a system and letting it age.. vs upgrading and fixing things that are broken.

The difference here is the amount of labor. In your first example, you propose no labor investment.. and in the 2nd, regular investment of labor. Of course that results in the 2nd system being better. If you invested the same labor required by your 2nd example in the 1st, you would likely have an equally usable and up to date system. Similarly, if you didn't invest any labor in the 2nd, you would have a broken unusable system (since the bugs would never get fixed).

Another way to think about this is that as systems age (bugs are found, exploits, etc), they create technical debt. You need to invest the time to address that debt, or you will suffer later down the road.. same as most tech debt.

jhasse3• 7 years ago

Google is an excellent example of a company that does this.

Chromium is still using Python 2 in its build system.

oftenwrong• 7 years ago

Python 2 is mature and stable, but not obsolete. It does still receive bugfixes as needed; the latest was version 2.7.16, released 2019-03-04. Maintenance will continue until 2020-01-01.

voidr• 7 years ago

If you have the need to go back to older versions of your code and patch them, you want to lock things down at least at the point of release.

_Codemonkeyism• 7 years ago

"but they tend to be temporary and easily fixed."

Only if your dependencies do the same, otherwise you have a new version of something and your dependency wants something old. Especially hard when older dependencies are no longer maintained.

For different environments and languages the problem might be bigger or smaller depending of the culture, strong/weak type coupling, cross compilation or runtime VMs.

This is the reason we've removed all Scala dependencies and now only depend on Java dependencies.

monksy• 7 years ago

My understanding about google is that they approach the dependencies as trying to build one statically linked binary and they tend to try to put all of the dependencies together in the same repo. (Mono-repo). For a large organization, I'm not convinced that's a sane way to go. Individual groups should have the primary responsibility for the modules that they produce.

What we need is:

1. Tools that will automatically upgrade for us and report back when there are failures.

2. Better automated tools on where our code is missing test coverage (automated upgrades should break our code when something changes)

3. Better CI/CD rejection support (Something breaks the build, force a revert of the code)

treis• 7 years ago

(1) and (2) aren't really broken, IMHO. For most cases always using the most up to date version is better than having 100% reproducible builds. After all, you have the docker image that you can distribute if you really need to. Better to pick up security and performance patches as they become available. If those updates break something then you can make the decision to fix on a known good version.

erik_seaberg2• 7 years ago

If you always pin, you have history to tell you which versions were good. If you mostly don't, you have to start disassembling a bunch of old images just to figure out what they were built from.

bytematic• 7 years ago

Probably going to want to use tagged docker repos so that updating certain packages, no matter the language, don't suddenly break your images

kkapelon• 7 years ago

I am actually preparing my own article titled "Docker antipatterns" that will include many more points like this.

42n4• 7 years ago

Very good clues!

j88439h843• 7 years ago

For reproducible builds, `python:3.7` isn't specific enough. python:3.7.3-alpine3.9 is more specific, for example. There aren't supposed to be breaking changes in the bugfix releases, but they'll happen anyway.

kam• 7 years ago

And `python:3.7@sha256:35ff9f44818f8850f1d318aa69c2e7ba61d85e3b93283078c10e56e7d864c183` is even better.

mistrial92• 7 years ago

overfitting here?

meowface• 7 years ago

Ran into this recently. Docker container was running into issues until I changed "python:3.7-alpine" to "python:3.7.3-alpine3.9". It was because a package I was relying on from "apk add" changed between Alpine versions.

I could probably safely make it "python:3.7-alpine3.9" (instead of pinning to Python 3.7.3), since the issue was the Alpine version, but at this point I'm starting to really buy into the whole reproducible build thing.

jo-wol• 7 years ago

The final example in the article is broken. Python interpreter as PID 1 can't handle linux signals.

itamarst8• 7 years ago

This is why I have a caveat at the top of the article as well as right after the last example. This particular issue is fixable with `docker run --init`, so not strictly necessary to fix in images.

deathanatos2• 7 years ago

It can if you install the appropriate signal handlers, which, if you don't fork, is quite easy. While for a real-world app you're probably right, the article's example is not fleshed out sufficiently to tell one way or another.

eyeareque• 7 years ago

Not locking it to a specific version is better for security updates. Do you want it to run stable with vulnerabilities or to run secure and broken?

bdcravens2• 7 years ago

Not locking it to a specific version is better for security updates.

The idea is that you should take responsibility for your containers and verify fixes and test your application.

Do you want it to run stable with vulnerabilities or to run secure and broken?

If these are your two choices, you have a staffing or a workflow problem.

jessemillar• 7 years ago

A broken Docker image can lead to production outages, and building best-practices images is a lot harder than it seems. So don’t just copy the first example you find on the web: do your research, and spend some time reading about best practices.

While I may not agree with absolutely everything in the article, this final point is paramount. Please don't blindly use technology because you managed to find a copypasta config that runs. Running != good.

zrobotics2• 7 years ago

Definitely very true. I write more C++ than anything else, and the sheer number of online examples that start with

  using namespace std;

is just staggering. Sure, it works in a toy example posted to stackoverflow, but it will cause problems in larger projects. I think globally there needs to be better emphasis on using best-practices in tutorials and examples; I remember this particular pet-peeve of mine also being present in college textbooks. Especially for content aimed at newbies, it should be frowned upon to show the wrong way to do things, since then it gets harder to show how to do it the right way. I've had people who were surprised to find out that they could type:

  using std::chrono::duration;
  using std::cout;

instead of pulling in the entire std namespace; simply because they'd only ever seen examples that did it the lazy way.

edit: lack of semicolons strikes again!

ajross2• 7 years ago

That's... true in the specific example, becuase C++'s standard library is a huge, promiscuous mess of obvious-looking symbol names that are just asking for a collision with a user name.

But in general the notion that we want to isolate "everything" into a namespace is a net loss. Clear and simple abstractions have real value, and short undeclared names are an important part of being clear and simple.

The modern convention of separately importing every symbol you use gets really out of hand, when most of the time it really is appropriate that you just declare "my code is using this API" and expect things to work without having to link your program by hand with a giant shipping manifest of symbols at the top of your source files.

nemetroid• 7 years ago

While I agree with the general point of using best practices in code samples, the Cpp Core Guidelines actually encourage[0] using

  using namespace std;

for std specifically, giving the reasoning that:

sometimes a namespace is so fundamental and prevalent in a code base, that consistent qualification would be verbose and distracting.

I also work mainly in C++, and personally I prefer using it, together with -Wshadow to catch possible issues.

0: https://github.com/isocpp/CppCoreGuidelines/blob/master/CppC...

batbomb2• 7 years ago

In general, in Go, Java, and Python I've resorted to copying in the Gopkg files, pom.xml, and requirements.txt, and then running the requisite dependency installer for the language (dep, pip, mvn, etc...) and then just copying in the rest of the repo, relying on the .dockeringore with a default-ignore for everything and specifying the individual files/directories you may want to add, and in some cases a rootfs folder when necessary.

This seems to be the happy medium for me. I don't have very strong opinions on requirements.txt always being the pinned output from a pip freeze, and it seems like pipenv may actually die in a few years, and poetry will evolve to take the mantle, but I do lots of things with conda anyway.

linuxftw• 7 years ago

The problems described here are called 'release engineering.' Dockerfiles don't solve release engineering, they provide an abstraction for building a release candidate, putting it through a pipeline, and then tagging a successful build as your release. In other words, the end-container is the immutable object that should be deployed, not the Dockerfile.

If you are building the container in each stage of your CI/CD pipeline, you are doing it wrong.

mychael• 7 years ago

This is an advertisement disguised as a technical post on container security.

bdcravens2• 7 years ago

It's a technical post on a company's blog. (which contains links to the company's products, but I didn't feel advertised to) Like thousands that have been posted on HN over the years.

nirvdrum• 7 years ago

Then I wish all advertising were like this. It's very informative and provides a solution rather than just pointing out the problem. I hope this page ends up ranking highly in search results because there are a lot of incomplete Dockerfiles employing questionable practices that sit at the top of search results and proliferate due to cargo culting.

obalado• 7 years ago

its an ad: https://pythonspeed.com/products/pythoncontainer/ Production-Ready Python Containers is getting closer to release, but it isn’t quite ready. So sign up below to get notified when the template is available for purchase.

neves• 7 years ago

Isn't it ironic that he isn't pining down the docker version?

mistrial92• 7 years ago

not everyone is on a upgrade-daily churn, and should not have to be ! if you are externally exposed, sure, because security .. but really, isn't there some room here for different life cycles ?

tuco86• 7 years ago

I have spent a ridiculous time building this so I'll take the opportunity and share. It builds python wheel packages in a build container and installs them in an app container. Works great for cpython and pypy. Also allows to build for alpine and works for most other languages. We started to build basically everything that way.

https://gist.github.com/tuco86/67d84dfb27268b1faf05d2dbb1acb...

Ok, I kind of cheated and added the user just now. Sue me. Also posted this in the other Docker related news. Sue me again.

Perceptes• 7 years ago

Looks like the last line needs to be updated to have the server listen on 8080 instead of 80. (I'm guessing this is left over from before you added the non-root user.)

rzimmerman• 7 years ago

Building the wheels in a build container and copying in the wheelhouse is a great idea. Thanks for posting this.

diminoten• 7 years ago

"Broken" means "does not work". These examples do work. I'm annoyed by this incongruity. "Not sustainable"/"Not Forward Compatible", etc. would have been preferable.

ru999gol• 7 years ago

why is running as root in the docker a problem? Isn't the whole point of containers to isolate the container? So what is the difference in a container running root or a user? If there is, wouldn't that be more of a docker bug?

CameronNemo• 7 years ago

If you are in a UID namespace, it likely is not a big deal. But if you were ever to have something escape from a container, it would be a much bigger problem if it was uid 0 in the root namespace.

IshKebab• 7 years ago

So in other words it isn't really an issue.

miduil2• 7 years ago

The page the article is linking to http://canihaznonprivilegedcontainers.info is mixing up running as root with running docker with --privileged. Latter one renders Docker security to zero, but is barely required.

I’m not saying “go run your all your Docker images as root”, but this is clearly FUD.

Non-privileged containers are still having "root", just with way fewer capabilities (See Docker [0] docs).

I’m not an expert, but I guess depending what you are doing the most problematic capability might be AUDIT_WRITE, because it is not namespaced and could be abused for DOSing syslog. But you might require it for things like sshd, sudo, adduser, passwd, …

Depending on how you are holding it the NET_BIND_SERVICE and NET_RAW can be an issue (depends on how your docker network looks like), but the others appear not to be a security issue per-se.

This page [1] gives a good overview on default capabilities, though they are also confusing to the reader with "better disable this".

I've created an issue, not sure if I have resources to fix their page though. [2]

[0] https://docs.docker.com/engine/reference/run/#runtime-privil...

[1] https://www.redhat.com/en/blog/secure-your-containers-one-we...

[2] https://github.com/mhausenblas/canihaznonprivilegedcontainer...

itamarst8• 7 years ago

Non-privileged containers running as root are a definite security risk.

Real world example: CVE from February 2019 which allowed escalation to root on host. It's preventable by (among other things) "a low privileged user inside the container".

See https://blog.dragonsector.pl/2019/02/cve-2019-5736-escape-fr...

Too2• 7 years ago

A root user inside the container can cause more havoc inside the container. One issue is that you might have sensitive data accessible from within inside the container. A hacker replacing system binaries inside the container might also affect how your application inside the container runs, a read-only user makes this much more difficult.

You can't assume all containers have same access and trust-level as a user because at some point you also have to process your requests and store your data - and that processing is most likely also be done in another container (even if your actual data-volume is mounted outside the container). For such containers same security measures as for a physicial machine applies.

geggam• 7 years ago

It seems like the folks making docker files could stand to learn package management with a mature package system before making docker files.

I wonder how many use docker after they learn ?

adrianmonk2• 7 years ago

I feel like programmers often fail to grasp the distinction between example code and production-ready code.

On one project, we made some example code available, and people would copy and paste it into their project, change a few lines, and launch it into production. Then they were surprised it didn't handle this or that situation or deal with this or that detail.

Yeah, no shit it doesn't handle those things, it's example code! You're supposed to read this code along with the documentation so you can get a gist of what the API is like. It's a learning aid, not a software deliverable. Your real code is going to be more complicated. This simplified code exists to get you past the "how the hell does all this fit together at a high level?" hurdle faster. Once you're over that hurdle, you can start on the real implementation.

thrower123• 7 years ago

Hopefully there is documentation; often there is not. It's borderline criminal that nearly all examples omit error-handling code, at least for the broad-strokes classes of errors that you, as the library author, know can and will occur.

I hated Java's checked exceptions, but it did let you have some idea what kind of errors were expected to be possible. Most of the time I'm excavating deep layers of source code, monitoring and logging to see what the errors might be, and tossing in catch-all handlers to keep the stack from unwinding too far if at all in doubt.

adrianmonk2• 7 years ago

Yeah, I agree with that. That is usually hard-won knowledge, too, so if it doesn't get transferred, that creates a lot of pain.

Unfortunately, a lot of people hate writing documentation and just want to get it over with as quickly as possible. (Ability to create working software doesn't necessarily correlate with ability or desire to share knowledge.)

Also, if you look at it in terms of incentives, it's tricky to reward people for making sure the documentation of gotchas is reasonably complete. By definition, these are things that people would probably not see until they're pointed out. So it's hard to tell if someone has really done it. (And when people are busy, things that aren't rewarded are unlikely to happen.)

dkarl• 7 years ago

I wonder if people who complain about this have thought about what it would be like for beginners to only get to see production code. Sometimes the cow needs to be spherical.

ptyyy• 7 years ago

This reminds me of that image of how to draw an owl. People forget that everyone starts somewhere and there is no magical jump from beginner level code to production code. Having good, well-explained examples of the beginner and intermediate were incredibly beneficial to me.

kissgyorgy• 7 years ago

We solved this by explicitly making the code examples non-copy-pasteable at all, for example include strings like "<put your URL here>" and things like that, so they won't even work by accident.

est2• 7 years ago

One more broken part is

    CMD [ "python", "./yourscript.py" ]

This breaks if you want to debug yourscript.py on startup. Better use a sh to wrap it.

deathanatos2• 7 years ago

Why/what do you think this breaks / what does wrapping it in a shell do for you?

E.g., for me, the following Dockerfile:

  FROM python:3

  RUN pip3 install ipdb

  COPY test.py /test.py
  CMD ["python", "test.py"]

where test.py is:

  import ipdb; ipdb.set_trace()
  print('Hello, World.')

run as `docker run -ti --rm $IMAGE_ID` works as expected:

  » docker run -ti --rm 52e98c118dc3
  > /test.py(2)<module>()
        1 import ipdb; ipdb.set_trace()
  ----> 2 print('Hello, World.')

  ipdb> p globals()
  {'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <_frozen_importlib_external.SourceFileLoader object at 0x7f809a8c8278>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, '__file__': 'test.py', '__cached__': None, 'ipdb': <module 'ipdb' from '/usr/local/lib/python3.7/site-packages/ipdb/__init__.py'>}
  ipdb> ^D

  Exiting Debugger.

  »

est2• 7 years ago

Can you add extra environ before python executes any code?

quickthrower2• 7 years ago

Nice eBomb. Describes the problem, offers value, does it politely so it is HN (and other places) friendly, then "there are still problems..." and then the paid solution.

I've seen docker images that do a git clone from the master head to get the source, so basically if their Github account gets hacked. You're f'd.

myroon5• 7 years ago

This Dockerfile linter would warn you about multiple of these problems and more: https://github.com/hadolint/hadolint

vorticalbox• 7 years ago

I would move the requirements.txt and pip install to after the user creation seeing as you'll invalidate that cache if your requirements change.

Best part is that was brought up as an issue in the article only to do the same thing in an example

hayd• 7 years ago

Running pip with sudo doesn't seem a great idea either...

zingmars• 7 years ago

I feel like the better solution to #4 is setting up UID namespacing for docker instead of (just) creating random users within the container. Even if you create a user, it's still going to run as whatever UID it has within the container (probably 1000 is most likely your UID if you're the only one using said system)

DoctorPenguin• 7 years ago

Isn't the example on how to make the referenced file better another contribution to the pool of "broken by default" images? Either that or I don't get the argument.

Simple Dockerfile examples are often broken by default

Related Stories