The Oncoming Train of Enterprise Container Deployments

http://www.juliandunn.net/2015/12/04/the-oncoming-train-of-enterprise-container-deployments/

by whatami • 11 years ago

33 9 11 years ago

I'm keen on unikernels too. At work I've argued that we should build a Xen backend for Garden so that Cloud Foundry can drive unikernel apps on equal footing with buildpack staged containers, docker containers and .NET apps. Mind you, I argue for a lot of things at work, very few of which are ever picked up; probably for the best.

But for all the virtues of unikernels, an awful lot of software assumes a linuxy userland. Outside of consciously isolated universes like Java and .NET, it might be tricky to get traction.

As for devops, the missing bit isn't the sexy stuff -- the parts that locate and monitor processes running in containerised environments. Kubernetes, Mesos, Diego and the dozens of others popping up every week are busy solving that.

What seems to be forgotten is that, as this article mentions, you still need a sane build-and-verify pipeline. Tools like Puppet, Chef and the like can be seen as image-building toolkits. Buildpacks already know how to take various stacks and turn them into runnable images (on Heroku or Cloud Foundry). Docker is a step backwards, in some respects.

Disclaimer: I work for Pivotal, the company which donates the majority of engineering effort to Cloud Foundry.

mwcampbell2• 11 years ago

IIUC, you say Docker is a step backward from both Chef/Puppet and Heroku buildpacks. Why do you think it's a step backward from Heroku buildpacks? Because a Docker image is the whole userland, whereas Heroku buildpacks deliver application slugs that run on a base maintained by the platform? I see a couple of problems with the Heroku buildpack approach:

1. Because buildpacks don't run as root, they can't take advantage of apt and the whole universe of Debian/Ubuntu packages. So it's common to see buildpacks pulling binary tarballs from S3 and dropping them into the slug. That is a step backward to me.

2. The set of packages in the base platform, at least for the Heroku Cedar stack, is very ad-hoc. I get the impression that it was largely based on what was needed to get Ruby and common gems to work.

jacques_chester3• 11 years ago

I worked on the Cloud Foundry buildpacks team for a while, so I have seen the good and the bad of the buildpacks approach. But actually using Docker in anger has made me appreciate buildpacks more than I used to. Docker is step back in some respects, analogous to how lower-level languages are step back in some respects from higher-level languages.

As for your questions:

1. A buildpack is meant by design to be an opinionated builder for a particular stack, not a totally pluripotent environment. If you need total control, you can have it by using Docker and sending those images to run on CF.

The binaries you see being pulled in are either specifically built by Heroku for Heroku or by Cloud Foundry for Cloud Foundry. They're not random links plucked out of a Google search. It's not as though the Debian team aren't building binaries too.

2. The set of packages for Cloud Foundry in the "stack"[1] or rootfs is set by the buildpacks team, based on balancing the needs of various buildpacks consumers. Heroku do something similar.

It's ad hoc insofar as any extremely consumed library is ad hoc.

The concept of buildpacks is that you, the developer, don't have to care about what is installed. You throw code over the wall and it launches. For operators, the attraction is that there is a single known base environment and that it can be upgraded easily when CVEs are announced.

In theory a Docker container can be whittled down to the minimum dependencies needed to run the software, but it requires tedious and often difficult-to-understand effort, so very few people do it. And indeed the whole reason to go through that effort is to make smaller images. When you have a single reference image, that argument is largely moot.

[1] https://github.com/cloudfoundry/stacks

jzelinskie• 11 years ago

My understanding from talking to people working on MirageOS is that a lot of the unikernel users are still deploying inside containers. Just like containers aren't going to "kill" virtualization, unikernels aren't going to "kill" containers.

I'd particularly like to stress the part of the article that mentions containers that bring all of userland with them. I hope this will be fixed once people are using tools like Kubernetes and are forced to think of containers as pieces of a larger system rather than short-sightedly "porting" their applications into a container. At CoreOS, we introduced Clair[0] specifically to show people how much userland is still inside their containers and vulnernable to known CVEs. ~80% of tags (downloadable containers) on Quay.io still contain Heartbleed, for crying out loud.

[0] https://github.com/coreos/clair

jacques_chester3• 11 years ago

I hope this will be fixed once people are using tools like Kubernetes and are forced to think of containers as pieces of a larger system rather than short-sightedly "porting" their applications into a container.

Could you elaborate a bit? I don't see how the scheduler changes the contents of a container image.

mwcampbell2• 11 years ago

I think the Alpine Linux image for Docker is a good solution to the point about userland bloat. Much better than unikernels, which would entail giving up all the benefits of running containers on a shared, mature OS kernel, particularly with regard to performance, resource pooling, and debuggability. I'm still waiting for bcantrill, who actually has experience with large-scale production systems (unlike me), to expound on why he believes unikernels are a terrible idea, as he said in a brief rant here:

https://www.youtube.com/watch?v=Ya6h2zKlpaQ&t=5110

shykes• 11 years ago

Personally I love both: minimal, secure-oriented distros like Alpine Linux, and unikernels. They will both play an important in building robust, scalable distributed systems.

simonvdv• 11 years ago

Even though using Alpine as the base image for a container is a lot beter/cleaner than other base image imho we shouldn't rely on distro package management inside containers.

Not only does running a package manager inside the container mean you'll need to satisfy its dependencies in your image you also increase the image's attack surface compared to an image without a package manager.

Ideally we'd have a simple way of installing stuff into images from the outside so you can always start `FROM scratch` and add the minimum deps you need to run your app. Adding stuff could be as simple as extracting tars with the tar's contents following the Filesystem Hierarchy Standard. Each tar could be a layer so it matches well with how Docker images work as well.

Since it isn't possible to extend the Dockerfile syntax I started prototyping a static binary written in Go to add to `scratch` to do this. It worked better than I expected :) Only thing I couldn't find was a distro that packaged it's packages this way and it would obviously suck to create another packaging standard.

The Oncoming Train of Enterprise Container Deployments

Related Stories