Dissatisfied with Docker

https://robert.ocallahan.org/2019/09/dissatisfied-with-docker.html

by khuey • 7 years ago

226 223 7 years ago

robert.ocallahan.org

chucky_z2• 7 years ago

I am currently on this train.

Having used rkt in the past, I went to revisit it recently only to find this: https://www.cncf.io/blog/2019/08/16/cncf-archives-the-rkt-pr...

I am so extremely disappointed in the CNCF as rkt (at the time, at least) seemed to be more "production ready" than Docker.

Are there any real alternatives? Is the answer "find something else that uses containerd in a more friendly way?" Is the answer "try to use podman/buildah, which are weird in their own way?"

roca5• 7 years ago

Podman looks interesting at a quick glance, though not supporting docker-compose makes migrating nontrivial for us.

devmunchies2• 7 years ago

and my attempt to get it run on mac were fruitless. I would have to do it in a Vagrant VM like 5 years ago.

3131s• 7 years ago

It was discussed recently in the podman Github issues that this functionality will be covered by the separately maintained podman-compose, which was recently transferred here:

https://github.com/containers/podman-compose

l_t• 7 years ago

Maybe containerd [0] is an alternative? It's being used in k3s instead of Docker.

[0] https://containerd.io/

edit: I didn't notice that you mentioned this in your comment, my bad. Can you explain why it doesn't work for you? I'm new to the Docker-alternative scene and thought it looked pretty good at first glance.

chucky_z2• 7 years ago

Docker uses containerd.

Here's an image straight from a blog post from last year: https://i1.wp.com/blog.docker.com/wp-content/uploads/2018/09...

https://blog.docker.com/2018/09/join-the-beta-for-docker-eng...

angry_octet• 7 years ago

Wasn't that because RedHat bought CoreOS, and RH is all k8s? And then no one was pushing rkt along with commercial support, so it died?

Agree the rkt was cool. I just don't know if RH is becoming more like MSFT or more like Oracle.

bionsystem• 7 years ago

More like IBM it seems :)

dharmab3• 7 years ago

We're implementing CRI-O so we can use Docker and Kata Containers side by side for different workloads. We're hoping if things go well we can fully migrate off Docker after a few years of transition, but it's in early stages.

syrusakbary6• 7 years ago

The article is completely on point. Because of all the reasons exposed there (and a few more) I started Wasmer, a new container system based on WebAssembly - https://wasmer.io/

Here are some advantages of Wasmer vs Docker:

* Much faster startup time

* Smaller containers

* OS independent containers (they can run in Linux, macOS and Windows)

* Chipset independent containers (so they can run anywhere: x86_64, Aarch64/ARM, RISC-V)

scanr• 7 years ago

Wasmer looks interesting.

I couldn’t read much as I get an “unexpected error” as soon as I start scrolling. It does say that it’s not optimised for mobile.

syrusakbary6• 7 years ago

Thanks for reporting it!

We will investigate and try to fix it ASAP :)

PS: if you could provide your device info (os and browser) it would be awesome

anaphor2• 7 years ago

One thing I also find sorely lacking in Docker is the ability to run your containers with the appropriate seccomp privileges (in order to enforce Principle of Least Authority). I know this is possible with Docker, but it's not really done much in practice because of various difficulties. I wonder how difficult it would be to do that with your tool?

syrusakbary6• 7 years ago

Since Wasmer is in control of all the syscalls its actually quite easy to manage the privileges in a more fine-grained way (think on CloudABI permissions on top of your containers)

nwmcsween3• 7 years ago

Seccomp is a horrible security model for containers, if the application or it's libraries use differing syscalls the seccomp ruleset is invalid

pbar3• 7 years ago

You might find the Kubernetes offer interesting. Pod Security Policies[1] expose seccomp and selinux configuration in the form of annotations.

[1] https://kubernetes.io/docs/concepts/policy/pod-security-poli...

ctidd• 7 years ago

Looks like an interesting project. I know the website isn't responsive, but adjusting the current meta viewport declaration would go a long way while you're working on a full solution. The minimum zoom of 1 makes it unusable in practice right now on a small mobile viewport due to the amount of panning it forces you to do.

gravypod3• 7 years ago

Is there a feature by feature comparison of wasmer to docker? I don't think Wasmer, while interesting, is a container system. It looks more like a bytecode vm.

Things I use and love from docker that to me feel "container"-y:

1. OSs as a library (FROM alpine:3.9)

2. Network namespaces so all applications think they are running on a machine with port 80 available

3. Service discovery through DNS

4. CPU and memory share allocation

5. Volumes (bind, temporary, remote)

6. docker-compose for single-command & single-tool development environments (all you need is docker and an internet connection)

syrusakbary6• 7 years ago

Wasmer is an application-based container while Docker is a OS-based container.

Because of that some of the things that you posted are a bit hard to compare.

We believe that by having a VM (based on a industry adopted specification such as WebAssembly) we can control much more granularly both execution (CPU, memory) and the runtime interoperability with the system (networking, file system, ...), solving most of the issues that Docker containers have

roca5• 7 years ago

Wasmer looks cool but FWIW it wouldn't address our (OP) needs. For one thing, in our containers we dynamically generate x86 machine code and execute it (yes, it has to be x86). Our needs are a bit special of course.

techntoke5• 7 years ago

Some things to note:

- Wasmer isn't available as a package on most distros from what I can tell. Instead they want you to curl a script directly to your shell, which is bad practice.

- They do not have a reproducible benchmark or a test suite to prove that it is actually faster than Docker and it is likely you can't even compare them. The Docker daemon and containers generally have very fast startup times.

- Docker containers are OS independent as well, but rely on Linux kernel features so they run more native on Linux. This isn't a negative. Most microservice apps are Linux-based.

- Adding abstractions for the chipset has always been slower than optimizing and compiling for multiple architectures. This is like someone saying Java is better because it provides a VM that translates everything. Instead Java has a reputation for being clunky and slow because of it. Well built apps and pipelines can easily be compiled for multiple architectures. See Alpine Linux for example.

jdnenej2• 7 years ago

This sounds like a great idea. Docker on ARM is such a pain if you aren't building everything yourself because half the packages are x86 only

geerlingguy• 7 years ago

More than half, I’d say 90%.

alpb3• 7 years ago

Nearly all points (daemon-less runtimes, daemon-less builds) the author has mentioned has been addressed by open source ecosystem in smaller standalone projects that are available.

Docker provides a nice high-level layer for the end user on the developer machine. Its CLI is efficient, so are the builds when they're local, with a cache. I still use it on my dev machine despite being aware of the alternatives.

Anyone security-aware company using containers in production most likely go with non-Docker approaches. Some notable examples include runc, Podman, Buildah, img, ko, Bazel/distroless.

solipsism2• 7 years ago

Distroless is not non-Docker, it's very much Docker.

alpb3• 7 years ago

Please explain.

You can build images using Bazel with distroless as the base image. Similarly I think Jib/ko use distroless images without needing a docker engine.

zests• 7 years ago

I'm newer to the Docker scene but haven't really found any of the complaints in this article realized in my work. Faster speed would be nice but I don't really mind it now.

I see a lot of complaints about the docker daemon and root privileges on HN and I've tried to understand where they are coming from but I can't get anywhere. For instance, I understand the reasoning behind "if there is no need for a daemon there shouldn't be a daemon" but I don't really understand what the actual/realized/tangible benefits of not having a daemon would be.

roca5• 7 years ago

We have observed the daemon getting into a bad state and stopping responding to container create requests, wedging our production system. I guess this kind of bug could happen with a non-daemon approach if there's shared state, but it seems less likely.

Not using any components with root privileges obviously makes things more secure.

Another benefit of not requiring a daemon or privileges is that then it should be easy to manage fully independent sets of containers. In particular a container should be able to manage its own nested containers in a straightforward way. This is painful in Docker.

wokwokwok• 7 years ago

It would be faster.

I also don’t really care if a container take 2 seconds or 100ms to start... but building docker images is painfully slow.

I’ve also ended up (numerous times) with the “docker daemon is borked” situation, which requires a restart to fix... and you can imagine how that sucks on a prod or multi tenant systems.

roca5• 7 years ago

One reason we care about containers taking seconds to start is that our CI tests have to start and stop a lot of containers, and it all adds up. And running them in parallel wouldn't help because they would just bottleneck on the global docker daemon... or break entirely by interfering with each other, unless we run multiple Docker-in-Docker setups, which would be even more painful and add its own overhead.

krferriter2• 7 years ago

I've also run into the daemon crashing on production systems and taking down all of the other containers on that host. I end up just killing the host and restarting the jobs on a newly provisioned one. There's a lot of overhead. If they could get it to run without requiring a persistent (and root) shared daemon that would be a big improvement.

shaklee3• 7 years ago

I think a lot of this depends on how you set up the context in your build. I've seen a lot of people just set the context to a specific directory with no ignores. The docker daemon by default tars up the entire context directory, which can be multiple GB. However, as you point out, even if you fix that problem, for some reason many of the stages take a very long time, even if they are just copying a small text file.

lmeyerov• 7 years ago

We hit security concerns here in two directions:

-- When our sw gets deployed on-prem by our users (or in their cloud, or wherever), our customers rather not deploy our SW with root for no good reason. Some of them have to do things like fill out forms ahead of time or on use due to this (!).

-- In turn, when we run third-party code, say as a normal dependency or a user-submitted script, we have no reason to trust it. That means we want to limit what it can do, and especially if something breaks out, have defensive layers. (Browsers & NodeJS are Technology From The Future in this regards.)

Note that almost all "docker exploit X FUD" articles, in fine print, make assumptions around running docker as root / privileged / etc. It's 2019, we know better.

Also, we happen to run an increasing amount of our stack as GPU code, which adds a (not so) funny layer to all this.

james_s_tayler7• 7 years ago

I didn't consider the on-prem use case, but I can see how that could become an issue.

aledalgrande2• 7 years ago

These are real problems with docker, but do we wanna talk about docker for Mac? A total performance disaster.

https://github.com/docker/for-mac/issues?q=is%3Aissue+is%3Ao...

thosakwe2• 7 years ago

I have no idea how, but Docker for Mac was taking up a whopping 65GB of space in one of the cache folders in ~/Library, although I had uninstalled it months ago. I wish I had taken a screenshot, but I was honestly dumbfounded. I had used it maybe once, ever.

EDIT: It turns out I’m not making this up/an edge case, at least 940 people have run into this too: https://github.com/docker/for-mac/issues/371

etimberg• 7 years ago

I have that problem too. Solution seemed to be to set up an alias to clean out old images, volumes, etc and run that every so often

semerda2• 7 years ago

Yap I confirm. 64 GB on my mac.. what da! Proof: https://www.flickr.com/photos/semerda/48720232588/

devin• 7 years ago

Seriously. It's absolute trash. I don't even want to think about how much time I've lost waiting for startup and teardown operations.

Edit: Before you downvote me too heavily, go give the issues list a thorough read. Some of the problems are quite bad. Perhaps I could have said it in a nicer, but really, it's bad, and it's been bad for a good long while now.

djsumdog3• 7 years ago

I hate how the Docker team called it native. Docker for Mac/Windows still run in a hypervisor because so much of Docker is specific to Linux/cgroups. There was a FreeBSD port of Docker that attempted to implement a lot of the Docker API using zfs+jails but it went unmaintained and was never ported to the newer modular Docker implementation.

You're always going to get that performance hit with the hypervisor layer there.

charles_f• 7 years ago

Docker for Windows is coming to use the WSL2, beta is already out.

teraflop• 7 years ago

Docker for Mac/Windows still run in a hypervisor because so much of Docker is specific to Linux/cgroups.

As I understand it, that what you describe is true of Docker for Mac, but not Docker for Windows which uses Windows' built-in container support (analogous to cgroups).

https://stefanscherer.github.io/how-to-run-lightweight-windo...

smudgymcscmudge2• 7 years ago

I preferred the old mac solution where they didn’t try to hide the vm from you. It was much more straightforward.

dawnerd2• 7 years ago

I just recently switched to using vscode with the remote ssh plugin and setup an ubuntu vm on my windows machine and performance is like 100x better.

Docker for mac is basically dead to me now, going to finish migrating my projects over to the VM and uninstall it from my laptop.

yoube• 7 years ago

Have you tried running the Vscode + Linux Subsystem + Docker on Windows stack? It's been pretty good so far (a good writeup on this is at https://nickjanetakis.com/blog/setting-up-docker-for-windows...)

EamonnMR• 7 years ago

For all of it's slowness, and constantly filling my hard drive with old builds because that's the workflow for some reason, it solves the real problem of MacOS constantly breaking packages. That alone makes it worth it.

james_s_tayler7• 7 years ago

Yeah. I have to run docker system prune from time to time.

par• 7 years ago

Docker for Mac is a performance nightmare. It requires state of the art $3400 macbook pro to run properly, and even then still takes seconds (or 10s of seconds) to spin up, start local servers, etc. Build processes that would've run locally take 3-10x longer running on docker for mac. The whole thing can be quite frustrating.

month13• 7 years ago

Wondering if it's worthwhile just rolling my own vm and run whatever i want in there. I'd lose all the ability to bind to my host's ports, but would be able to control the daemon a bit better.

atarian2• 7 years ago

I just want to be able to save a container binary to a USB drive and then run it from a different computer without having to install anything.

lytedev2• 7 years ago

There are a number of ways to get this kind of functionality. Can you expound on "different computer"?

juandazapata• 7 years ago

What part of "different computer" is confusing to you?

TheChaplain• 7 years ago

I believe that already exist and is called "AppImage".

astockwell• 7 years ago

The first time I emailed a buddy a Go cross-compiled binary and he opened and it ran no problem (opsec aside), our worlds changed.

atarian2• 7 years ago

This is exactly how I'd like containers to be as well.

m_sahaf• 7 years ago

You're looking for binctr: https://github.com/genuinetools/binctr

techntoke5• 7 years ago

Installing Docker or another container runtime is easy. I'd suggest creating a portable OS that you can boot from USB, or creating a local registry. You may also want to consider just creating a script to bind mount the binaries.

est2• 7 years ago

All I want is a process that can be frozen and copied to multiple servers.

start/stop is merely a state change.

lytedev2• 7 years ago

This is interesting... May I ask why this is an important feature to you?

est2• 7 years ago

If you think about it most "Service", web or rpc, can be scaled in this way.

blacksqr• 7 years ago

DMTCP (Distributed MultiThreaded Checkpointing)

http://dmtcp.sourceforge.net/

ethagnawl• 7 years ago

This sounds a lot like the Smalltalk VM.

ehotinger• 7 years ago

That's quite an oversimplification of the problem, copying stack/heap/etc. Anyways, check out CRIU as a starting point.

e12e3• 7 years ago

(and note that lxd/lxc allows migration based on it. Not sure about copying - I suspect you run into similar issues as fork - who owns the open file handles and other resources?)

soraminazuki• 7 years ago

You’d be much better off using a VM if you want snapshots. Process snapshots requires that you can:

1. extract all kernel state related to the target process

2. replicate the state on another machine

Both of these are extremely complicated problems.

The first problem is difficult because it’s nearly impossible to isolate all relevant state in the kernel. Process state is closely intertwined with the entire kernel state, making it all too easy to produce an incomplete snapshot that can break the system in all sorts of ways. Worse, things would get more complicated if multiple processes are involved.

The second problem is just as hard if you consider external dependencies or different kernel versions. It might simply be impossible to recreate the process even if you did succeed in creating the perfect snapshot.

Process snapshots are simply not worth the trouble.

djsumdog3• 7 years ago

These are all pretty good points. I can understand why Docker allows any base layer OS, but they could have made their own packages or limited a single distro and it would be easier to check for outdated packages and security issues in containers.

The cgroups and Linux specific hooks keep Docker from being implemented natively anywhere else. The fact you have to share the entire Docker socket for containers to be able to control other containers, or that it's not trivial to run Docker-in-Docker, is terrible.

I did a similar writeup about the things I hate about container orchestration systems:

https://penguindreams.org/blog/my-love-hate-relationship-wit...

krferriter2• 7 years ago

We were using Docker-in-Docker in our cluster to run user-defined containers, and something about it resulted in the outer container disk usage blowing up to like 20GiB per container. I think something due to the overlay filesystem driver. We were under a time constraint so we just grew all the host root filesystems, but it was pretty inconvenient and not inexpensive to do so. I have to assume this also had a drastic negative impact on storage I/O performance in the inner container as well.

geggam• 7 years ago

Docker in Docker feels like a root kit generator to me

imiric2• 7 years ago

The fact you have to share the entire Docker socket for containers to be able to control other containers, or that it's not trivial to run Docker-in-Docker, is terrible.

FWIW, if you enable the remote API, which, granted, isn't as trivial to do securely as it should be[1], then you can connect from any Docker client by simply setting the `DOCKER_HOST` env var and using the right TLS certs. This makes Docker-in-Docker much easier to manage, avoids the security issues of sharing the Unix socket, and works remotely of course.

[1] https://docs.docker.com/engine/security/https/

djsumdog3• 7 years ago

I created an ansible role that does this for me:

https://github.com/sumdog/bee2/blob/master/ansible/roles/doc...

It creates client certs and copies them locally too, so I can connect to Docker remotely over a VPN. Still this doesn't solve the original problem I talked about. It's not about securely connecting to the daemon. Even if you connect securely, you still essentially have root access on the host machine.

I've considered writing a proxy that restricts what commands can be forwarded on to the Docker host socket (e.g. allowing for container IPs x,y and z to restart containers, but not to create new ones or pull images). There doesn't seem to be fine grained security or roles built into the docker daemon itself.

Running docker in a docker container would give you a throw-away docker to use for things like Jenkins, Gitlab-CI, and other build tools without giving it access directly to root on the host.

devmunchies2• 7 years ago

Do BSD jails solve these problems? The little i know of them, they are good at containerization but don't solve the distribution problem that docker containers do.

nickjj4• 7 years ago

I've been using Docker since 2015ish and the container start up / stop speed is really the only thing that bugs me.

Everything else is fine for day to day usage IMO (on Windows and Linux at least) and very much worth the trade offs, but having to wait multiple seconds for your app to start is tedious since it plays such a heavy role in both development and even in production. Each second your app is not running is downtime, assuming you're not load balanced.

I opened an issue for this almost a year ago but not too much has come from it other than identifying there was maybe a regression in recent'ish versions: https://github.com/moby/moby/issues/38077

We're talking multiple seconds in Docker vs ~150ms as a regular process to start up a typical web app process like gunicorn in a decently sized Flask app (many thousands of lines of code, lots of dependencies -- a real world app).

james_s_tayler7• 7 years ago

I take it you've never worked in an enterprise Java shop with a few million line legacy code base deployed to whatever the hell webserver I can't remember the name of right now.

Seconds is great in comparison. Minutes, not so much.

cs02rm0• 7 years ago

JBoss, Websphere? I know that pain, hadn't even occurred to me that anyone would think containers had a slow startup time. Especially as they've often replaced VMs for me.

TheWizardofOdds3• 7 years ago

The long startup time is somehow what keeps me from really liking Docker. In the end (if I understood correctly), it is supposed to be the go-to tool for serverless architecture. If my serverless function needs more then a second to startup, it's not usable for me.

Even the hello-world container, which is only a few kB in size needs roughly a second to startup.

utopian3• 7 years ago

In the end (if I understood correctly), it is supposed to be the go-to tool for serverless architecture.

You misunderstood. No one calls Docker a “serverless” architecture.

_hao• 7 years ago

I'm working on a web service that takes 4-5 minutes to startup because of all the shit it has to warm-up. I'd be happy if it takes even a minute IF I could port it to a newer tech stack and can containerize it.

croo• 7 years ago

Few seconds are my dream coming true. Try working on an application where every deploy is a coffee break... :(

xorcist2• 7 years ago

Starting the process probably isn't what's taking so long though.

StreamBright2• 7 years ago

Firecrackers are fixing that.

https://firecracker-microvm.github.io

hbt• 7 years ago

- start the container with docker run and run sleep 999

- then start your application with docker exec

you will have instant execution.

Only restart your container if the environment changes (new build etc.)

For development, mount your source code in readonly.

aloknnikhil• 7 years ago

In fact, a global system daemon should not be needed. Either users should be able to run their own daemons or container management should avoid having a daemon at all, by storing container state in a shared database.

Absolutely love Podman. You can even define registries to work with docker hub, easily.

https://podman.io

jsilence2• 7 years ago

Interesting!

Are there management tools like Portainer available? Or would, even better, Portainer work with Podman?

nullify88• 7 years ago

Regarding portainer working with podman, not yet it seems. Here's a gh case for it https://github.com/portainer/portainer/issues/2991

nikisweeting6• 7 years ago

Yeah but no docker-compose support :/ dealbreaker for us. Theres an alpha podman-compose project but we couldn't get it running.

likeclockwork• 7 years ago

You really don't need it since podman has pods of containers that communicate with each other. You can export and playback the pods as kubernetes yaml.. but I just use a bash script + an environment file to configure, create and launch pods. What else is docker-compose really doing for you?

dharmab3• 7 years ago

Both Podman and CRI-O support Pods, which can fill the same use case as docker-compose

ses19843• 7 years ago

There are two kinds of software: software no one uses, and software people complain about.

notyourday3• 7 years ago

There's also software that was created to solve a specific problem that got misappropriated to do something else

There used to be an ISP called pilot.net. It was a crappy ISP but to solve its ISP billing problem it wrote a billing system for telcos.

There was a company that tried competing with AWS selling hosting based on hyperthreads of CPUs. It wrote its own provisioning system because it did not want to pay for Virtuozzo. The company's name used to be dotCloud.

otabdeveloper4• 7 years ago

There's also software that was created to solve a specific problem that got misappropriated to do something else

Docker is a classic case. Docker must be the craziest, most over-engineered solution for packaging developer artifacts in the Universe.

oblio5• 7 years ago

What's the link between pilot.net and dotCloud? Did dotCloud use the billing system from pilot.net?

draw_down• 7 years ago

Mmm, very clever.

james_s_tayler7• 7 years ago

Why have I not heard this before?

Brilliant.

I'm going to file this right alongside "software is never finished, only abandoned"

ses19843• 7 years ago

I don't know if I would call it brilliant, it's pretty funny though. I didn't make it up.

Your quote is first said about art and attributed to Da Vinci.

e_proxus• 7 years ago

There's also the almost never used software that people complain about being forced to use.

Kaiyou2• 7 years ago

Also feature complete software that is regularly used by lots of people, but nobody complains about, because devs don't mess with it anymore. Think ls, grep, yes, ...

oblio5• 7 years ago

Devs can't mess with it anymore, you mean? There's a myriad alternatives to ls, grep, etc. Dunno about alternatives to yes, maybe there's a si/oui/da out there? :)

Sometimes the alternatives are considered superior to the originals (ripgrep) but the originals are just fossilized in place. In 2019 I don't even think we'd be able to have less as the pager instead of more, because we're so conservative about universals defaults, it's sad.

ses19843• 7 years ago

I will happily complain about grep. My complaints might not be valid but I'll happily do it. I hate that bsd and gnu grep are different. I can never remember when and which flag I need to pass when my expression has stuff in it the default can't handle. I don't like that it's slower than alternatives like ag. I wish it would automatically handle compressed data.

Also the saying is not as pithy if you add more kinds of software.

bob1029• 7 years ago

What's wrong with booting a VM off a standardized base image (e.g. an AMI), and then applying simple deployment scripts for each application you need to run? You could probably replicate 90% of the justification for using docker with some basic scripting.

  git clone https://github.com/myprofile/my-cool-app
  cd my-cool-app
  chmod +x deploy.sh
  ./deploy.sh

That's it. The above script would be responsible for getting your application runtime environment up and then getting the application running as a persistent service with reasonable defaults. Most cloud vendors let you put something like that in your VM startup configuration. All you would need from this point is to ensure that any sort of desired management functions are built into the app itself. Perhaps having a central management service it communicates with across the public internet would be a good place to start. You don't need a lot of tooling to make a huge impact here. If you own the codebase behind it, and have a fundamental understanding of your deployment techniques, you can easily pivot and embrace radical new approaches. If you are stuck on Docker, this doesn't seem as optimistic from where I am standing.

roca5• 7 years ago

Off the top of my head: two things in our (OP) case: 1) Memory overhead 2) Slow access to filesystem storage shared between containers

lacampbell• 7 years ago

This is what I ended up doing. Don't laugh, but I spent about two weekends furiously learning docker for a side project, only to replace it with a shell script of about 100 lines.

I'm sure containers have valid use cases for larger teams, but for one person I didn't see the point at all - especially as I use the same OS for dev and production.

samvher• 7 years ago

Docker is not only about having reproducible environments. I mostly work as a single developer and I use Docker all the time, mainly because I like the isolation properties and because it's super easy to launch certain services (e.g. set up a postgres database). In the past really often I would have issues with conflicting dependencies, that's all gone now. The fact that your shell script did the job for you is obviously a good thing, but maybe that just means that in this particular situation Docker didn't have much to offer for you.

james_s_tayler7• 7 years ago

I use docker all the time when doing side projects by myself. It's nice because I don't know ahead of time what the dependencies will be but I don't really care. I just pull in all the container images via docker compose and suddenly everything is there and talking to each other.

I find it very ergonomic outside the need to run docker system prune from time to time.

I also value it highly when some open source project I want to play with has docker compose in its readme. Super easy just to spin it up.

jordanthoms• 7 years ago

That approach works, but if you are going to go into production with it you might need to support:

- Rollbacks

- Automatic deployment of new releases from CI, rolling releases

- Healthchecks, detecting if/when the server exits and making sure that VM gets killed

- Canary deployments

- Autoscaling (could use an autoscaling instance group for it, but what if you need to scale based on other metrics)

- Log aggregation

- Monitoring

- Service discovery, load balancing between services

- Service mesh

- Fast startup of instances (starting a fresh VM and waiting for it to setup everything from scratch could take ~3-4 minutes, docker is in the ~30s or less range).

- Bin-packing (run a high-cpu low-memory workload colocated with a low-cpu high-memory workload for maximum efficiency)

etc, etc, all of which you get either out of the box or without a huge amount of work if you adopt something like Google Kubernetes Engine (which takes a few clicks to spin up a cluster).

If you don't need any of that stuff and don't want to learn Kubernetes, it's totally justifiable to go that way, but personally I would take Google Kubernetes Engine over something like that any time. There's some up-front cost learning how it works which then pays off very quickly.

KaiserPro3• 7 years ago

None of these features are a function of docker, they are a function of the orchestration layer.

bin-packing isn't what you describe, its shoving as much stuff on a machine as possible. Proper resource dependencies management allows what you describe, something k8s is weak on compared to other orchestration systems.

ay• 7 years ago

What about hashicorp’s nomad ? In my experience it does pretty much all of the above at a fraction of the complexity...

techntoke5• 7 years ago

This is way slower than Docker. Rootless containers are already a thing. If you still want VMs then see Kata Containers. Containers have a lot of benefits, the major one for me is reproducibility and application sandboxing. There is a reason containers exist and that every large cloud platform has adopted them, including Microsoft adding support in Windows.

notyourday3• 7 years ago

This is way slower than Docker.

add-job bake-instances --app=my-id --instances=200

Oh, you never wrote tooling and you dont have a jobbing engine? Well, that's the complaiing that you can't get an an Uber because you did not download an app. It is sort of funny because every single shop that is "oh containers!" either creates them manually ( slow and painfully ) or just uses random ones developers find.

There is a reason containers exist and that every large cloud platform has adopted them, including Microsoft adding support in Windows.

It is the same reason why Whole Foods sells sushi: that's teh way to sell salmon for $85/lb to hipsters. Hipsters hate buying salmon for $22/lb but they love the rolls.

spookthesunset• 7 years ago

Okay so write your own crappy (excuse me, graceful and elegant) pile of shell scripts that nobody in your company but you understands? Sounds good to me.

The second you leave, somebody is gonna rip that crap out and replace it with docker. All while mumbling under their breath...

james_s_tayler7• 7 years ago

This. Times 100x. This is one of Dockers biggest value adds.

eeZah7Ux• 7 years ago

Nonsense. Applications have a lot of complexity and therefore code around initialization. Adding a little script to deploy them is harmless.

It's also the approach used by various FAANGs operating at a large scale.

dharmab3• 7 years ago

Low efficiencies. When you work with 1000s of cores, a VM per app instances is literally millions of dollars annually of wasted CPU compared to using a container orchestrator.

broknbottle• 7 years ago

switch to podman?

https://podman.io/

nikisweeting6• 7 years ago

No docker-compose suppport.

ptman• 7 years ago

https://github.com/containers/podman-compose

k__2• 7 years ago

I saw it can be used as a drop-in replacement.

Does podman have better startup performance than docker?

Using a docker to debug and it is a bit of a pain.

neckardt• 7 years ago

I'm surprised nobody is mentioning LXC[1]. I'm by no means a containers expert, but they claim to be more secure since they default to running as non-root. Unlike docker, I had no trouble installing LXC with apt, while with docker I often got an outdated version. I'm now using LXC for all of my basic container applications (self hosting a wiki and a few other sites).

[1] https://linuxcontainers.org/

judge2020• 7 years ago

Docker isn't up to date on the official Ubuntu (and debian too?) repos for some odd reason. A better way is either snap (snapcraft) or adding docker's own apt repository.

FreeHugs4• 7 years ago

Aren't those images provided by Ubuntu and Debian? Then it would not be under the control of Docker which versions are provided.

docker.io provides Debian 10 as far as I can tell.

e12e3• 7 years ago

Docker (the daemon and tooling) is generally upstream-supplied (ie: docker provides apt repos etc).

Not uncommon for rapidly moving upstream. I wouldn't recommend running distro-supplied docker in general (and neither do docker).

Curated apt repos is nicer than snaps, IMNHO.

fiatjaf• 7 years ago

snap never works.

dbleyl• 7 years ago

lxc is awesome, wondering the same thing. Also has some niches in the kubernetes space. Using lxc for development DBs and private servers.

dvfjsdhgfv• 7 years ago

Because LXC has no marketing money like Docker. For me it's just one of these cool tools I use every day without giving it a second thought.

jsilence2• 7 years ago

How is the tooling compared to Docker? Genuinely interested. Are there platforms like Portainer available? Is something like docker-compose possible?

FreeHugs4• 7 years ago

How do you create an image for LXC?

With docker, they have the docker.io service with all types of distros you can chose from.

StreamBright2• 7 years ago

You can use the following:

    lxc image import metadata.tar.gz rootfs.tar.gz --alias hello

Not sure which part of the process are you referring to though.

rlpb2• 7 years ago

The project provides prebuilt images for all the common distros: https://us.images.linuxcontainers.org/

Use them in one command. For example:

    lxc launch images:debian/sid/amd64

Ubuntu, which I believe is the most popular image base in Docker, ships official cloud images, and you can also use these in one command:

    lxc launch ubuntu:bionic

jeff_vader• 7 years ago

One reason it's not very popular (from my experience): Docker Desktop for macs, which makes Docker experience on mac os feel somewhat close to native. Meanwhile, to use LXC you'd have to work inside VM (Virtualbox or similar). Last 3 companies I worked for (in London) were almost 100% mac os (which is a little sad).

MrBuddyCasino• 7 years ago

If you use "kind" to create a K8S cluster on MAC, using LXC is totally transparent. You can just load regular Docker images via "kind load docker-image". It even takes less RAM than minikube, because the running Docker VM is re-used.

aitchnyu• 7 years ago

How cross-platform is LXC containers? Somehow I still use Vagrant in my pet project, since I want Windows and Mac users to have a dev environment in a few steps.

notyourday3• 7 years ago

Have you heard of this amazing thing called "a VM"? It is like docker but when you pee at the wall it always splashes back right at you so you quickly learn that you never want to pee at the wall.

jbergknoff• 7 years ago

It's interesting to see the complaint about containers starting too slowly. I haven't seen much discussion about it before this article and this comment thread, but it's one of my biggest pain points with Docker. I know we can save some time by skipping some namespaces (e.g. `--net host`) but I've still never been able to get satisfyingly fast container execution in that way. (e.g. `time echo` -> 0.000s, `time docker run --rm alpine echo` -> 1.3s; come to think of it, this is even slower than it used to be)

Still, Docker is the best method that I've seen for distributing software, especially cross-platform. Not just for shipping a containerized web app to production, but also running dev tools (e.g. linters, compilers) and other desktop and CLI applications. I know some people run lots of stuff in containers (https://github.com/jessfraz/dockerfiles, probably the most prolific), but I think this is a largely underappreciated facet of Docker.

My team at work is heavily Mac while I'm on Linux. The dev workflows are identical on my machine, on their machines, and in Jenkins. Nobody has to have a snowflake setup, installing this or that version of Python, we're all just `docker run`ning the same image. It's great.

Unfortunately, Docker for Mac's IO performance is abysmal. If past performance is any indication of future results, that's never going to change. I'm constantly on the lookout for other ways to share tools cross-platform that don't involve Docker. Things like podman and binctr are exciting, and I've played with them, but I don't see them filling this niche.

anordal• 7 years ago

... then I suppose Selfdock is for you.

* Does not give or require root.

* Fast: Does not write to disk.

* Fast: Does not allocate memory.

* No daemon.

https://github.com/anordal/selfdock

aequitas• 7 years ago

Give up the idea of data volume containers. Given that volumes are the way to go, no other filesystems in the container need to, or should be, writable.

Interesting philosophy, it will pose some issues when replicating a Docker 'build' like system though. But that could be separated from the 'run' system with cached layers.

edit: Btw, Docker also seems to support this: https://nickjanetakis.com/blog/docker-tip-55-creating-read-o.... Also allowing explicit creation of writable tmpfs mountpoints.

k__2• 7 years ago

I read podman can be simply aliased to docker and it works. Is this the case for selfdock too?

KaiserPro3• 7 years ago

A few things that I would add to that list:

o No primitives to deal with secrets.

o Terrible disk handling (aufs was just horrid, overlay2 I think misses the point. device mapper is just, silly)

o poor speed when downloading and uncompressing images.

Of all of them, the most serious is the lack of secrets handling. Basically you have to use environment variables. Yes, you can use docker compose and stuff appears, but thats only useful if you use compose/swarm.

In this day and age, to have secrets as a very distant afterthought is pretty unforgivable.

antocv• 7 years ago

What do you consider a secret? What other system do you recommend which has the feature you want?

I cant see how one would inject data into a process without ptracing/gdb-attaching to it, messing with its internal memory, or just use environment variables or argument list. If you put it on a file somewhere, you still have to tell the process where the file is by an argument or env and also manage that "volume" and uid space of that file and process.

Kubernetes secrets have nothing to do with containers - it anyway usually exposes the secret as env var or volume and file.

heyoni• 7 years ago

I smell potential for a fun tutorial with that last part!

KaiserPro3• 7 years ago

A secret is anything that you don't want to be public. normally API keys, or some other similar token.

Its not even a case of stopping people attaching a debugger, there is simply not standard mechanism. Some sort of secret store should be a primitive in any container/VM/unikernel built since 2012.

The "recommended" way is to map a volume in to a container. But thats rubbish, there is no standard place to put it, and crucially no API to fill the volume full of secrets. How do I map the specific volume to a container? you can't its manual.

what would be nice is either a tagged value store, so you can specify a tag when you build a container, so that you have a way to signal which secrets your container wants. Or a lump of the container that you can encrypt, that has a key value store in it.

There are lots of documents telling you that you _shouldn't_ embed secrets in your container, but there are not many howtos that don't involve the engineering equivalent of gaffer tape and wet string.

Most of this has now been solved by better[] orchestration systems.

[] not great though.

xorcist2• 7 years ago

The most basic problem with Docker is the use of a daemon that is not init.

Access control is at best problematic.

Upgrading the daemon without losing state is tricky.

Requiring daemon access to build images is insane.

Building this functionality into something like systemd would be more robust but it's way harder to sell as a product.

Smithalicious• 7 years ago

Yeah Docker should get with the times and be assimilated into systemd like everything else

ahartmetz2• 7 years ago

Okay Lennart, if you say so.

paulddraper2• 7 years ago

the most serious is the lack of secrets handling. Basically you have to use environment variables.

That's how I always handle my secrets. (12 factor app)

Am I missing a better way?

fhgsfkcc• 7 years ago

secrets don't belong in env vars, because many languages and/or frameworks will happily dump all your env vars to users in the event of misconfiguration or some errors.

e12e3• 7 years ago

I don't think you're wrong - but you might want to look at docker swarm over compose (even on single nodes) and:

https://docs.docker.com/engine/swarm/secrets/

(as well as the sibling "config")

tigroferoce• 7 years ago

Beside technical considerations, the main point of Docker, to me, is its diffusion. Just like the actual physical containers it gets inspiration from, you can find Docker almost everywhere. It has become a Lingua Franca for devops, even in enterprise environments, and it will be very difficult to get rid of it.

peterwwillis• 7 years ago

Containers Without Docker: https://dzone.com/articles/containers-with-out-docker

Dockerless: https://mkdev.me/en/posts/dockerless-part-1-which-tools-to-r... https://dzone.com/articles/dockerless-part-1-which-tools-to-...

What's funny is, Docker is only actually useful because of how many features it has, all its supported platforms, all its bloat. You won't ever be totally satisfied with any alternative because it takes so long to make something like Docker, and someone will always need that one extra feature.

mschuster91• 7 years ago

In fact, a global system daemon should not be needed.

You will need a daemon running as root to bind to ports below 1024.

In addition, in many cases you want a bind-mount onto your filesystem that also supports arbitrary UID/GID on files, which means you will need a root daemon. The problem is of course that anyone having access to the docker daemon can simply say "bind host / to container /mnt" and then can hijack /etc/sudoers for a privilege escalation on the host.

It's mutually exclusive to have usable containers and a system that is secure against privilege escalation by the users at least and (in case of docker-in-docker implemented by bind-mounting the Docker socket in the container) by anyone accessing the container and achieving RCE there.

rotten• 7 years ago

I'd like it to be more like git. Yes Docker has "push" and "pull", but I want branches, and automatic attach when I do a checkout, and rollback when I screw it up.

As a developer I'd like to be able to check out a docker image, work _in_ it, and merge it to master when I'm done. When my code is deployed, I'll know _exactly_ what is running.

Managing secrets and environments (so the container knows when it is in production instead of running on a developer's laptop) is important to get this to work well.

It feels like it is half way there already. I'm looking forward to when it is as straightforward as git for a developer to use. I'm not too worried about start up time - the biggest drawback to slow startup time is when you are running very sensitive autoscaling that is tearing down and spinning up new nodes very quickly. If you have that problem you may want to rethink your node size, hysteresis, and scaling thresholds.

bokieie• 7 years ago

We tried to use LXC directly and realized that Docker simply does it all for us, and better.

Dissatisfied with Docker

Related Stories