Docker Compose best practices for dev and prod

https://prod.releasehub.com/blog/6-docker-compose-best-practices-for-dev-and-prod

by brycefehmel • 4 years ago

398 158 4 years ago

prod.releasehub.com

kh_hk8• 4 years ago

docker compose is unnecessarily nerfed for using in development. I swear every company I work with has this problem and I always end up carrying along my hybrid bash-to-docker-compose cmd line tool that makes the dev experience under compose nice.

Did you know that compose accepts stdin for `-f` files? Anything that outputs yaml is a valid "compose provider". Self plug, more on that, https://eskerda.com/complex-docker-compose-templates-using-b...

jayd16• 4 years ago

So whats the complaint? Compose seems pretty easy to work with to me.

kh_hk8• 4 years ago

For me the problem is that yaml (and the compose specificatio n) is not self sufficient to support complex scenarios. You will start by setting things and discovering that ENV vars * on the global scope of the compose specification are not supported so you end up having to template and output yaml.

Try having 40 microservices and quickly iterate between fixing bugs in them. Some you want mounted, others you want running unmounted. I understand the premise is already broken, you do not want 40 microservices, but that's not a precondition you can fix without a time machine.

[[* EDIT: by env vars on the global scope, I mean setting things on the global scope by using an env var, say, you want a certain volume named after an environment variable. You can only use ENVs as values.

    # this works
    foo: "${SOME_ENV}"
    # this does not
    ${SOME_ENV}: "foobar"

pkrumins4• 4 years ago

Docker-compose just works and my team gets shit done. Does your team get shit done? If they do then what are you complaining about? If it hits production then it’s all good!

kh_hk8• 4 years ago

All I am saying is:

* docker compose could be better

* docker compose does not improve that much

* by mixing docker compose with templates you can get complex stuff going, but is not a much advertised feature.

Most of the "cool dev tool env" that appear are based on kubernetes, or something that is _not_ docker compose, which makes them difficult to adopt. I would like docker compose to improve on a direction that makes development easier, that's all.

fio_ini7• 4 years ago

You can accomplish the same thing using environment variables. You can pass them in at build time or deploy time. `ENVIRONMENT=dev docker-compose up` or `docker-compose build --build-arg ENVIRONMENT=dev` and in your case maybe you write some function `find_answer() { something-async ... }; ENVIRONMENT=$(find_answer) docker-compose ...` You can also read in .env files... Which in my opinion is cleaner since you can check the configurations into source control for YAML or .env files which is easier to read than hundreds of lines of bash and it's esoteric syntax. I love bash but I'm just sayin.

kh_hk8• 4 years ago

It's a simple example, maybe too simple to get the point. What I want is to conditionally set things on a yaml file. If you have ever used a templating language, you know this problem

    services:
        redis:
            ...
    
        {{ if some_condition }}
            volumes:
                bla bla
        {{ end if }}
    
        {{ if some_other_condition }}
        another_service:
            ...
        {{ end if }}
    
    volumes:
        {{ for volume in volumes }}
            - ...
        {{ end for }}

biggerChris2• 4 years ago

Docker is cool and all.

Kids these days don’t know anything about Tmux and port 433. Exposing localhost to the internet and getting your internet monitored by your ISP and getting letters in the mail; for hosting movies and Storage services.

Lol.

aidos• 4 years ago

What’s port 433? You mean 443?

dijksterhuis7• 4 years ago

433 for those who didn't know (like me)

https://en.wikipedia.org/wiki/Network_News_Transfer_Protocol

The Network News Transfer Protocol (NNTP) is an application protocol used for transporting Usenet news articles (netnews) between news servers, and for reading/posting articles by the end user client applications.
Well-known TCP port 433 (NNSP) may be used when doing a bulk transfer of articles from one server to another.

biggerChris2• 4 years ago

Yes. I meant port 443.

siftrics• 4 years ago

If it's going over port 443, how would your ISP know you're hosting movies?

The way ISPs catch you is by seeing that you're connecting to IPs that are known to host content illegally. There's now way for your ISP to know what's going _out_ of your port 443 without breaking TLS.

jjice• 4 years ago

The post said 433, not 443. I read it as the HTTPS port as well at first.

qbasic_forever2• 4 years ago

Your ISP doesn't care about what you're connecting to and accessing, at least not until they get a court order demanding that info.

They care a LOT about what you're serving to other people from their network though as there might be some legal issues directed back at them.

pkrumins4• 4 years ago

Love your comment! Come work for me!

sgt• 4 years ago

Love your compliment and belief in other people. Come work for me.

koprulusector2• 4 years ago

What does tmux have to do with anything?

Macha• 4 years ago

A common deployment strategy for small scale services, especially around the 2005-2015 era, was to just copy files to your VPS, ssh in and start it up with nohup or in a screen/tmux session so it stayed running when you closed your ssh session. screen/tmux were especially helpful for stuff like irssi or minecraft servers which had a server side console you might want to return to.

This was a step up for novice sysadmins/developers from "FTP to the webroot" without having to learn real deployment tools, how to write init scripts, etc. Of course it wasn't good enough for anything business critical, but I'm sure it happened there too.

Also, if you then wanted to access that service from your college network, who may have gone out of their way to block gaming, a common workaround was to host it on port 443. By 2010-2015, colleges were using DPI sophisticated enough to see that "that's not http traffic" on port 80, but tended to treat 443 as "that's encrypted, must be a black box". That's how I ran mine in ~2013.

edfletcher_t1373• 4 years ago

Back in the day we used screen because tmux didn't exist yet.

sureglymop• 4 years ago

Am kid, do know

CincinnatiMan• 4 years ago

I recently learned that moving the compose file around and running commands on it with it in different directories puts things into a bad state. This is because it apparently tags images and containers with context on where the compose file was run from. However, you can negate this by always passing in a custom project name with -p.

dugmartin• 4 years ago

I think it is just the parent directory name plus the service name (at least that is what I've seen at work where we have many docker-compose files with common "app"/"db" service names).

switchbak2• 4 years ago

There's something about that decision that violates my mental model. Feels similar to a referential transparency issue: the state of the system should be the same regardless of the name of the directory something lives in.

turtlebits• 4 years ago

While that may be true, you should never destroy things that are out of your scope.

Gordonjcp3• 4 years ago

You can specify a project name, but giving things rigidly-defined names breaks the whole idea of it.

Why would you want two different instances of the same project to have the same name?

pbalau2• 4 years ago

Which is true if you pass -p. The current directory is merely the default.

lazide4• 4 years ago

Docker has tons of smells/irritations like this. Overall, it is worth it, but damn is it weird sometimes.

remram3• 4 years ago

It would probably break any bind-mounted volume too, like

    volumes:
      - ./data:/use/lib/data

I don't know why anyone would expect this to work.

kossae• 4 years ago

To mitigate this, I usually always require a .env file for docker-compose containing directories with at minimum a `COMPOSE_PROJECT_NAME` envvar to prevent these naming issues. As long as the env file travels with the directory there are no issues.

zmmmmm• 4 years ago

That's a feature not a bug and it's one of the things I like :-)

It means that if I run multiple instances of a compose setup, they are completely isolated from each other because they live in separate namespaces. It's a very handy feature (for example, if I'm working on 3 branches of the code, just check them out in 3 different directories etc).

Of course it has the consequence that if you move it around it's going to break things, but that's actually the idea. They should break! Because to compose they are meant to be segregated, and if they interacted it would actually be a bug.

bdcravens2• 4 years ago

Surprised it didn't mention profiles, which let you launch only part of your "stack" (think background processing in addition to web app, etc)

https://docs.docker.com/compose/profiles/

Prior to that feature, I had a hand-rolled approach using bash scripts and using overrides. Worked well enough, but felt like a dirty dependency.

Cyph0n• 4 years ago

Wow, this is a pretty cool feature. Looks similar to Ansible tags.

xwowsersx• 4 years ago

Wow, did not know about this. Thanks for the tip.

_joel• 4 years ago

Neat, this would have saved some headaches in the past (about 2018). Was it a feature then? If so I wasted some time reinventing the wheel, it seems!

pbalau2• 4 years ago

It's quite new, I'm still waiting for some free time to ruin it.

bdcravens2• 4 years ago

It was added in early 2021 (1.28)

Ajedi322• 4 years ago

One thing I really like about profiles is that they get activated implicitly if you reference one of their services when running docker compose. So you can define development, debugging, or maintenance tools in your compose file and just run them with `docker compose run some-random-tool` without having to worry about them interfering with the standard `docker compose up` workflow.

aqibgatoo• 4 years ago

Wohoo! Pretty neat, i have been making multiple docker compose files.

lucasyvas• 4 years ago

I just wasted so much duplication effort in a new monorepo by not knowing about this feature - know what my next commit will be I guess!

glintik3• 4 years ago

Wondered that author didn’t touch such important thing as backups of running containers.

lazyant3• 4 years ago

what's the use case for backing up running containers? the code in the docker image is immutable (or should be), config goes into .env file and permanent data should go to a mounted volume in the host or an external database, so what do you want to backup?

glintik3• 4 years ago

ok, how do you backup your database volumes? when db is running.

gottie• 4 years ago

I find docker compose override files whilst powerful ultimately much more confusing and error prone to work with. Instead I would recommend just duplicating your compose files and changing each separately, perhaps using a templating engine. It is often much easier to work with a single compose file containing all the config in one easy to read block. This compared to having to do a yaml merge of all the various blocks in your head whilst reasoning about override files.

Similarly yaml anchors are cool and can easily reduce duplication in your files. However I've found the majority of Devs and users find them confusing and unexpected in a compose file. As a result I would recommend not using them and just sticking with plain yaml for simplicity's sake and not being afraid of duplication.

One other alternative to yaml anchors is instead using the env_file: property on your services (https://docs.docker.com/compose/environment-variables/#the-e...) . This way you can place all of your env variables in a single file and share this same of env variables between many services without having to have large duplicate env blocks in your compose file.

dijksterhuis7• 4 years ago

I usually suggest passing `--env-file` to the actual compose binary. Gives you the "multiple deployment versions" without faffing about with external templating tools.

    docker compose --env-file ${INSTANCE}.env -f docker-compose.yml up

In the `*.env` file:

    CONF_A_VALUE="some_value_a"
    CONF_B_VALUE="some_value_b"
    CONF_C_VALUE="some_value_c"
    ...

In the compose file:

    services:

      service-1:

        # you need to be more explicit here
        environment:
          SOME_NAME_FOR_A: "${CONF_A_VALUE:?err}"
          SOME_NAME_FOR_B: "${CONF_B_VALUE:?err}"

        # but it gives you simple templating in other places
        labels:
          com.company.name.label.a: ${CONF_D_VALUE:?err}
          com.company.name.label.b: ${CONF_E_VALUE:?err}
          com.company.name.label.c: ${CONF_F_VALUE:?err}
        ports:
          - "${CONF_G_VALUE:?err}:8020"

      service-2:

        # also gives services more freedom to diverge from one another (if required)
        environment:
          MAYBE_ANOTHER_NAME_FOR_A_BECAUSE_COWBOY_TEAM_REASONS: "${CONF_A_VALUE:?err}"
          SOME_NAME_FOR_C: "${CONF_C_VALUE:?err}"

Where `?err` forces the variable to be set and non-empty: https://docs.docker.com/compose/environment-variables/#subst...

It's just a shame that `docker stack` doesn't have an `--env-file` argument yet for swarm deploys.

dijksterhuis7• 4 years ago

I think the YAML anchors section would lead to an unparseable compose file?

No `services` block above the `api` and `web` services, on the same level as `x-app:`.

So the `<<: *default-app` anchor won't be recognised as the block that defines it is never terminated prior to use?

(I've not tested as currently on the windows gaming box. this is purely from reading it, so happy to be corrected).

duskwuff• 4 years ago

Yeah, the example in this blog post is nonsense. The services shouldn't be defined inside the same block as the defaults.

postalrat• 4 years ago

"August 18, 2022"

qbasic_forever2• 4 years ago

One of the coolest lesser known features of Docker is time travel.

newman314• 4 years ago

Is there a "proper" way to have per host configs? I have a setup where I need to set "hostname: <HOSTNAME>" depending on the system I'm on and right now I'm having to resort to using "docker compose -f compose.<hostname>.yml up -d" where compose.<hostname>.yml extends compose.yml

azeirah• 4 years ago

Is there a reason why you wouldn't be able to use environment variables for this? Or am I misunderstanding your issue?

koprulusector2• 4 years ago

docker compose automatically looks for .env file with the format key=value and will auto populate variables in the docker-compose.yml with matches.

Ex:

docker-compose.yml:

hostname: ${HOSTNAME}

.env:

HOSTNAME=foo.bar.com

fortyseven• 4 years ago

Maybe some "mobile responsiveness" best practices should be looked into next.

0xbadcafebee4• 4 years ago

If you don't use the same Docker Compose file for Production, don't use it for Development. Your two different systems will diverge in behavior, leading you to troubleshoot two separate sets of problems, and testing being unreliable, defeating the whole "it just runs everywhere" premise.

Docker Compose is fine for running smoke tests/unit tests, but if Dev and Prod run differently, there's really no point to using Docker Compose other than developing without internet access.

(It goes without saying that Docker Compose only works on a single host, so that won't work for Prod if you need more than one host, but single-host-everything seems to be HN's current fetish)

tjoff2• 4 years ago

That is an absurdly black-and-white view.

dijksterhuis7• 4 years ago

I have lost count of the times I've heard "but it works fine on MY machine, it must be the CI/CD that's broken"

I then find out that no-one bothered to run the compose file that matches what gets put into prod on their local machines.

My point is: your view might become more black and white on this matter after the N-th "but it works fine on MY machine" comment.

EDIT: Where N = your personal tolerance level of bullshit.

bananaoomarang• 4 years ago

Compose also works well with Swarm in my experience.

nicoburns• 4 years ago

Meanwhile here I am developing on macOS (no Docker or VMs involved) and deploying to linux. We do run tests on a linux CI runner, and we have a staging environment that's identical to prod. But really cross-platform support tends to be really good these days. I see no need for identical environments for development.

dijksterhuis7• 4 years ago

defeating the whole "it just runs everywhere" premise.

Straight up 100% hard agree.

It goes without saying that Docker Compose only works on a single host, so that won't work for Prod if you need more than one host, but single-host-everything seems to be HN's current fetish)

technically this is only correct for compose file versions up to and including 3.8.

Part of the push with the new mainline compose V2 plugin has included the updated compose specification [0] which seems to have unify quite a lot of differences between the "compose" file for compose deployments and the "stack" file for swarm deployments.

FYI You can switch over to the new version with the compose V2 plugin by remvoing any `version` key from your compose file.

Although the `deploy: restart_policy:` values are currently not unified, much to my sadness.

[0] https://docs.docker.com/compose/compose-file/

twblalock2• 4 years ago

Also, don't use Docker Compose in dev if you use Kubernetes in prod. Just use the same thing everywhere. Otherwise the fact that something works in dev tells you nothing about whether it would work in prod.

0xbadcafebee4• 4 years ago

^ 1000%

frenchman992• 4 years ago

Running a local k8s cluster doesn't guarantee that a remote cluster on some cloud provider will work the same either.

I've heard of and worked with multiple dev teams that develop with compose and deploy a totally different way (k8s, AWS auto scaling groups, heroku, etc), to great success.

Devs know about compose and understand it. As long as the containers behave as planned, the DevOps people can figure out how to deploy them on whatever production environment.

shroompasta• 4 years ago

This is a questionable comment

If you don't use the same Docker Compose file for Production, don't use it for Development. Your two different systems will diverge in behavior, leading you to troubleshoot two separate sets of problems, and testing being unreliable, defeating the whole "it just runs everywhere" premise.

Anyone ever developing a react project on docker compose would never run a build step for development.

In fact for most setups, I probably wouldn't even run react on docker-compose for development, and just use the dev server straight up on my machine.

Furthermore, working with any python http servers is substantially easier when working with a dev server in development, rather than running your server through gunicorn or other wsgi servers - not to mention the ability to hot reload for development.

I'm sure that there are many other cases where it's necessary and convenient to separate production and development docker compose files.

I don't think it's fair to say that there is no point in having separate docker-compose files as the lion's share of dependencies that need to be consistent is inside each container, not on the docker-compose configuration level.

rdsubhas2• 4 years ago

... if Dev and Prod run differently, there's really no point to ...

"If dev and prod run differently" — Why is there an "if" here at all? Isn't it obvious to verify with two eyes in the real world? Is there any company in the world that runs production from somebody's laptop at home? Or distributes server racks for people to take home? It looks like an extremely disillusioned question out of touch with reality.

1. For development, engineers want to run a local throwaway mysql/redis. In production, engineer want to use a proper managed mysql/redis. The compose file, and envvars/links will be different. This is extremely normal, it makes sense.

2. Development laptops will always be underpowered than servers. The cpu/memory requirements will be different. Again makes sense.

3. For development, engineers will build and run the docker image locally. In production, these are separate, you would build and publish the image separately in CI, and run only published images in prod. Again, makes sense.

These are all perfectly logical things that make practical sense.

Trying to share/ reuse/ resemble is fine. But saying "dev===prod" is just a disillusion out of touch with reality.

Edit: For context, yes I do docker for a living for last 10 years, since v0.4 (lxc) days. I've been to and presented in meetups. If someone says they're trying to get dev as close functionally to prod, by doing X, Y, Z and sharing practical tips — you know you can listen to them. If someone says "dev===prod" and when you ask how they talk abstract principles, you quietly get out of there.

0xbadcafebee4• 4 years ago

For development, engineers want to run a local throwaway mysql/redis. In production, engineer want to use a proper managed mysql/redis.

Bad idea. You develop against a completely different local system, then you push to prod, and "oh no it doesn't work". Yeah - because you're developing against a completely different system! It behaves differently. It causes different bugs. You take a ton of time setting up both different systems differently, duplicating your effort, duplicating your bugs, having to fix things twice for no reason, and causing production issues when expectations from local dev don't match prod reality.

Every experienced systems engineer has been repeating this for decades. Listen to them. They all say exactly. the. same. thing. "Make all your systems as identical as possible." That doesn't mean make all your systems different whenever it is more convenient for you. It means that if it's possible for you to be developing using the same tech you use in prod, you should be doing that, unless it is impossible. If you think it's not possible or practical, you simply haven't thought hard enough about it.

The whole selling point of the cloud is that you can spin up environments and spin them down in minutes and pay pennies for it. Not a shared environment where everyone's work is constantly conflicting, but dedicated, ephemeral environments that actually match production. Literally the only example I know of where this doesn't work is with physical devices where you have a lab of 10 experimental pieces of test gear and you have to reserve them to test code on them. That can make remote development difficult. Still not impossible though.

The cpu/memory requirements will be different.

Unless you're developing using the same servers as prod. There is no reason that you have to use your laptop. If you think "it's more convenient!", that's only because you have done zero effort to make Prod-like ephemeral environments more convenient.

For development, engineers will build and run the docker image locally. In production, these are separate, you would build and publish the image separately in CI, and run only published images in prod. Again, makes sense.

That is the antithesis of what Docker was created for. The point of containers was to say "it works on my machine", and use that exact same image in production. Not a different image in CI that nobody developed their code against. It's just going to result in bugs that the developer didn't see locally, so they will need to spend more time to get it working. Your laptop can do the same build CI can on the same branch and push the same image that CI can.

What I'm saying is nothing new. Look into cloud-based development environments and the dozen different solutions made solely for rendering your local code in a remote K8s pod as soon as you write to a local file. Imagine a world where you don't waste your time testing something twice and fixing bugs twice.

satyrnein• 4 years ago

If the dev environments are on AWS with real S3 buckets (etc), and everything runs as an auto scaling group, is it not possible to use the same exact docker images (and terraform) in dev and prod?

We haven't achieved it (yet?) but that's the direction I was aspiring to!

zoomzoom• 4 years ago

Totally agree that in practice you often need several configurations for the same systems across different environment types. For example, you probably also need a different docker-compose for running in CI when compared to dev or a deployed environment. The same applies for k8s and terraform, and many others.

Managing all these different configurations and the matrix of interactions between them is why we're trying to integrate the SDLC into one configuration at Coherence (withcoherence.com). [disclosure... I'm a cofounder]

strzibny• 4 years ago

It is. Convenience. I mean you replaced perhaps running staff locally that also didn't match production in the first place...

I am now using this setup[0] for development, but it doesn't really match production 1:1.

[0] https://nts.strzibny.name/hybrid-docker-compose-rails/

edfletcher_t1373• 4 years ago

The article is titled "for dev and prod" (emphasis on the latter), yet the very first suggestion is to bind-mount your source outside the container so you can mess with it at any time? How is this good advice for production?! (Update: this is indeed great advice for development, and it's actually in a sub-section dedicated to that! My bad.)

Update: reading comprehension, it's a thing! :) Thanks chrsig for pointing out that this is under an appropriate sub-heading. D'oh!

chrsig• 4 years ago

it's under a subheading of

Docker Compose Best Practices for Development

pcorsaro• 4 years ago

Just curious, why would a bind-mount be bad in production? I understand if you're running multiple app servers and need the code to be deployed in a lot of places that they'd be bad, but if you're just using a single server, what's the downside of using bind-mounts? You could use git to push code changes up to your server, and they'd be picked up immediately without having to rebuild and push an image.

adamckay• 4 years ago

Usually for production you want to know with 100% certainty what version of the software is running. Whilst you can do that with Git commit hashes you can't be sure no one has performed a hotfix and modified code on the production server which is not committed to source control. Then there's also the potential problem of pulling the latest updates from source control but forgetting to restart the container so it's not obvious you're actually running outdated code.

It's also a lot easier if there's a bad deploy to roll back the update by reverting the image tag in the compose file and restarting rather than checking out specific older commits and risk getting into a funky state with detached heads and the like.

solardev2• 4 years ago

If you happen to be unlucky enough to need Docker for LAMP/LEMP stacks, Lando is a neat open-source util one level of abstraction above Docker Compose: https://lando.dev/

Like with `lando init` and then `lando start`, it'll fetch a LEMP "recipe" (say, for Wordpress or Drupal), configure all the Docker containers for you, set up all the networking between them, get PHP xdebug and such working automatically, and leave you with an IP and port to have your app ready at. It supports a bunch of other common services too, like redis, memcached, varnish, postgres, mongo, tomcat, elasticsearch, solr, and only very rarely do you need to drop down to editing raw docker compose or config files.

Having had to use the Docker ecosystem for a few years, Lando made my life way easier. Although these days I just tend to avoid Docker altogether whenever possible (opting for simpler/more abstracted stacks, often maintained by a vendor like Vercel, Netlify, or Gatsby).

user3939382• 4 years ago

I was kind of puzzled by your preamble:

If you happen to be unlucky enough to need Docker for LAMP/LEMP stacks

We run Docker LAMP and have a new version underway that's LEMP, haven't run into any problems. xdebug works fine, I even got it working recently in ECS. I'm not sure what's different about LAMP/LEMP in Docker vs any other stack.

solardev2• 4 years ago

Sorry for being unclear. It's not that Docker doesn't work well with LEMP, it's that Lando in particular was designed around older stacks (LEMP/LAMP and caches) and wouldn't be appropriate if you were trying to spin up, say, Next.js or Deno or Cloudflare Workers or React Native stacks or whatever.

The "unlucky" part is just having to work with a LEMP stack at all, and that's just my personal bias leaking through my post, sorry. Having grown up with that stuff and used it until just last year, I am so so grateful that I was able to finally move into a frontend job where I don't have to manage the stack anymore. A new generation of abstracted backend vendors (headless CMSes coupled with Jamstack hosts) makes it so that there is a sub-industry of web devs who never have to touch VMs directly anymore. I, for one, couldn't be happier about that.

(But of course there will always be other use cases that require a fuller/closer to the metal stack, and also backend and ops people who love that work. I don't fault them in the least, I greatly respect them, I'm just glad I don't have to do that.)

jaimehrubiks• 4 years ago

Do people here set CPU limits? (in either docker or kube)

In my company we only set CPU requests + Memory req+limits. I read somewhere that CPU limits are kinda broken and are not reliable. In some experiments CPU limits tend to cause unnecessary throttling, which is why some people don't recommend them.

However, I've also found some cases where I can't make the app to use only some of the CPUs, some frameworks tend to read the number of total available CPUs on the node and try to use them.

hu3• 4 years ago

In Windows dev machines, under wsl2, I always set CPU and RAM limits by creating a file in %USERPROFILE%/.wslconfig with something like:

  [wsl2]
  memory=10GB
  processors=4
  swap=2GB

By limiting wsl2 I also limit the docker containers that run inside wsl2.

Mavvie• 4 years ago

In our k8s clusters we generally set both requests and limits on CPU and memory. We haven't noticed many problems, except for when some apps didn't even have CPU requests and dominated all the CPU on a cluster (which didn't even autoscale because the scheduler doesn't know about the problem).

I think CPU limits (and also, choosing to set memory limits equal to the requests or higher) depends on your goal. If you want to get the best use out of your hardware, set low requests and high limits and you'll be able to minimize "wasted" scaling. But if your goal is consistent performance and availability, setting high requests and high limits (usually equal to each other) save you from a whole host of problems.

Just my take. I have a fair bit of experience but am far from an expert on the topic. And my experience is only on the scale of dozens of applications/dozens of nodes...people with higher scale experience may feel differently.

JadoJodo• 4 years ago

One of the things we've really struggled with is sharing dev compose files between Windows and macOS. We were unable to make bind mounts work correctly with Windows, due to performance issues (this used to be documented in the Docker docs, but I can't seem to locate it now). This resulted in us falling back to VSCode's remote container tooling.

Has anyone else had this issue with Windows? The last time I looked into the issue was about 8 months ago, so it's possible the issues have been addressed since then.

tmpz222• 4 years ago

You're trying to create a uniform developer experience across two platforms that are wildly different.

If you find yourself spending a ton of time on a one-size-fits-all-disappoints-everybody solution, maybe its time to build two effective solutions instead.

bluedino• 4 years ago

That's kind of the point if containers, isn't it?

nickjj• 4 years ago

We were unable to make bind mounts work correctly with Windows, due to performance issues (this used to be documented in the Docker docs, but I can't seem to locate it now)

Docker Desktop volume performance on Windows is great if you're using WSL 2 as long as your source code is in the WSL 2 file system. It also works great with WSL 1. I've been using it for a long time for full time dev.

In fact, volume performance on Windows tends to be near native Linux speeds. It's macOS where volume speeds are really slow (even on a new M1 using Virtiofs). Some of our test suites run in 30 seconds on Windows but 3 minutes on macOS.

But the Docker Compose config is the same in both. If you want, I currently maintain example apps for Flask, Rails, Django, Node and Phoenix at https://github.com/nickjj?tab=repositories&q=docker-*-exampl..., I use some of these exact example apps at work and for contract work. The same files are used on Windows, macOS and Linux.

The only time issues arise is when developers on macOS forget that macOS' file system is case insensitive where as Linux is not. Docker volumes take on properties of their host so this sometimes ends up being a "but it works on my machine!" issue specific to macOS. CI running in Linux always catches this tho.

KronisLV• 4 years ago

Docker Desktop volume performance on Windows is great if you're using WSL 2 as long as your source code is in the WSL 2 file system. It also works great with WSL 1. I've been using it for a long time for full time dev.

My experience actually was the exact opposite. This one project had horrible IO performance on Windows, when running a bunch of PHP containers, with the WSL2 integration in Docker Desktop set to enabled.

It was bad to the point of the app taking half a minute to just load a CRUD page with some tables, whereas after switching to Hyper-V back end for running Docker, things sped up to where the page load was around 3 seconds.

I have no idea why that was, but after switching to something other than WSL2, things did indeed improve by an order of magnitude.

freenatec• 4 years ago

If you’re using WSL 2, but your project files are in windows-land and vscode is reading them via the mnt path (like /mnt/c/code/myproject), then you’re going to have show-stopping performance problems.

bacchusjackson• 4 years ago

Maybe I'm bias but I really don't like the recommendation to use anchors. It makes the compose file look super cryptic but that's just my opinion.

Ajedi322• 4 years ago

I like to divide up my compose files into three:

1. A `docker-compose.yaml` for basic configuration common among all environments

2. A `docker-compose.dev.yaml` for configuration overrides specific to development

3. A `docker-compose.prod.yaml` for configuration overrides specific to production

Then I have `COMPOSE_FILE=docker-compose.yaml:docker-compose.dev.yaml` in my `.env` file in the root of the project so I can just run `docker compose up` in dev, and `COMPOSE_FILE=docker-compose.yaml:docker-compose.prod.yaml` in a `prod.env` file so I can run `docker compose --env-file=prod.env up` in production. Sometimes also a `staging.env` file which is identical to `prod.env` but with a few different environment variables.

It's been working pretty well for me so far.

xorcist2• 4 years ago

In my experience, the "base" configuration should always be in use somewhere. Otherwise you inevitably end up with cruft, when default values turns out to have no practical value.

In your example, "dev" should probably be the default configuration, which "prod" overrides. It's a tiny detail but I think it pays over time.

A similar situation is when you have lots of almost-identical settings between many (micro)services.

lazide4• 4 years ago

100% agree - I find staging useful here. It’s the default ‘non-production’, and has realistic data shared in a way that anyone in dev can access it. Think ‘sacrificial prod with no actual real customers hitting it right now’. Depending on data sensitivity requirements, it might have a subset of real data, or no real data.

Having prod be the default is a problem, as it’s inevitable someone will accidentally break production/write junk to it when they didn’t realize they were talking to prod.

Having dev in it inevitably results in broken prod pushes because no one has ever tried it outside of a synthetic dev environment.

heliodor3• 4 years ago

Sometimes dev and prod go in different directions so it makes more sense to inherit a common file. For example, dev might have a whole bunch of settings for some debug tool whereas prod might have a whole bunch of settings for the email server config.

sandGorgon5• 4 years ago

docker compose was one of the most spectacular and developer friendly deployment tool i have used.

unfortunately the ecosystem is on kubernetes. you want to integrate with spot instance bidding on AWS...u need kubernetes. you want monitoring tools...u need kubernetes, etc

the Compose spec is now open and standardised - https://www.compose-spec.io/

a kubernetes distro that can be managed entirely using compose files will be a massively impactful project. Not like Kompose which converts Compose files to k8s yml files....but entirely on Compose files.

I really hope someone does a startup here.

frenchman992• 4 years ago

What do you mean with monitoring tools? You could deploy Prometheus/Grafana or an ELK stack on Compose so I'm not sure I understand what you mean

sandGorgon5• 4 years ago

im talking about managed tools... not self hosted ones.

e.g. datadog, etc https://www.datadoghq.com/blog/monitoring-kubernetes-with-da...

that said, Grafana's own hosted solution "Grafana Cloud" gives Kubernetes operators...but not for anything else. https://grafana.com/docs/grafana-cloud/kubernetes-monitoring...

the entire ecosystem is supporting k8s. there's not much choice here.

jeremy_k2• 4 years ago

When you say 'a kubernetes distro that can be managed entirely using compose files will be a massively impactful project. Not like Kompose which converts Compose files to k8s yml files....'

Do you mean from the your perspective, you set up your docker-compose and this startup ends up deploying pods into Kubernetes based on your compose file? Basically cutting out the middleman of you having to deal with Kubernetes yaml files?

sandGorgon5• 4 years ago

yes. basically. and this is a path that multiple people are trying to solve. e.g. AWS CDK8s, https://tanka.dev/, etc

Compose would be awesome.

khuey• 4 years ago

Perhaps I'm missing something but you can throw containers at Fargate Spot on AWS and it works just fine.

sandGorgon5• 4 years ago

sure. and it works great.

Docker Compose is also unnecessary for that. I'm talking to the demographic of users who want to run Compose/K8s/Rancher by themselves. For most people, i dont recommend it - you should use Fargate.

kalev• 4 years ago

This is something i’ve been thinking about for a long time and your comment made me think about it again. I’ve seen many companies developing using docker-compose but deploying their software in a different way (on bare metal, kubernetes or nomad). Why hasn’t anyone build this yet? I’m afraid I’m missing something so starting on this my own is risky.

RandomBK• 4 years ago

One topic I'd like to see more discussion around is how best to set up staging environments for Docker Compose projects.

Ever since the V2 migration, I've been banging my head against multiple bugs in Docker Compose itself, which has severely damaged my trust in it as a production-ready utility. I no longer feel like I can safely update Compose without running extensive testing on each release.

For example, my journey since migrating to V2 has been:

* Originally on 1.29.2

  - Stable and I did not encounter any issues.

* Swapped to 2.3.3.

  - As part of the migration from V1, docker-compose lost track of several networks and started labeling them as external. Not a huge deal.

  - docker-compose stopped updating containers when the underlying image or configuration changed (#9291). This broke prod for a while until I noticed nothing was getting updated.

* Updated to 2.7.0, which was the latest release at the time.

  - This introduced a subtle bug with env_file parsing (#9636), which again broke prod in subtle ways that took a while to figure out.

* Updated to 2.8.0, which was the latest release at the time and before the warning was put up.

  * This broke a lot of things as it renamed containers and generated duplicate services, etc.

* Finally swapped to 2.9.0. No issues yet, but I'm keeping an eye out.

imwillofficial• 4 years ago

This was an awesome article.

It lays out every single hard lesson I discovered when working with docker compose.

Great minds think alike! (Or I just got lucky)

rubyist5eva• 4 years ago

docker compose is old hat, using podman to generate k8s Kubefiles for pods is where it's at

layoric• 4 years ago

I tend to have a compose file that is designed to be used with `envsubst` for deployment via the CI, so a CI specific template. This way the CI can create the final file and `scp` it over to the host before calling `docker compose up`. The file always ends up with the same name and same location on the host so `docker compose` can detect any changes. So far haven't had any issues, but definitely not as clear as having `compose.dev.yml`/`compose.prod.yml` in source control, though very flexible.

Made a video of using this pattern with Blazor WASM, SQLite and Litestream here [1].

[1] https://youtu.be/fY50dWszpw4

robertwt7• 4 years ago

huh weird, I still remember back then reading that docker-compose was not supposed to be used in production somewhere? However upon reading docker-compose official docs now they actually mention how to use in production.. of course because of that I had to learn kubernetes for months (no regret), however if docker-compose is suggested for production back then, I would have used it just for the speed of deployment

junker101• 4 years ago

We've found https://tilt.dev/ to actually be a great alternative to docker-compose that also keeps the local setup and knowledge more inline with production (which is Kubernetes hosted)

eric_khun• 4 years ago

Running Kubernetes in production, we've also decided to adopt tilt and get rid of Docker compose. Our own tooling with docker compose has been to painful to maintain, and there was too many workaround to try to replicate what we have in production.

secondcoming• 4 years ago

Reading the comments here is giving me a headache. Will nobody admit that this is all a nightmare?

mirekrusin2• 4 years ago

If this gives you headaches, don't read anything k8s - you may end up changing profession.

preseinger• 4 years ago

docker compose in prod?? what???

who would do this??

wejick• 4 years ago

With the rising popularity of podman and its native support (and preference) of kube pod, it will be interesting to see whether people sticking to compose or moving. I personally use podman-compose and still not convinced to convert the compose file to pod, which podman can facilitate. However with podman-compose I found host network mode is not working for me.

8n4vidtmkvmk• 4 years ago

kubernetes, kustomize + devspace. same tools for dev and prod. works great

wseqyrku• 4 years ago

I wonder if this works with overrides:

docker-compose.yml (build current dockerfile) docker-compose.image.yml (fetch stable image)

To avoid duplication, the second file only specifies the image.

Docker Compose best practices for dev and prod

Related Stories