Docker-compose.yml as a universal infrastructure interface

https://ergomake.dev/blog/docker-compose-as-a-universal-interface/

by vieiralucas • 3 years ago

128 100 3 years ago

ergomake.dev

klysm• 3 years ago

I think the direction of podman running k8s yaml directly makes more sense here.

mardifoufs• 3 years ago

Why? What's better with podman in this context? Genuinely curious since I can't think of any real advantage.

moondev• 3 years ago

The only advantage I see is it dosen't require a kube-apiserver. Other than that it feels like just running a quick throwaway cluster with kind and writing actual manifests is more accurate and portable.

Takennickname2• 3 years ago

Podman doesn't support Docker compose. Instead, it uses the same YAML files that are used by Kubernetes.

dunk010• 3 years ago

Makes a great deal of sense. It was basically the original idea (when compose was fig). Oftentimes though, people cling desperately and aggressively to existing complex systems...

mikepurvis• 3 years ago

Having a complex system under the hood is not inherently bad, so long as you have non-leaky abstractions that allow that complexity to be reasoned about (pods, services), sane guardrails that let you know when you're throwing away key guarantees (privileged containers), and reasonable defaults for portions of the system that you don't yet need or care to engage with (resource limits, among others).

Kubernetes may be a system that invites overcomplication, but it in no way requires it— in a world where anyone can run k3s or microk8s on their own laptop and a Kubernetes hello-world is like 10 lines of yaml, I don't think it's a stretch to posit that that's the interface that makes sense to use up and down the stack.

brodouevencode5• 3 years ago

If this article is any indication of the current state of the industry then the devops movement has busted. We're back to dev and ops.

TechBro86155• 3 years ago

The problem is YAML, but it's too late to stop that momentum now.

Is anyone using Cue or dhall in a real production scenario? I'd be curious to hear reviews of those, as they look promising.

tristan9574• 3 years ago

I would be more interested in a stripped down JavaScript runtime for configuration. Some tooling already supports this paradigm. Embed something like QuickJS or a Lua interpreter and have the script export a config object. It feels too much like we want these plaintext config files to turn into fully-fledged scripts.

zamalek2• 3 years ago

I share your, and all Norwegians', distaste for YAML. One saving grace I have found is that YAML is a superset of JSON. JSON isn't great, but it's certainly better than YAML.

I toyed around with dhall to generate k8s deployments. It was fantastic, but then I realized there was no chance that others would be willing to learn FP. Cue is pretty great, but it still has many sharp edges in the tooling.

Ultimately, though, all configuration files tend toward turing completeness. Just use an actual programming language. If you need to dynamically parameterize it then you can import a JSON file or whatever.

edganiukov• 3 years ago

We do - https://foundation.namespace.so/docs. Some feature of "Namespace" are kind of similar to Ergomake, but we use own format with CUE instead of docker-compose. Here is comparison to docker-compose - https://namespace.so/blog/from-docker-compose-to-namespace.

yourapostasy• 3 years ago

CUE [1] is kinda-sorta "transpiled" into by Pulumi's language support facility [2]. In general the greater the domain space being expressed the more I'm leery of this kind of approach; leaky abstractions creep in faster the larger the volume of concepts embraced. Infrastructure is a pretty big concepts volume. Whatever the source system (Pulumi in this case) does not provide a hook for that the target system (CUE in this case) supports for example, hacks tend to start to emerge. If there was both this layer of abstraction and a "break glass" layer available down to something analogous to an Abstract Syntax Tree level of primitives, then the target systems at least have hooks to compose new abstractions via some kind of DSL into the source system perhaps. There be dragons there too, though; no panaceas, but at least it is better than all sorts of hacks.

I want to find out with CUE's BCL heritage to fix what they got wrong with BCL at Google being used by Borg, what the CUE team is doing to try to avoid the Second System Effect biting them as they build CUE.

Both CUE and Dhall [3] seem promising when first deployed, but some field experience is reporting challenges I typically associate with immature tooling ecosystems surrounding them [4] that come up short when trying to reason beyond what the out of the box experience can quickly support when working at scale.

One part I haven't quite figured out with these configuration expressions is how to seamlessly flow their models between organizational data models (to answer who owns what type questions, example) and operational data models (to answer who is on call type questions, for example), and gracefully manage the mistakes inevitably made in those models, without mountains of manual toil and bespoke in-house glue code.

[1] https://cuelang.org/docs/about/

[2] https://www.pulumi.com/blog/extending-pulumi-languages-with-...

[3] https://dhall-lang.org/

[4] https://news.ycombinator.com/item?id=32102203

poolopolopolo• 3 years ago

If anything, situation is much worse than when we used to have "System/Network administrators", the cloud was a lie.

bradwood2• 3 years ago

Yeah. Everything then was a "layer 9" problem.

brodouevencode5• 3 years ago

Don't know that I'd go so far as to blame the cloud (maybe the marketing materials but not the actual cloud) but I do share the sentiment.

bradwood2• 3 years ago

100%. It's not hard to learn a bit of k8s yaml if you're an appdev FFS.

mieubrisse2• 3 years ago

Little bit of self-promotion here, but seeing this comment is exactly what we felt at Kurtosis: https://docs.kurtosis.com/explanations/what-is-kurtosis

We had DevOps for a while, but now systems have gotten so complex (and our tooling hasn't kept up) that we're back to pre-DevOps days with Dev throwing stuff over the wall to Ops.

morelisp2• 3 years ago

"Progress" is that now also ops gets to throw half-functional operators, underdocumented CRDs, and pluggable CRI/CNI/CSI all not quite configured to work together properly back over the wall to devs who have no hope to debug anything anymore!

dudeinjapan• 3 years ago

This article is not even wrong.

mac-chaffee2• 3 years ago

The Kubernetes API is itself a leaky abstraction over many different concepts. Putting docker-compose over top of that is just an even leakier abstraction.

Even their example of using Ingresses is already leaky because there is no similar concept of Ingresses in docker-compose, so they're hacking it with labels. What if I need to set a higher request timeout for my app? I have to set a docker-compose label that converts into an Ingress annotation (the leaky part of the Ingress abstraction)? And this is somehow less cognitive load?

Like all similar attempts to simplify deploying apps, it only works if all your apps fit in a nice box. Anything slightly non-standard (like needing different classes of storage volumes) will expose the leakiness. Even Kubernetes kinda still expects all your apps to be stateless web services and will fight you otherwise.

AcerbicZero• 3 years ago

I thought maybe I was just getting old and crusty, but this really does capture my primary complaint with K8s - everything that isn't standard, is so special and unrelated to the underlying technology that you need to relearn how K8's interacts with every layer of abstraction just to have half a chance of it working.

MuffinFlavored3• 3 years ago

everything that isn't standard

Could you give an example? I thought k8s and its YAML/practices was the standard?

freedomben2• 3 years ago

Yep. And the end result is a hodge podge of labels that make up an API that is much worse than the original interface, and requires significant amount of bounds/error checking in order to provide useful error messages until the codebase for it is no longer simple.

techn00• 3 years ago

How would you make it less leaky then?

rlyshw• 3 years ago

Helm charts are pretty standard, git/kube-native, and lead much more nicely into the more sophisticated operator/controller model.

One part I haven't quite put a handle on is where/when to use CRDs in either helm or OLM. The community still seems split on how to use CRDs.

mac-chaffee2• 3 years ago

I just give developers access to k8s and teach them about it. That's really just my admission that the Kubernetes API isn't perfect, but it's better than many alternatives. If I were to build something to hide k8s from developers, eventually I'd have to add so many extra options that I'd probably end up re-inventing the Kubernetes API anyway.

That may just be unique to my job. Maybe another company could force all apps into nice boxes that prevents their bespoke platform API from leaking.

indentit• 3 years ago

Interesting idea. I wonder how some examples would look for teams which have a docker compose file which don't include some services - because they are mocked for tests or expected to be run separately etc. Would it still be possible to write a k8s operator to transform the docker compose file perfectly to a k8s manifest or would it be lacking key information?

hosh• 3 years ago

Helm did something like in the earlier versions. Helm 3 got rid of it to the great cheering of many. It didn't ingest Docker Compose files, but it is the same architectural pattern.

https://www.techtarget.com/searchitoperations/news/252474360...

On the other hand, you have something like CloudFoundery on top of K8S:

https://github.com/cloudfoundry/cf-for-k8s

paxys• 3 years ago

Intriguing idea, but I don't see how a docker compose manifest can be translated to "just work" in a kubernetes cluster without making a long list of non-intuitive implementation decisions. And if devs are supposed to now learn all the intricacies of compose when using it in an kuberentes context then it isn't very different from any other custom kubernetes config abstraction. Worse in fact due to all the confusion it will bring with it.

ttfkam2• 3 years ago

Honestly, I love compose files for local development. Launch and init a database, a DB admin tool, middleware, and a UI server in a single shot.

New team member: "How do I get the stack running?"

Existing member: "Clone this repo and run docker-compose up -d"

20 minutes and a cup of coffee later:

New member: "Cool! Ready to start!"

Does it look like our AWS stack? Only superficially. Which can be a good thing. The compose file shows you the broad strokes, not the extreme details. Compose files help with the basic mental model. Want to know more? Read through the CDK stack code.

But using a compose file for a cloud deployment interface? No thank you.

ocimbote• 3 years ago

In theory, it works everywhere. In practice, for any sufficiently large project, teams can't (either in terms in possibility or ability) size down the running services so that the whole stack runs locally.

And then starts the hunt for tradeoffs, with a million pitfalls.

I have seen the pattern fail too often to be only optimistic on the approach. I now tend to favor ephemeral deployments with only one locally running service connected to a bunch of remote services.

ttfkam2• 3 years ago

News flash: 90% solution not suitable for folks with 10% of problem domains. Full story at 11 o'clock.

shortrounddev• 3 years ago

Yeah recently at work I created a compose.yml for our kafka pipeline. We have a collector and an enricher which produce to and consume from a kafka topic, respectively. The enricher then runs some functions on the data it consumes and produces to another topic. These services are all super microservicey, and so testing locally outside of docker was difficult (not to mention the PITA that is setting up a zookeeper broker on Windows)

The compose.yml runs everything on a docker network and uses redpanda for the kafka broker. Iteration became SUPER easy. Previously, we were just writing config files locally and uploading them to S3, then restarting the dev instance of whatever we were working on in k8s to re-ingest. This was, as I'm sure you can tell by now, tedious

aitchnyu• 3 years ago

I gave up on compose since I couldnt figure out VSCode autocomplete for Python and JS, with the library files living within the container. Once I needed to run a command on a docker container (wasn't allowed to commit it) to make it work. To update pip/npm stuff, I do it outside of docker and then update the images.

Has anybody found a better way with just open source tools?

twblalock• 3 years ago

If you're going to run in prod in Kubernetes then you need to live with the idea of applying Kubernetes manifests. So, live with it. Docker Desktop made that easy, and some of the alternatives like Rancher and Podman also make it easy. Plus, how are you going to test your ConfigMaps and Roles and RoleBindings if you aren't actually running in Kubernetes locally? How do you know if a change to your Helm chart is going to work?

If you are not running in prod in Kubernetes, but you still run in prod with Docker, what the heck are you doing? ECS? You still need a control plane to deploy the images and Compose isn't really up to that task for production workloads.

mgaunard3• 3 years ago

Cloud technology is so bad. Just run your programs as normal processes. No bloated images and runtimes, no slow virtual bridging layers, no barriers preventing you to access the hardware or communicating with the user.

The hardware has finite resources, and the best way to share them is to run things natively on the operating system.

It's also all simple and straightforward, everyone understands ssh and simple Linux sysadmin.

WirelessGigabit2• 3 years ago

I disagree. While there are bloated images and runtimes, at least they are separate.

Unfortunately most applications are developed in a way that they believe they are the only one on the system. Ruby gets called, and they expect 2.7.

And there is another one which is using Ruby 3.2. And it's incompatible with 2.7.

Now what? Do you want to spend time

- maintaining 2 versions of ruby on a system

- keeping them separate

- ensuring they don't mix up

Shove them in a container, run them in parallel and all they surface is a port.

I'll take the container.

thriftwy3• 3 years ago

Dockerized delivery starts easy unless you realize you need a new container for every small thing you need to run, and you end up with a pile of containers a short while later.

Docker containers, in my experience, have not figured out auto updates yet. You may be running containers with critical system vulnerabilities and is not bothered to check.

So the basic premise of Dockerization 1.0, "I have exactly one thing which I will put in a single container with all of its dependencies, reproducibly built" is wrong.

mgaunard3• 3 years ago

Well you need to do that anyway if you are indeed building those applications.

bak3y• 3 years ago

Spoken like someone who's never done Linux sysadmin at scale.

mgaunard3• 3 years ago

I've done hundreds of nodes, all managed to a very low level (I've been to the datacenter and optimized placement of servers within the cabinet to enable shorter cables to the patch panel) which is way beyond what most companies need.

In general it's better to have fewer and more powerful and reliable nodes that you own than to have many disposable small ones, as shared instances just don't work well and aren't cost-effective.

Cloud in general just makes everything more expensive while removing control. The costs on AWS if paying for three years upfront is the same as buying the hardware outright, except you have little say in what the hardware is and how the infrastructure is set up.

goodpoint• 3 years ago

If anything it's the opposite.

Takennickname2• 3 years ago

I think what you really want to say is your last sentence:

It's also all simple and straightforward

What you're really trying to say is that cloud technologies are complicated for you.

thriftwy3• 3 years ago

You must be really young if you do not understand that aversion to complexity is likely a single best trait a developer may have.

freedomben2• 3 years ago

After pursuing a very simmilar idea several years ago, even with simple apps I quickly ran into significant complexity, where important (and very consequential) decisions had to made for the developer (aka a "strongly opinionated" framework), particularly around questions of horizontal scalability. That also locked you into a specific cloud/platform/pattern/etc that often just led to teams needing to "eject" back to enough real k8s features that the abstraction became a real source of extra complexity. We ended up abandoning the project.

I do wish them luck though.

bovermyer• 3 years ago

I like the idea of giving developers a template format they could follow, and then those templates would be transpiled into Kubernetes manifests by the platform.

I would not use docker-compose.yaml for that, though.

linuxftw3• 3 years ago

If you have application developers that can't grok what a deployment is and what a service is, fire them. Just fire them. The "run a container" part of k8s is a solved problem that we never need to revisit. There are much more interesting areas to tease apart, like networking and service mesh, and there's some room for idiot proofing those for developers (OpenShift has some nice tooling in this area, among others).

jackblemming• 3 years ago

Are you aware at some point in your career you didn't know what those things were either?

diarrhea• 3 years ago

Perhaps they were fired for it and hold some fetishised, eternal grudge.

ehutch79• 3 years ago

grok means 'to come to understand' as in to learn.

So if your developers can't learn this stuff...

alex_sf• 3 years ago

At some point we also didn't know what a terminal was, how to type, or how to read. That doesn't mean they aren't baseline expectations for a software developer.

lisasays• 3 years ago

If you have application developers that can't grok what a deployment is and what a service is, fire them. Just fire them.

No - fire their managers, and their managers as needed.

By definition, that's where the responsibility lies. Not with the leaf-node developers. Who are doing exactly as they've been trained and/or vetted by their managers.

rrgok• 3 years ago

This is exactly the attitude that I've been complaining in other threads in HN. What else should the application developer know? How to fill taxes for the company?

I hope the people who manage your k8s infrastructure know how to develop the application they are deploying. If not, just fire your k8s teams. You cannot became k8s person without knowing how to develop the application, there are more interesting areas...

That sounds dumb, isn't it?

linuxftw3• 3 years ago

Writing a k8s deployment is like writing fizzbuzz. If your k8s team can't write fizzbuzz, yes, fire them.

itherseed• 3 years ago

I just want to point out that already exist a tool that convert a docker-compose.yml file to several k8s manifests: https://kompose.io . You have to adjust some details because is not perfect, obviously.

chrissoundz• 3 years ago

"Kubernetes manifests are an unnecessarily detailed abstraction for most software developers. Developers don't need as much control. Therefore, they need higher-level abstractions, which hide more detail."

Agreed... Though I think there is a different solution. Instead of coming up with all these hardcoded formats and standards (docker-compose, k8s, etc, helm, ansible) which essentially are just 'functions' that map X to Y.

Instead of defining 'docker-compose' files, just define the containers itself. And if required, map that to a docker-compose, or k8s manifest or whatever. Remove all these unnecessary layers that aren't really that helpful - because each standard or implementation has it's own parser, error handling etc - why bother with all this?

And just use something like Nix for this. Or just define it as a common python value. Otherwise we end up building all sorts of templating and ugly programming-in-yaml. You could take a look at https://docs.hercules-ci.com/arion/#_arion_compose_nix to see what this might look like - uses Nix to do something like docker-compose.

osigurdson• 3 years ago

One criticism I have regarding this article is the intended audience surely does not need a detailed explanation of what an abstraction is (complete with dog, golden retriever and chihuahua examples).

arthurcolle• 3 years ago

Kind of a weird set of examples to describe the concept of an abstraction at all... I hate contrived examples. It was my least favorite part of all of my coursework, GenEds included.

rabuse2• 3 years ago

At my current company, we use Docker Swarm with multiple compose files to run all of our services. It's incredibly easy to scale up or down services, and if you need to get more intricate with service details/configs, it allows for that. Love it.

ianbutler• 3 years ago

Shameless plug for my own attempt at doing something like this, but creating a higher level abstraction on top of helm and terraform for composing different services and projects to quickly get projects up and running on Kubernetes with sensible defaults and the ability to quickly break out into configuring the helm charts if you need to handle something special.

https://github.com/TorbFoundry/torb

0xbadcafebee• 3 years ago

I have to chuckle that they're still referring to docker-compose.yml as opposed to compose.yaml, and docker-compose as opposed to docker compose. The legacy files and commands will be EOL in June.

https://docs.docker.com/compose/compose-file/

hikerrrr• 3 years ago

Uhh hellooo Terraform? I was just discussing docker-compose today, Terraform has the advantage of State Management..

Terraform is just as easy, but probably a combination of the two docker-compose and terraform are much more effective.. Universal Infrastructure interface is the name of Terraform's game.. They already done what you hoped docker-compose did.

Terraform deserves serious kudos for interoperation between all the clouds, docker, vmware, and literally tons of API modules.. Their Infrastructure Modules are nothing to laugh about in a JSON compatible example.. Serious, they even give Ansible a run for their money when Red Hat was all the blaze.

Randomdevops• 3 years ago

If you say interface there needs to be an implementation; and docker already does that with contexts:

  https://docs.docker.com/engine/context/working-with-contexts/

Which allows pushing to AWS, Azure and K8S.

Of course each implementation is opinionated, AWS for example is deploying to ECS with ALBs.

And it is all good for a list of services, but when you add configuration it becomes a mess:

  secrets:
    foo:
      name: "arn:aws:secretsmanager:eu-west-3:1234:secret:foo-ABC123"
      x-aws-keys:
        - "bar"

So now you can no longer use your neat docker-compose file to deploy locally on your docker as it is referring to an external secret.

So to keep the interface clean you would need to split it up in a Service model and a Deployment model that exposes the needed configuration for the target platform plus the service configuration for the specific environment.

Or something...

Thinking of something generic that works in all cases is hard and complex...and opinionated hence the many platforms, I think.

I posted this a while ago https://news.ycombinator.com/item?id=34225669

rad_gruchalski• 3 years ago

Docker compose universal? Hello kafka-1, kafka-2, kafka-3 and various compose files depending on the cluster size. Thanks, compose is nice for local dev and as a „this is a minimum of a reference to make the stack talk together”, but no. Compose is missing so much. Services, rollouts, instance count, secrets.

Docker-compose.yml as a universal infrastructure interface

Related Stories