BuildKit in depth: Docker's build engine explained

https://depot.dev/blog/buildkit-in-depth

by kylegalbraith • 2 years ago

174 44 2 years ago

inetknght• 2 years ago

Buildx also has a completely different build output and it's completely useless for anything. Use `--output=plain`.

Also, buildx won't commit the intermediate layers during the build. So if something fails, you can't just grab the previous intermediate layer and do `docker run --entrypoint /bin/bash` on that layer to poke around.

I've had cases where BuildKit will get stuck and `docker buildx prune -a` will leave things around. The way to fix it is with `systemctl restart containerd.io`. This seems to happen a lot when I interrupt the build (eg, I know it will fail and need to restart it with a fix), but it's also happened on successful builds.

All of that is on top of docker itself having extremely poor error messages.

I really do not like docker or buildx.

adamgordonbell5• 2 years ago

If you build your dockerfile with earthly, we have a -i flag that will pop you into a shell on a failure, and we are using buildkit, so it's totally possible.

4oo4• 2 years ago

Well said, these are the reasons I use podman/buildah now.

EdwardDiego• 2 years ago

Was it a drop in replacement or did you need to give things a jiggle to get it going?

throwaway894345• 2 years ago

Honestly when I dug into the podman/buildah code it was just differently cursed. It was reallly disappointing.

cpuguy83• 2 years ago

"BUILDX_EXPERIMENTAL=1 docker buildx build <args> --invoke /bin/sh"

There's actually a whole debugging client available.

radus• 2 years ago

This is great! I’ve been waiting a long time for this and it seems like more debug features are in the pipeline too: https://github.com/docker/buildx/issues/1104

Docs: https://docs.docker.com/engine/reference/commandline/buildx_...

mdaniel2• 2 years ago

and its "oh, you want multi-arch, do you?" friend. While prosecuting this <https://gitlab.com/gitlab-org/gitlab/-/issues/339567> I learned that they bundle binfmt into the buildkit docker image <https://github.com/moby/buildkit/blob/v0.12.5/Dockerfile#L11...> which itself is compiled against a static copy of qemu <https://github.com/tonistiigi/binfmt/blob/deploy/v6.2.0-26/D...> with various /proc/sys/fs/binfmt_misc wizardry <https://github.com/tonistiigi/binfmt/blob/deploy/v6.2.0-26/c...>

We ended up switching to using multiarch/qemu-user-static:7.2.0-1 <https://hub.docker.com/layers/multiarch/qemu-user-static/7.2...> which also mutates binfmt_misc in buildx's context in order to exec the static copy of qemu in it but I find their shell script much more legible about what's going on: <https://github.com/multiarch/qemu-user-static/blob/v7.2.0-1/...> not to mention the fact that it more clearly represents the recent qemu version versus secreting it away inside the moby/buildkit:heheh-good-luck-friend

They have an entire GH label for that: https://github.com/docker/buildx/labels/area%2Fqemu and my journey into the buildx sewers started from https://github.com/docker/buildx/issues/1170

zamalek• 2 years ago

I see you're trying to do multi-arch builds. At reasonable (on bare metal) speeds you say?

Delete the entire docker stack. It's utter shite. Then install Podman and Buildah, and rejoice! For all the evil buildx magic can now be done explicitly.

This solution was born when trying to do native builds on CircleCI - where you do not have a buildx cluster with multiple architectures. You have to build the archs as independent build steps, then pull them together afterwards. This is something that Docker never considered, and Docker only functions on the happy path. Buildah has multiarch manipulation commands: https://danmanners.com/posts/2022-01-buildah-multi-arch/ (not a plug, first blog on search)

We migrated to GHA, but still use separate builders to avoid the Docker trite. We also ditched DCT (Docker signing) for Cosign because - again - DCT is home-grown garbage.

AkihiroSuda• 2 years ago

Also, buildx won't commit the intermediate layers during the build. So if something fails, you can't just grab the previous intermediate layer and do `docker run --entrypoint /bin/bash` on that layer to poke around.

`buildg debug` (Dockerfile debugger based on BuildKit) to rescue: https://github.com/ktock/buildg

adamgordonbell5• 2 years ago

I did not know about the CargoWharf frontend!

Writing your own front end to BuildKit can be pretty simple. At my work, this is sort of a starter task for everyone who joins the team.

My frontend was based on intercal and I don't recommend anyone use it but it was fun to play around with[1].

[1] https://github.com/adamgordonbell/compiling-containers/tree/...

notso411• 2 years ago

Really? Just busy work? Senior engineers incapable of thinking of good onboarding tasks?

adamgordonbell5• 2 years ago

Nope, i mean i can see how youd think that, but familarity with BuildKit is just part of the job.

We are building on top of it, and sometimes adding to it.

code_biologist• 2 years ago

It seems sane. He works at Earthly, a containerized build platform: https://earthly.dev/

nicornk• 2 years ago

So how did you „further optimized [buildkit] to build Docker images up to 40x faster“

maxmcd• 2 years ago

hopefully depot will reply, but from my perspective it is mostly laid out on their homepage. they are comparing against builds in other CI products that use network-backed disks, virtualized hardware, and don’t keep a layer cache around. Depot provides fast hardware and disks and is good at making the layer cache available for subsequent builds.

You could likely get very similar performance by provisioning a single host with good hardware and simply leverage the on-host cache.

kylegalbraith4• 2 years ago

Hey, thanks for the question! Depot co-founder here.

We've optimized BuildKit for remote container builds with Depot. So we've added things like a different `--load` for pulling the image back that is optimized to only pull the layers back that have actually changed between the build and what is on the client. We've also done things like automatically supporting eStargz, adding the ability to `--push` and `--load` at the same time, and the ability to push to multiple registries in parallel.

We've removed saving/loading layer cache over the network. Instead, the BuildKit builder is ephemeral, and we orchestrate the cache across builds by persisting the layer cache to Ceph and reattaching it on the next build.

The largest speedup with Depot is that we build on native CPUs so that you can avoid emulation. We run native Intel and Arm builders with 16 CPUs and 32GB of memory inside of AWS. We also have the ability to run these builders in your own cloud account with a self-hosted data plane.

So the bulk of the speed comes from persisting layer cache across builds with Ceph and native CPUs. The optimized portions of BuildKit really help post-build currently. That said, we are working on some things in the middle of the build related to the DAG structure of BuildKit that will also optimize up in front of the build.

mdaniel2• 2 years ago

with Ceph

Seeing that reminded me of some healthy discussion in https://news.ycombinator.com/item?id=39235593 (SeaweedFS fast distributed storage system for blobs, objects, files and datalake) that may interest you. control-f for "Ceph" to see why it caught my eye

codethief6• 2 years ago

Does anyone have a good reference on how exactly Docker/BuildKit/Podman/… build Dockerfiles or, more generally, run containers using the underlying OCI container runtime (runc/crun/…)? I think I have a rough understanding but I'm particularly interested in how exactly the rootless setup works, the details of layer caching (beyond "it uses overlayfs"), etc.

ttymck2• 2 years ago

I got familiar with container intervals via, basically, "building images without docker"

One example: https://ravichaganti.com/blog/2022-11-28-building-container-...

c0wb0yc0d3r• 2 years ago

Did you ever try running any containers without pushing to a registry? That is the part I can't wrap my head around yet? How do I get the run time to start the container locally?

codethief6• 2 years ago

Thanks for the link!

I got familiar with container intervals via, basically, "building images without docker"

That's been my approach, as well. However, right now I'm stuck at getting overlayfs to work without privileges (easy: use `unshare`), while not breaking rootless runc (apparently not so easy).

kylegalbraith4• 2 years ago

Not to shill another blog post of ours in here, but maybe this is helpful.

It's not exactly what you're asking for as it relates to running containers via runc. But this walks through how OCI layers are actually built up behind the scenes. In case it's helpful: https://depot.dev/blog/building-container-layers-from-scratc...

codethief6• 2 years ago

Thanks, I had come across that post already but I find the structure of OCI images to be the easy part to understand. The tougher part is to get runc to run everything without root. :)

Anyway, fortunately I seem to have found a solution for now (running runc with an overlay rootfs without root), see the link in the other sibling/nephew comment I posted.

verdverm2• 2 years ago

Dagger (https://dagger.io) is a great way to use BuildKit through language SDKs. It's such a better paradigm than Dockerfiles, I cannot imagine going back.

Dagger is by the same folks that brought us Docker. This is their fresh take on solving the problem of container building and much more. BuildKit can more than build images and Dagger unlocks it for you.

DavyJone• 2 years ago

I like the idea of Dagger and have been following for a while, but it seems like its tailored to get people to use their Cloud product. eg. Caching.

Is that not the case?

verdverm2• 2 years ago

Caching is definitely their product and one of the main benefits.

1. You can get the developer UX improvements without shared caching

2. You can run it in k8s and get shared caches there. I believe it should support any BuildKit caching solutions, and they will help you figure that out in Discord

kylegalbraith4• 2 years ago

Dagger is quite cool, and we've been doing some collaboration with their new Zenith stuff. We actually have a module where you can route an image build in your Dagger pipeline to Depot and do interesting things with the result as we stage your image in an ephemeral registry [0].

[0] https://github.com/depot/daggerverse/tree/main/depot

pimterry• 2 years ago

I've spent a bunch of time playing around with Docker's API directly. BuildKit is notable because its API is completely undocumented AFAICT - certainly the API listed under https://docs.docker.com/engine/api/v1.44/#tag/Image/operatio... is just the API for a normal non-BuildKit build. If you want a BuildKit build, you have to use the Docker CLI.

I did some digging and wiresharking, and I'm pretty sure it's undocumented because the API is _insane_. It starts as HTTP/1 from the Docker client to the Docker engine, but a key BuildKit feature is that the engine pulls files & data from the client on-demand, which is hard in a normal REST API, so how does it do that? By renegotiating the connection to flip the direction after the client connects.

That means: the client sends an HTTP/1 request, the server offers to upgrade to HTTP/2 by in reverse, and then the server becomes an HTTP client and the client becomes an HTTP server, still on the same existing connection. All actual communication then happens as gRPC, but backwards.

Absolute madness, and very difficult to document or support in 3rd party SDKs (and so they haven't) but it's very clever. Some more context here: https://twitter.com/i/web/status/1423353288129396740

kylegalbraith4• 2 years ago

This is one of the things that led us to write this post! It's almost all undocumented, as you're mentioning, and there are quite a lot of complexities underneath the surface that often feel very inefficient as well.

We're working on another project in this realm that might interest you. Happy to send over more details via email if your interested in better build APIs. My contact info is in my profile.

potamic• 2 years ago

Staged docker builds seem like a bad idea. Why would you want to tie your build specification with your runtime specification which are inherently different concerns? You lose out on independent evolution and isolation between the two. And for organizations, it forces a structure where a single team needs to manage both.

What problems does it solve that cannot be easily achieved with separate build and runtime files?

basicallybones• 2 years ago

I use both, so here is what I'm doing and why I find value in staged Docker builds.

I am running a (relatively new) NX monorepo, which deploys ~6 microservices to various K8s environments using Skaffold. All services are Typescript, Node 20.11, with pnpm as a package manager.

I have separate build files for certain common base images, which mostly are Node-($v)-alpine with a few CI/common dependencies installed. I use Skaffold to compose the image builds.

When I Dockerize each service, I need to install dependencies for that specific service only, and in that case the install/build specification must be the same as the runtime specification. Rather than creating separate build image files and composing them with Skaffold, I simply use a build stage to install the microservice dependencies (using one of the base images with NX and PNPM), then copy them to the final image (which does not have those deps). This is a pretty common pattern, and I like working with it. The advantages to build stages are readability and simplicity (no extra orchestration needed), and the whole setup makes it very difficult to make mistakes.

BuildKit in depth: Docker's build engine explained

Related Stories