Docker on MacOS is slow and how to fix it

https://www.paolomainardi.com/posts/docker-performance-macos/

by riccardomc • 4 years ago

193 203 4 years ago

paolomainardi.com

EdwardDiego2• 4 years ago

My preferred fix: don't develop in Docker.

sieabahlpark• 4 years ago

Have fun managing osx and Linux dependencies then when you could just maintain one.

notwokeno2• 4 years ago

When I used to use a mac I just put all my "Linux" (GNU really) dependencies in a prefix and that worked pretty well. Docker is kind of overkill for what people use it for IMO.

princevegeta892• 4 years ago

And also there is some additional fun dealing with broken or incompatible packages whenever you upgrade OSX

pjmlp9• 4 years ago

Indeed, one can maintain only a macOS application.

nicoburns2• 4 years ago

In the languages I work with (primarily JavaScript and Rust), cross-platform compatibility tends to be pretty much a non-issue. I develop on macOS, deploy on Linux, and it Just Works. No extra work required.

acdha• 4 years ago

That really hasn’t been an issue for me since Homebrew came out a decade ago - using Python, Rust, Node, Java, etc., all of which have mature stories for cross-platform development. The main area I’ve run into problems are legacy projects where it’s not “Linux” but more like “one old Linux distribution with a 32-but binary nobody can reproduce” and Docker is really the least troublesome part of those projects.

Spivak• 4 years ago

If you're not IT, god speed y'all, then it's not all that bad honestly. I set up my stuff with Ansible and 90% of the "porting" work was the mapping between rpms and brew packages.

The only headache I get sometimes is because I have the GNU utils first in the path which makes compilation scripts mad sometimes.

thunky• 4 years ago

My preferred fix: don't develop in Mac.

princevegeta892• 4 years ago

Unfortunately Mac is the best in terms of hardware and portability and for compliance stuff

vforvendettador• 4 years ago

For many Mac is the only sensible option. There's one developer in our company that uses Linux and it's a lot of pain to setup.

Mac has the best balance between coding, utility tools and "other work stuff".

Windows probably on par if not more for "work stuff" but falls badly in the coding & tooling department.

Linux is OK ish for coding and utility but falls behind for "other work stuff" and certainly a pain to just keep it updated.

So in our company, everyone in the development & support team uses Mac (except this one guy who insisted in Linux), most in the sales & marketing team use Windows.

kevinhq6• 4 years ago

this is hard. what if you're on project with

- backend (docker) - needs linux based machine - client app (iOS) - needs xcode on macOS

both are in one repository.

EdwardDiego2• 4 years ago

I agree, but my company is in a very regulated area, so they standarised on Macs because of the very strong controls built into them at a hardware level.

I do miss my thinkpad and Fedora.

RockRobotRock2• 4 years ago

This is an unhelpful and useless comment

mdavidn• 4 years ago

And so is yours?

I do often find myself wondering whether Docker saved developers or system administrators any time. Is Docker really better than building an AMI and provisioning EC2 instances on-demand?

hakanderyal2• 4 years ago

I finally gave up the effects of Docker on performance and battery life and switched to Windows. I still don't have a long lasting battery, but at least performance is better.

rad_gruchalski• 4 years ago

I have been working in a Citrix workspace Windows machine for the past 6 months with a Ubuntu 20 wsl2. I was very much anxious of the experience before onboarding but I have to say, I barely deal with the Windows side of things. Having said that, if I was to move away from mac, I‘d just go directly to Linux.

hakanderyal2• 4 years ago

Gaming is my guilty pleasure. While the gaming ecosystem on Linux is getting better everyday (thanks, Steam!), the limited time I spend on gaming doesn't allow me to tinker, I just want everything to run without limitations.

therealmarv• 4 years ago

Is not this the main way to speed up Docker on Mac: use a beefed up Linux Virtual Machine (VirtualBox, UTM, tart) and run Docker inside this Virtual Machine?

dijit• 4 years ago

the 'DOCKER_HOST' variable (and the fact that all SDK's seem to support it) is honestly the greatest bloody thing in the entire ecosystem.

My workflow for the past 3 years with Docker has been: set up some desktop machine somewhere, configure docker, configure ssh like normal: set DOCKER_HOST=ssh://<tailscale_ip> on my laptop.

Docker responds as if it's local, but I get absurd build/fetch speedup (since the wired connection is faster than Wifi) and it's not running inside a slow VM.

Recently I've been using colima on my Mac natively, but I keep reaching for the DOCKER_HOST option.

silverwind• 4 years ago

I assume port forwarding would be a pain in such a setup, right?

acchow• 4 years ago

This assumes you want a distinct storage drive within your VM.

Many developers prefer to code in their host OS but run the image via Docker for Mac. They also want instant real-time code changes to appear inside the running Docker image. I suppose you could have some of the disk live within the VM and the code portions be memory mapped or Rsynced. I haven’t thought through the downsides.

charrondev• 4 years ago

There was a project for that called docker sync (it could use of 3 mechanisms (one being rsync) to continually keep the files up to date in the container. The problem I found was it would just randomly stop syncing with no warnings or errors of any kind. It was very flaky.

Nowadays make sure you use their new virtual machine thing in docker for Mac and add :cached in your compose file of any mounted volumes and I found that alleviated my issues. It used to be really bad though.

KyeRussell3• 4 years ago

This doesn’t address the root problem. If you want the FS features that the “””native””” Docker Desktop provides, you end up with the same drawbacks re performance.

doctoboggan• 4 years ago

What sort of workloads are people doing where the filesystem access is limiting them? I develop python web apps on a mac and use dockerized postgres and a dockerized flask app. I don't seem to experience any noticeable issues. When I am developing I mount the source code directory as a volume so code edits are synced live into the running docker container.

I also develop frontends using vue, managed by npm. In my experience this doesnt need to be dockerized since npm installs everything in a subdirectory per project. Is there a benefit to running this as a dockerized app?

taf2• 4 years ago

nginx is pretty unusable via docker - I guess file system cache is the issue…

osrec• 4 years ago

You can turn caching off, I believe.

jeffrom• 4 years ago

Workloads with a lot of files, for example a large elixir web app with hot reload / fswatch enabled, have upwards of 20 second page load time. More than enough to mess up my flow.

xtracto• 4 years ago

Magento in docker on Mac is horrible for this reason.

a_t48• 4 years ago

I work on a _large_ C++ codebase on Linux. If I'm on my main (Linux) machine, things are fine, bind mounts are okay. If I'm stuck on my MacBook then compilation performance is...bad. I suspect it's due to heavy filesystem access from the compiler (reading source, writing object files, etc). At some point I need to confirm this by copying in my source directory.

pmontra2• 4 years ago

A customer's Mac with a M1 is only 50% faster than my Intel laptop from 2014 at running Rails tests, because they run in a docker container: 50s vs 75s. The difference between the two machines should be much more than that (CPU, RAM, data bus, etc.)

brodock• 4 years ago

You should try running your dependencies on docker but ruby on the host machine.

zeta0134• 4 years ago

One issue I ran into at my previous employer was pylint on a large Python codebase. Pylint is slow on the best of days, but the difference on an M1 Mac running under docker (to standardize the version and settings across the team) was something like 10x as slow; several *minutes* to lint the codebase, which we absolutely required before code could be checked in. It finally got a lot better when VirtioFS came out, which, when enabled on an arm64 image, sped up filesystem access dramatically; suddenly my lints were taking seconds again.

nyrikki• 4 years ago

Are your Docker images x86_64? On an M1 Mac x86_64 images run under qemu which is very slow, If an ARM64 image is available it should run a bit faster.

nikanj• 4 years ago

Are you linting the whole codebase, or just files changed in the commit?

dgunay• 4 years ago

I tried running our Golang mock generator through Docker on an M1 Mac and it was much slower than running it directly on the host. Probably since it reads every file in the codebase and writes out a file for every interface it finds.

latchkey• 4 years ago

mockery? That thing runs super slow just on its own.

mianos• 4 years ago

Mentioned in the article, node packages that create tens of thousands, sometimes hundreds of thousands of files.

Arbortheus4• 4 years ago

My company has a large python repo. Anyone that develops in this repo on a Mac is considered a sadist because…

Running Pylint on a Linux machine in docker: 3 hours from no cache

Running Pylint on a Mac in docker: 9+ hours from no cache unless VirtioFS is used, which makes it closer to 4 hours.

rootusrootus• 4 years ago

Funny thing is that I switched from virtualbox for Vagrant on OSX over to Docker because vbox file operations were so incredibly slow.

andix5• 4 years ago

Often overlooked: there is also podman machine and podman desktop (for windows and macOS). It is not as fancy as docker, but fully free and open source.

It provides docker compatibility to some extent, you don’t need a license and it’s much less heavy than docker desktop. If you need kubernetes, there is also minikube, which provides a lot of options.

Most of the things discussed in this article still apply for podman machine and minikube.

dangoor• 4 years ago

There's also Rancher Desktop. I don't know if it's less heavy, but it doesn't require a license and also includes k8s.

https://rancherdesktop.io/

andix5• 4 years ago

I didn’t try it out yet. But usually thing from rancher are awesome.

laserlight• 4 years ago

I've recently tried podman on a Mac and `podman machine start` failed with an “unknown error”. It was a disappointment.

depereo• 4 years ago

I had similar issues when podman desktop first came out. Working much better on a re-try this week.

null0ranje• 4 years ago

I want podman to be a drop-in replacement, but it just isn’t there. I’ve wasted more time running down random errors I podman, so much so that I’ve switched back. Maybe in another year or two…

Arbortheus4• 4 years ago

Podman has been a massive disappointment for me. I stupidly gave up my docker desktop because the company was trying to make cost savings and it was supposedly a drop-in replacement.

Some short-lived containers like our repo’s linter takes easily 4x as long to run in podman as it did with Docker. Immediately I have lost productivity.

It’s incredibly unreliable, every time I start my computer I have to podman machine stop then podman machine start because there’s something broken about how it gets initialised at startup. I’ve spent ages debugging random broken functionality.

It doesn’t support docker-compose. There’s a community project called podman-compose, but it’s not great because it won’t do stuff like build containers concurrently, and it has random weird quirks about volumes already existing when you do podman-compose up —-build whereas docker doesn’t complain for the same compose file.

Overall podman has been a massive regret for me, and I wish I hadn’t given up my docker desktop just to save a minuscule amount of money.

andix5• 4 years ago

I haven’t used it much I have to admit. But I also experienced the issues you are talking about.

I thought it was because I also have docker desktop on the same machine, but probably that’s not an issue at all and it’s podman remote that is unreliable.

sitta2• 4 years ago

When did you last use it? It _does_ support docker compose. Has for a while now actually.

I love it on Linux. The Mac version is not as smooth yet, but, for my use case, still works a hell of a lot better than docker desktop. There is something deeply wrong with docker desktop's networking, and I literally have to restart it almost every time I make a change to one of our services. Not an issue with podman.

SirensOfTitan• 4 years ago

The most recent version of Colima supports both virtiofs and mac's native virtualization framework (macOS 12.5+). I get totally acceptable performance using it.

jscheel4• 4 years ago

Docker is cripplingly slow on MacOS. I have a maxed out 16" mbp... starting rspec on our app takes 55-60 seconds. Compare that to my coworkers on Linux and Windows, who both see sub 10 second boots, and it's absolutely impossible to be ok with those numbers.

kevinhq6• 4 years ago

this is very true. not to mention it's very slow on its networking... i've given up docker on macOS long ago. If I was to use docker, I'll switch to my debian laptop. way much much faster.

tut-urut-utut• 4 years ago

Happy to give up Docker, but still using MacOS as a daily driver?

rubyist5eva4• 4 years ago

I have a nearly maxed out Mac Studio and my experience was the same, until I got my work to fork out a license for parallels and I just installed docker in an ubuntu VM and configured my DOCKER_HOST on the host to talk to the vm. Now it's crazy fast.

chrisbolt• 4 years ago

How does file sharing performance compare?

jscheel4• 4 years ago

You are maintaining all your files in the VM though, right? I'm playing with setting up a sync from vm to host so I can do that but still use my local tools.

navels• 4 years ago

Might be worth digging into why your rspec startup times are so slow. I recently dockerized our dev setup with separate containers for our rails backend, mysql, localstack, and rabbitmq. With Docker Desktop configured to use VirtioFS and the native virtualization framework (which is now I think the default), speeds are great. I've left my non-M1 coworkers in the dust.

jscheel4• 4 years ago

It's not just rspec, any rails boot is slow. We have a pretty large project with a significant number of files and a significant number of gems, so the initial boot that loads all the constants etc is what does it.

zenosmosis• 4 years ago

I use Docker inside of Parallels running Linux and get far better performance than Docker Desktop.

kevinhq6• 4 years ago

We had to abandon Docker because we had folks with macOS on the team. But, other tasks (email, conference calling, scanning, word, upgrading without breakage) came with more friction, and those tend to fill up ever larger shares of my day.

source: https://kvz.io/macos-install.html

tropshop• 4 years ago

I have Apple Silicon but develop on docker x86. The game changer for me was macOS Ventura with rosetta support for linux vms.

I use UTM to run Debian 11 ARM. The update-binfmts command is absolutely magical, docker images will happily run both arm and x86 binaries.

Battery lasts all day and the machine stays ice-cold.

https://docs.getutm.app/advanced/rosetta/

KyeRussell3• 4 years ago

It’s no surprise that these replies have seen the usual jarring unfriendly personalities come out of the woodwork to advocate for the absolute use of desktop Linux.

TimTheTinker• 4 years ago

My preferred fix: pay for a Parallels Pro license and run Ubuntu on a VM, then run docker there. The VM is configured to start on login and run in the background.

I have the Remote SSH plugin set up in VSCode, a `vmlogin` alias set up in bash, and all container ports forwarded in the VM's config.

rubyist5eva4• 4 years ago

Can vouch for this approach, been using it for rails development for the past 6 months after my work swapped my macbook pro for an M1 Max Mac Studio and it's been solid.

kdrag0n7• 4 years ago

Funny that this came up — shameless plug: I've actually been working on a new Linux+Docker+Kubernetes solution for macOS recently! Already added quite a few improvements over existing apps including Docker Desktop, Rancher, Colima, etc:

- Fast networking: 30 Gbps! vs. 150 Mbps with Docker VPNKit. Full VPN compatibility, IPv6, ping, ICMP and UDP traceroute, and half-open TCP connections.

- Bidirectional filesystem sharing: fast VirtioFS to access macOS from Linux, but also a mount to access the Linux filesystem from macOS. This setup can help with performance: for example, you could store code in Linux and edit it from macOS with VS Code (which can take the performance hit of sharing), so the container runs with native FS speed.

- Not limited to Docker or Kubernetes. You can run multiple full Linux distros as system containers (like WSL) so they share resources.

- Fast x86 emulation with Rosetta

- Much lower background CPU usage. Only ~0.05% CPU usage and 2-5 idle wakeups per second — less than most apps, while Docker wakes up ~120 times per second. Made possible with low-level kernel optimizations. Also, no Electron!

- Better solutions to other problems that can occur on macOS: clock drift is corrected monotonically, dynamic disk size, and more I'm working on now. Will look into memory usage too, although I can't guarantee a good fix for that.

- No root needed.

Planning to release this as a paid app in January. Not OSS, but I think the value proposition is pretty good and there will be a free trial. Not sure about pricing yet.

If anyone is interested, drop me an email (see bio) and I'll let you know when this is ready for testing :)

Also, feel free to ask questions here or let me know if there are other warts you'd like to see fixed.

maccard• 4 years ago

What are the tradeoffs?

jitl2• 4 years ago

It cost money sounds like the big one

kdrag0n7• 4 years ago

In general, I don't expect anything to be worse than existing solutions, but not everything will be better.

Enabling Rosetta can have a minor performance hit on memory-intensive workloads in the VM (not only x86 ones) because of TSO memory ordering, so it'll be optional. Hypervisor.framework doesn't have an API for third-party VMMs to set this and doesn't seem to let the VM modify ACTLR_EL1 either, so unless I can find a private API for it, I'm stuck with Virtualization.framework's limitation of Rosetta being either on or off for the entire VM at boot time.

Memory usage is probably the biggest uncertainty right now. It should be at least slightly better, but I'm not sure if I can improve it much more due to Virtualization.framework limitations. Still looking into it.

Networking is implemented with my custom userspace proxy for VPN compatibility. Servers are forwarded to localhost automatically, but you can't connect to the VM by IP because the network doesn't exist from the host's perspective. I've ran into too many issues with Apple's NAT setup and host-only networking is a private API, so this is postponed for now. Should be able to do better with root.

Graphics won't be supported at launch, but I could look into it later if there's interest. Not sure how feasible acceleration will be if I can't find a way around having to use Virtualization.framework.

Let me know if there's anything specific that I missed!

rickette• 4 years ago

Biggest question: is it backwards compatible with Docker? Docker CLI and docker-compose is used in tons of scripts. To have any change of this being adopted in a team setting it needs to be a drop in replacement.

kdrag0n7• 4 years ago

Yes, Docker CLI will be configured to talk to the VM.

llIIllIIllIIl• 4 years ago

Is there any way to join the waitlist for the software?

kdrag0n7• 4 years ago

No formal waitlist yet, but drop me an email (see bio) and I'll let you know when there is!

paolomainardi• 4 years ago

Blog post author here, cannot wait to see the solution out in the wild, it will be open-source ? Can you let me try it ?

kdrag0n7• 4 years ago

It won't be open-source, sorry.

Planning to release this as a paid app in January. Not OSS, but I think the value proposition is pretty good and there will be a free trial. Not sure about pricing yet.
If anyone is interested, drop me an email (see bio) and I'll let you know when this is ready for testing :)

viraptor• 4 years ago

Docker VPNKit, fast VirtioFS, Much lower background CPU usage

Are there some new Linux drivers involved, or is this "just" a better tuned VM?

kdrag0n7• 4 years ago

No new drivers, but I did make some changes to the Linux kernel. It's mostly a better tuned VM and services on both sides, e.g. a custom fast networking stack in place of Docker's VPNKit.

(Also, by "fast VirtioFS", I meant the same VirtioFS implementation tested in the article because it's faster than other solutions — sorry if it wasn't clear.)

wood-porch• 4 years ago

A project I made solved this by running docker on AWS and doing two-way file sync on changes. Runs quite nicely and transparently.

https://github.com/lime-green/remote-docker-aws

Lots of benefits: speed, battery, fan noise

pak9rabid• 4 years ago

And money

trillic2• 4 years ago

I use a shitbox of a thinkpad with my dev environment in the cloud, but they run Linux so I can just as easily run it locally if I have no connection. I buy them for $300-500 on eBay, upgrade them to at least 16gb of RAM and a 1TB Samsung SATA SSD. I have 2 cold spares ready to go, credentials loaded, just need to update and sync a few git repos. 14 inch 1080p IPS displays, i5 processor, 16GB of RAM.

The money I save not paying for Apple laptops could pay for a crazy overpowered dev VM until the end of time.

I used Apple laptops for about 10 years until about 5 keyboard replacements with the butterfly switch debacle.

xwowsersx• 4 years ago

I have "use virtualization" turned on in Docker Desktop for Mac, but I don't see different options for the file sharing implementation. I'm currently on version 4.14.1 (91661).

Separately, with "use virtualization" turned on, should I also enable "VirtioFS accelerated directory sharing"?

ehutch79• 4 years ago

It’s under experimental features in 4.14. It’s the default where available in 4.15.

You might need to upgrade both docker desktop and macOS.

SanjayMehta• 4 years ago

We gave up on Docker on macOS long ago. Wherever we absolutely need Docker, we just throw a Linux machine at it. Developer time is costlier than hardware.

mccolin• 4 years ago

With VirtioFS on the scene I just don’t have this experience anymore. Docker for Mac is significantly faster than it used to be, particularly when using named volumes.

Mutagen also improved the experience but I prefer VirtioFS as it’s “built-in”

westoque• 4 years ago

this is one of the reasons i changed my OS to linux since docker there works natively. i believe there will always be a bottleneck for peak performance on macOS until we are able to run natively, not virtualized.

aliljet• 4 years ago

Have any of you tried to chase the microvm train instead of docker on MacOS? Thus far, it feels like this is a deeply lost cause with a passing hope that somehow you can hack a solution to nested virtualization and drop into a KVM-style experience on a guest vm and then go a layer deeper with microvms on top of that guest. Oofff. What an absolute horror show.

benfrancom• 4 years ago

I’m interested to see how project finch works out compared to Docker on a Mac. It’s an open source client for container development.

https://github.com/runfinch

When I tried Rancher Desktop it didn’t work so well.

xenoscopic• 4 years ago

If anyone has any questions about Mutagen (or integrating it into their Docker-based workflows), I'm happy to help.

Just one clarification on the article: Mutagen offers Docker Compose integration, not Composer integration (Composer is a PHP package manager). However, as mentioned, DDEV is a great option if you're looking to do containerized PHP development while using Mutagen for performance improvements.

totalhack• 4 years ago

I fixed this problem by not getting a MacBook Pro this year when my last one from 2018 had no battery and two letters falling off the keyboard.

Bonus: paid half what a similarly spec'd M2 Air would cost.

nsonha3• 4 years ago

Why would you want to mount node_modules? it's useless on the host because the binaries are for a different arch

rcarmo• 4 years ago

I stopped using Docker desktop and just forwarded my Docker CLI to a VM’s TCP port the instant I found how it exported my Mac file system wholesale to the Docker VM. Never looked back, and these days I just use sshfs or VSCode to develop remotely (which works everywhere).

ffactory• 4 years ago

Cool blog theme, feel like I've seen it before ;) https://filippo-orru.com/

drcongo• 4 years ago

I have a large, fairly complex Django application that I run in containers on a MBP with an M1 Pro, and it runs twice as fast as the AMD 16 core production server. It was a bit of a dog on an Intel Mac, but everything I use Docker for is blazing fast on Apple Silicon. Is there something about Python apps that makes this different?

Docker on MacOS is slow and how to fix it

Related Stories