The Magic Nix Cache, a GitHub Action for speeding up your Nix workflows

https://determinate.systems/posts/magic-nix-cache

by biggestlou • 3 years ago

195 68 3 years ago

bketelsen• 3 years ago

this looks awesome, can't wait to try it out.

grhmc11• 3 years ago

Graham Christensen here, cofounder of DetSys. Happy to answer any questions! The Magic Nix Cache has been a huge boon to us internally, and we're really excited to share it with the world today.

zetalyrae• 3 years ago

Does it work with nix-shell? I don't know how to use flakes yet.

grhmc11• 3 years ago

Great question. Yes. Anything that Nix builds during your workflow will get cached. Give it a try and let me know how it goes?

biggestlou4• 3 years ago

Yes! The Action isn't directly aware of what Nix itself is doing or which commands are being run; it's only aware of the Nix store. So whether you're using flakes or channels it works the same.

mikepurvis4• 3 years ago

As someone who has led a ~2yr adoption of Nix in an enterprise scenario, I'm really glad for more projects that help with bridging the gap between Nix and containers/k8s.

You can get fast startups if you're willing to define your containers upfront (dockerTools, nix2container) and/or adopt a dynamic container-server (nixery, flakehub). And you can get reasonably fast substitutions if your binary cache is on MinIO in the same cluster as the workers.

But I feel like there's still room for a "magic" /nix/store that skips the copying and decompression stage altogether— something that works using standard nix invocations (like Magic Nix Cache), but presents itself as a Kubernetes Volume, so that in cases where a path already exists on-node, the existing files (in the cache pod) are simply mounted/served directly into whatever container ran a nix command.

I don't feel like I really know enough about either k8s or nix to assess the practicality of such a thing, but the thought of lightning-fast substitutions for arbitrary Nix workflows is massively appealing.

grhmc11• 3 years ago

It is great to hear about your work in the space!

Several years ago I looked at implementing a custom component for k8s which would exchange nix store paths instead of containers, substitute, and bind mount them in at run time. It was an interesting experiment, but was Quite Difficult to pull off for someone who wasn't already familiar with k8s.

I've seen some projects similar to what you're describing though: the lightning-fast substitutions. It was incredible! They had the benefit of a fabulously fat network connection, though, and I'm not sure the experience translates very well. We will see!

saghm2• 3 years ago

But I feel like there's still room for a "magic" /nix/store that skips the copying and decompression stage altogether— something that works using standard nix invocations (like Magic Nix Cache), but presents itself as a Kubernetes Volume, so that in cases where a path already exists on-node, the existing files (in the cache pod) are simply mounted/served directly into whatever container ran a nix command.

Mostly a digression since it's not Nix, but I've wondered a bit about this sort of thing when building AUR packages on Arch Linux. The very last step of the builder is to compress the package, which in the vast majority of my uses is followed immediately by installing it, which of course decompresses the package. I've wondered why there isn't some (non-default) option to say "I don't need to keep the package itself around; just install it as soon as its built". I'm sure for my specific use case there's a simpler solution, but I've always wondered if there's a hacky way to get around things more generally by making a tool that can mimic the expected compression API but then creates a "fake" compressed artifact that no-ops (or maybe puts in a valid header followed by non-compressed data) and then injects that implementation into the PATH. You'd be able to invoke it with something like `fakecompress --zstd makepkg -si`, and it would invoke `makepkg -si` with the no-op zstd implementation.

alex_hirner• 3 years ago

Any pointer to flakehub?

mananaysiempre• 3 years ago

Typo: “speed future up Nix workflow runs”.

miduil2• 3 years ago

Nice, really hope I'll find some sweet way to cache similarish with GitLab-CI. Also kinda been thinking about how cool it'd be to run Kubernetes with Nix natively (so instead of a docker layer registry you have nix paths mounted together to overlayfs)

grhmc11• 3 years ago

I think it should be pretty straightforward to make the Magic Nix Cache work on GitLab, too. They have a similar caching API. We'll take a look!

randomblast• 3 years ago

Is the same true for Azure Pipelines? Given Actions forked from Pipelines I'd imagine it would be straightforward.

ivanbakel• 3 years ago

I would be thrilled to see this ported to GitLab CI. This is an incredibly (for lack of a better word) sexy tool on top of Nix, and would perfectly fit a use case we have for using Nix for caching build artifacts.

pxc• 3 years ago

I would love to use this kind of thing with GitLab CI (and AzDO) at work!

tracnar• 3 years ago

Nice. The GitLab caching API is basically saving and restoring a folder, so if you allow pointing to a local folder instead of the GitHub API you should be good!

whateveracct• 3 years ago

I just spun up a gitlab-runner on NixOS (super easy due to how NixOS works)

grhmc11• 3 years ago

Nice! Yeah, we're obviously big fans of NixOS over here :). In cases where build infrastructure is highly ephemeral, this sort of cache would make a lot of sense. We'd love to help get it working there!

miduil2• 3 years ago

Yeah, you are just trading security with those runners though. I want the cache to be secure and not being able to take over other repos/branches/tags ci-jobs.

takeda5• 3 years ago

This is what I'm using with gitlab: https://github.com/takeda/nix-cde/blob/master/contrib/gitlab...

It caches on two levels (instance's /nix/store on EBS and then also binary cache on S3).

kevincox2• 3 years ago

My solution is here: https://kevincox.ca/2022/01/02/nix-in-docker-caching/

I basically expose the daemon socket into the docker container so that it requests builds from the host. It means that everything is cached right on local disk. If you need more oomph than one machine will provide the cache won't be shared between different machines without extra effort but you can do a lot of building on a single machine (especially if a lot of stuff is using Nix so cached).

grhmc11• 3 years ago

One project cut their CI time from 18m to 3m: https://github.com/awkward-squad/hasql-interpolate/actions. I wonder who will see the biggest cut!

Note that when PRs merge to the default branch, their cache doesn't come with them. This is how GitHub Action's cache works, as a security measure. However: subsequent rebuilds will, and PRs off the default branch will too.

iso8859-12• 3 years ago

How does the Magic Nix Cache know which parts of my build are deterministic and which aren't?

I suppose maybe it will only work if I split my build up into multiple steps such that Nix will know to skip those first steps. If Nix knows that, I suppose the Magic Nix Cache also knows?

biggestlou4• 3 years ago

Could you possibly clarify what you mean by "skip those first steps?" Which steps are you referring to?

iso8859-12• 3 years ago

Presumably, the Magic Nix Cache speeds up your build by caching the output of the "first steps" (really leaves in a tree of inputs). Otherwise, how would it work? So by steps, I am referring to the steps that your Nix derivation consists of.

grhmc11• 3 years ago

Nix's design gives good guarantees about reproducibility. It may not be bit-perfect, but in general it is _very_ good. Maybe the Zero to Nix article on Caching, and its linked pages will help? https://zero-to-nix.com/concepts/caching

The long and short of it is a merkle tree of hashed inputs :).

takeda5• 3 years ago

Nix builds things in sandbox (no network access, no file system access etc) this can make creating derivation for application that has build with side effects a frustrating experience, but that brings reproducibility (or at least gets us really close there) so generally this is not a concern.

As for splitting build into steps. I am not entirely sure what are you trying to do, but you should not need to do that. Each package in Nix has a hash, which is generated from things like hash of the source code, of the dependencies, compile flags, system architecture etc. This means that you could have multiple versions of the same package in /nix/store with different compile options or different dependencies. When you need a given dependency you know all the information to generate the hash and can easily know if you need to build it or you can use cached version. This is what makes binary cache a pretty much plug and play and don't need to worry what files to cache or whether you should split build into stages.

aidenn0• 3 years ago

You tell it; when you define a derivation, you are promising it is at least logically deterministic.

dlahoda3• 3 years ago

i replaced cloud autoscaled docker runners with fixed set of machines with nix store cache pee machine. 10x imrovement.

grhmc11• 3 years ago

Yep, that works great for cases where you trust all your contributors. We do the same for some projects internally, too. However, for public projects where you don't it gets a bit ... dicier. Glad to hear that's working for you!

dlahoda3• 3 years ago

we have public oss project. i guess we were not attacked yet.

prs from forks run only main workflows, and these run nix which is kind of isolated enough.

i guess one could attack with some infinite nix store bomb.

esafak2• 3 years ago

Are you saying you replaced Docker with Nix? I am still trying to understand the use case for Nix. Can someone explain? Is it a complete replacement for Docker?

rgoulter2• 3 years ago

Is it a complete replacement for Docker?

Docker mainly does two things: there's Docker images as a way of sharing some container-image, and the container runtime to allow running those images for tasks or services in isolated/fresh ways.

Docker images make it easy to distribute software which runs the same everywhere.

Nix is a package manager which tackles that problem, but without using container images: Nix is for distributing software so it has the same behaviour everywhere.

Nix users are often very enthusiastic about Nix because it also enables all sorts of neat developer experiences. e.g. Nix is great for setting up development environments.

In terms of Nix-vs-Docker, Nix is also capable of building Docker/OCI images. So, you could use Nix instead of writing a Dockerfile.

takeda5• 3 years ago

It's really hard to explain it because Nix is a paradigm shift (I think that's why it is hard initially grasp it).

But you can use it for things like:

- declare your application with all dependencies explicitly, so when someone else wants to build it they can (I would argue this is the primary purpose and rest is just built on top of that)

- common dev environment (so other developers can get the same dev environment as you with all exact same build tools)

- build toolchain (for example if you do embedded environment)

- you could use it as a replacement for homebrew/mac ports

- a configuration file holding your .dot files (home manager)

- if you use NixOS (OS that was built around Nix) then you have OS with a built-in configuration management (i.e. salt/puppet/ansible/chef) that is truly declarative

I think this[1] also shows some crazy stuff you can do with it.

Regarding question around Docker, the great thing is that if you define your application as a nix derivation, then you can easily generate docker that just contains your application and dependencies (the docker in that case is just a deployment unit). The reproducible environment is what docker promised, but practically failed to deliver. Instead of deliver reproducibility, it actually delivered repeatability.

[1] https://youtu.be/6Le0IbPRzOE?t=109

Karupan• 3 years ago

As a complete nix noob, will this help with caching node dependencies? We have a few projects that take over 20mins for a `yarn install && yarn build`. I’ve read setting up Nix for node isn’t that straightforward, but that was a couple of years back. Has anything changed with respect to node projects?

biggestlou4• 3 years ago

You may find something like node2nix helpful (https://github.com/svanderburg/node2nix). This converts your package.json into a Nix expression that can then be cached. You're right that it does require some setup and a bit of Nix knowledge but could yield significant benefits and take a good chunk out of that 20 minutes.

Another option might be to use pnpm instead of Yarn and cache your pnpm dependencies. pnpm actually works a bit like Nix in that it creates a pnpm-lock.yaml file with content-based hashes for the full package.json dependency tree. This enables it to quickly determine which parts of the dependency tree it needs to build and which are already available.

soraminazuki• 3 years ago

Nowadays, there's buildNpmPackage [1]. It's included in Nixpkgs, actively maintained, and easier to work with IMO.

[1] https://github.com/NixOS/nixpkgs/blob/master/doc/languages-f...

duped• 3 years ago

In order to support resolving dependencies to multiple versions (eg A and B depend on C but at incompatible versions), what npm does is drop the version override for C inside the nested node_modules directory (node_modules/A/node_modules/C, for example).

That means that node_modules (as created by a package-lock.json) can't really be cached or be built from caches since it depends on the particular version solution found by npm for a particular project.

So there's only so much that Nix can do. It can cache it about as well as using a naive caching scheme with actions/upload-artifact or similar (create a tarball of your node_modules and just cache it across runs, update when you need to).

Basically node_modules is inherently large. If you want better performance for caching dependencies use a better programming language environment.

crabmusket• 3 years ago

Or fewer dependencies!

maxloh• 3 years ago

That is supported officially:

  - uses: actions/cache@v3
    with:
      # npm cache files are stored in `~/.npm` on Linux/macOS
      path: ~/.npm
      key: ${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}

wereHamster2• 3 years ago

I have given up installing node dependencies via Nix. I use Nix to make nodejs available but then install node deps with npm/yarn/pnpm.

Node packages sometimes pull additional files from the internet in a postinstall script, or do other funky stuff that's incompatible with Nix. So the idea that you can construct a pure derivation from a package-lock.json or yarn.lock file is a pipe dream.

SirensOfTitan• 3 years ago

Slightly off-topic, but how do folks around here create production packages in nix? I use nix for my dev shells and machine configuration, but haven't yet built production packages using it.

More concretely, let's say you have a python backend that uses poetry. Do you just use `poetry install` in your derivation for python-deps? Do you use something like poetry2nix or node2nix and do all of your package management in nix?

grhmc11• 3 years ago

"It depends." poetry2nix is pretty good! Generally, exporting to an OCI image, or targeting a production environment that can run Nix closures natively is the way we go about it. Many of our services use buildLayeredImage[0] and target Fly.io.

0: https://grahamc.com/blog/nix-and-layered-docker-images/

flurie• 3 years ago

targeting a production environment that can run Nix closures natively

Do you have some examples for this? The ones that come to mind are things like bare metal/vms with nix, or perhaps disnix, but those are a pretty hard sell over more popular orchestration systems, and I’d like to have more alternatives.

takeda5• 3 years ago

So I so far use it for dev environment and also for CI pipeline (the benefit is that Nix allows caching between runs, it helps with things like:

- getting dev environment on CI to be identical to user dev - with minor changes the project is not rebuilt or or rebuilt minimally - the caching works across branches, so for example merging a feature branch to master, if nothing changes the build on master will be very quick

I created something similar to nix-cache for gitlab, but I had to create a dedicated runner running NixOS.

If I could use NixOS for deployment, at that point I would just point the same binary cache to the machine and use the same derivation to build the app. Because the app was already build by CI, it would just download the compiled version. No need for artifactory or similar. In that scenario (you using poetry) you probably would just use poetry2nix to generate the application.

If the OS is not NixOS, but you still want to deploy via nix, then IMO this[2] looks interesting, basically it packages everything in self extracting archive. That you can extract and then run the app.

Other alternatives are these bundlers[3], which includes building toArx (works in a way similar to the previous one but pretends everything is in a single file), RPM, DEB, docker (you would have more control over it if you would use the code directly instead of a bundler though)

And the last option (probably the most obvious one) is that you can simply just use the tool to build the package. Since you're using poetry, then you can generate a wheel from it.

[1] https://github.com/takeda/nix-cde/blob/master/contrib/gitlab...

[2] https://github.com/Ninlives/relocatable.nix

[3] https://github.com/NixOS/bundlers/blob/master/flake.nix

sgammon3• 3 years ago

this is awesome. i've yet to try Nix, but we are starting a remote cache service and we'd love to be compatible with Nix' caching mechanism

we're at https://less.build if you want to take a peek -- we will look at adding S3 support! :)

sgammon3• 3 years ago

side note i am happy anytime build caching is on the front page of hckrnews for any reason

a_t482• 3 years ago

I'm all for faster builds...but how does your product differ from using something like https://ccache.dev/manual/4.7.html#_remote_storage_backends ? (I ask specifically because you advertise being a drop in ccache replacement, or similar)

sgammon3• 3 years ago

we are drop-in compatible with ccache, in the sense that you can plug ccache right in (as an HTTP backend), and gain features like authorization policies, analytics, and global caching backed by Cloudflare.

that's where we are beginning and then we're working to add spicier features on from there. but first and foremost, we want to serve the ccache/gradle cache crowd with a fantastic protocol-agnostic backend which "just works" and pays for itself in terms of time saved.

and thanks for asking :) we are very new

wereHamster2• 3 years ago

Just this past week I thought about setting up custom github runners on NixOS machines so that Nix is pre-installed (doesn't need to be installed via a github action) and so that the Nix store can be shared between runs. Though I don't really want to manage the machines, so I might give this new github action a try…

yakshaving_jgt• 3 years ago

The Magic Nix Cache, a GitHub Action for speeding up your Nix workflows

Related Stories