mschwaig

I've been thinking if this could be done in Nix and toyed around with the idea just a little bit using nano-gpt.

Hermeticity always seems to mean isolation, but depending on who you ask it does not always mean computing some sort of hash over all build input as the 'identity' of a particular step in the pipeline, like Nix does.

If you do that hash-based identity part, looking up intermediary results and resuming from there happens using this sort of hash.

Does savanah do that, or will it resume where it left off based on a less strict notion of identity?

I could see arguments for either approach.

A consequence of universal healthcare that people don't talk about much is that it turns unhealthy citizens from an individual cost into more of a collective one. So it makes sense that countries with universal healthcare regulate in favor of their citizens as opposed to their food industry, because they're paying for the consequences more directly.

I think outright shortening copyright terms could be a beneficial policy along similar lines.

I would blame how Austria, a very small country, is organized into 9 provinces that actually have their own budget and can pass their own laws on some topics.

Rail service is funded at the federal level, so there's less arguing about who pays for what. Bus service, however, is managed by regional transport associations funded by the provinces. This creates disincentives for cross-province bus routes because no single province wants to pay more than its 'fair' share for a service that primarily benefits voters in another province.

Similar dynamics play out at the city/province level. Take Linz, the provincial capital of Upper Austria: the city has had a social democratic (SPÖ) mayor continuously since 1945, while the province has had a conservative (ÖVP) governor for exactly the same period of 80 years. This disincentivizes the province government from helping to fund public transport within or into the city, because it's a win for social democratic city voters, while the more conservative rural voters would rather take the car anyway since they often can't do the whole trip by public transport.

Arguably the reason for the excellent public transport in the city of Vienna is that they are also their own province. Their mayor/governor, who has been a social democrat as well for the last 80 years, always controls both levels of funding.

If you make a conventional AI agent do packaging and configuration tasks, it has to do one imperative step after the other. While it can forget, it can't really undo the effects of what it already did.

If you purpose-build these tools to work with Nix, in the big picture view how these functional units of composition can affect each other is much more constrained. At the same time within one unit of composition, you can iterate over a whole imperative multi-step process in one go, because you're always rerunning the whole step in a fresh sandbox.

LLMs and Nix work together really well in that way.

Yes, the cycle times are bad and some ecosystems and tasks are a real pain still.

I also agree with you when it comes to the task of auditing every line of Nix code that factors into a given system. Nix doesn't really make things easier there.

The benefit I'm seeing really comes from composition making it easier to share and direct auditing effort.

All of the tricky code that's hard to audit should be relied on and audited by lots of people, while as a result the actual recipe to put together some specific package or service should be easier to audit.

Additionally, I think looking at diffs that represent changes to the system vs reasoning about the effects of changes made through imperative commands that can affect arbitrary parts of the system has similar efficiency gains.

I think as AI gets smarter, defenders should start assembling systems how NixOS does it.

Defenders should not have to engage in an costly and error-prone search of truth about what's actually deployed.

Systems should be composed from building blocks, the security of which can be audited largely independently, verifiably linking all of the source code, patches etc to some form of hardware attestation of the running system.

I think having an accurate, auditable and updatable description of systems in the field like that would be a significant and necessary improvement for defenders.

I'm working on automating software packaging with Nix as one missing piece of the puzzle to make that approach more accessible: https://github.com/mschwaig/vibenix

(I'm also looking for ways to get paid for working on that puzzle.)

I mentioned another alternative to adding flake-specific metadata to data structures that are transferred over the network, as part of the signed traces or otherwise, in a comment on that PR Eelco linked.

It's keeping flake-specific data locally, to guarantee that it matches how the user ended up with the data, not how the builder produced it. I think otherwise from the user POV such data could again look misleading.

As a current PhD student (working on Nix stuff) let me take this opportunity to congratulate you on your successful PhD defense and publicly thank you for your writing. That you write and what you write are inspiring.

I work with Nix a lot, and I had never seen `__findFile`.

It's kind of crazy how much there is to know about Nix. I wish there was a bit less surface area to the language. On the other hand it's really interesting how much specialized knowledge there is in the community around various topics. Some people package things, some people write library code, some write glue code that wraps other build tools, some write VM-based tests, some write generators that transform store paths into things like container images, some just manage their dot files, some are experts for how we deal with some specific proprietary ecosystem like cuda, some write infra code or tools around the Nix code, some work on one of the Nix implementations.

Human-readability was one of the aspects that I enjoyed about using CCL,the Categorical Configuration Language (https://chshersh.com/blog/2025-01-06-the-most-elegant-config...), in one of my projects recently.

It saves you from escaping stuff inside of multiline-strings by using meaningful whitespace.

What I did not like about CCL so much that it leaves a bunch of stuff underspecified. You can make lists and comments with it, but YOU have to decide how.

I wrote a paper about how I think trust should work for software dependencies.

It very much builds on the hash-based cache lookup mechanism this paper calls constructive traces (in contrast to what they call deep constructive traces) to eliminate transitive trust relationships.

https://dl.acm.org/doi/10.1145/3689944.3696169

Thanks, and thank you for reading! :)

I want to try to become an independent researcher, when the funding for my PhD position runs out.

My idea for financing this is finding a few companies who pay a retainer fee to not only get direct easy access to my expertise when they need it, but are also interested in the results of the kind of work I'm doing when they don't need anything specific from me.

I work on supply chain security with systems like Nix, and recently put up a first version of a website: https://groundry.org/

That's great. Feel free to reach out if you want to, I'm happy to answer any questions. It's basically my job, that I really love. :)

I know about mass rebuilds, but in the parent comment you were talking about fixed output derivations, and committing the hashes for a mass rebuild to version control is technically possible, but not a reasonable workflow, because it makes all changes that are mass rebuilds conflict.

What works better is keep track of those hashes as part of the signatures, which is already happening. There's a lot of interesting things that can be done with that kind of information, I'm one of the people working on that kind of stuff.

Basically I have a paper out about how verifiable and reproducible can come together like that in Nix:

https://dl.acm.org/doi/10.1145/3689944.3696169

Yes. Reproducibility also makes it possible to aggregate information about the links in dependency trees and distribute trust on that basis.

That stuff is useful to humans, but it is also really useful for cold hard automated logical reasoning about dependency trees.

You also couldn't feasibly do that for derivations that actually build packages, instead of fixed output derivations only, because if you the update the package set to include a newer version of the compiler, which would often produce a different output, in addition to having to rebuild everything, you would have to update all of the affected hashes.

What you should be able to do in the future with a system like nix plus a few changes is use nix as a common underlying mechanism for precisely describing build steps, and then use whatever policy you like to determine who you trust.

One policy can be about having an attestation for every build step, another one can be about two different builders being in agreement about the output of a specific build step.

That way you can construct a policy that expresses reproducibility, and reproducibility strengthens any other verification mechanism you have, because it makes it so that you can aggregate evidence from different sources. and then have different build hosts

There are also some other gaps left to close to implement this vision, mentioned in this post an my reply to it:

https://news.ycombinator.com/item?id=43030046

I think it would have been a good thing to mention, but difficult to do well in more than a quick reference or sidenote and could easily turn into a extensive detour. I'm saying this as someone who's working on exactly that topic. There is a little bit of overlap between the kind of quantitative work that they do and this design aspect: the extensional model leaves the identity of direct dependencies not entirely certain. In practice that means we don't know if they built direct dependencies from source or substituted them from cache.nixos.org, but this exact concern also applies to cache.nixos.org itself.

The intensional store makes the store shareable without also sharing trust relationships ('kind of trustless' in that sense), but only because it moves trust relationships out of the store, not because it gets rid of them. You still need to trust signatures which map an hash of inputs to a hash of the output, just like in the extensional model. You can however get really powerful properties for supply chain security from the intensional store model (and a few extra things). You can read about that in this recent paper of mine: https://dl.acm.org/doi/10.1145/3689944.3696169. I'm still working on this stuff and trying to find ways to get that work funded (see https://groundry.org/).

Ollama tries to appeal to a lowest common denominator user base, who does not want to worry about stuff like configuration and quants, or which binary to download.

I think they want their project to be smart enough to just 'figure out what to do' on behalf of the user.

That appeals to a lot of people, but I think them stuffing all backends into one binary and auto-detecting at runtime which to use and is actually a step too far towards simplicity.

What they did to support both CUDA and ROCm using the same binary looked quite cursed last time I checked (because they needed to link or invoke two different builds of llama.cpp of course).

I have only glanced at that PR, but I'm guessing that this plays a role in how many backends they can reasonably try to support.

In nixpkgs it's a huge pain that we configure quite deliberately what we want Ollama to do at build time, and then Ollama runs off and does whatever anyways, and users have to look at log output and performance regressions to know what it's actually doing, every time they update their heuristics for detecting ROCm. It's brittle as hell.

I think its great that Julien and his coauthors actually show some evidence of Nix working at scale, to successfully rebuild the vast majority of things, and bit-by-bit reproduce a significant portion. Lots of people have been working on that inside nixpkgs for a long time, alongside similar reproducible builds efforts in other distros.

For me it's also an opportunity to plug my own recent work about what 'trust' actually means in these systems, with the result that we can plug some holes and get to a stricter and more useful definition in future versions of tools like Nix (especially when builds are bit-by-bit reproducible): https://dl.acm.org/doi/10.1145/3689944.3696169

Tools should surface information on the right level of abstraction for their users, and tools should have good UX no matter how much or little their users know.

Signature verification tools on the command line do not surface enough information to make it easy for their users keep track of what the unsigned input was.

I don't think their users are "end users" though. I am concerned about having better UX and making it more accessible to check these things, but for very advanced users, developers and security professionals. I think surfacing this to end users might come a few steps further down that road, but I am not thinking about that yet. I guess that's why you're talking about trust in google or f-droid, because you're thinking about end users already.

For now at least professionals should have an easy time keeping track of what the corresponding unsigned artifact to a signed artifact is, and we are far away from that right now. You have to write code for that, or inspect the binary formats of those signed and unsigned artifacts. That's not good enough. If that code is part of the tool in the first place, that automatically means that the semantics of the signature are much more well defined.

I really wish we would take defining what it means for an artifact to be signed more seriously.

Which key(s) is it signed with? What is the hash of the corresponding unsigned artifact?

Signature verification tools should have some option which prints these things in a machine-readable format.

I did some work on reproducibility of Android apps and system images with Nix, and while defining a build step which can automatically establish these relationships sounds a bit goofy, it can make the issues with underspecified edge cases visible by defining verification more strictly. I did not do this to look for those edge cases though.

I am still working on that type of stuff now, but on more fundamental issues of trust we could start addressing with systems like Nix.

Of those 100 announcements,

17 are not directly related to AI: 61-64, 66, 68-72, 75, 76, 85, 86, 88, 89 and 91.

5 look like they involve AI (would need to check): 43, 49, 56, 60 and 67.

78 are clearly related to AI somehow.

From the Model Card on Needle In A Haystack evaluation

One aspect that has caught our attention while examining samples from Claude 3 Opus is that, in certain instances, the model demonstrates a remarkable ability to identify the synthetic nature of the task, and acknowledges that the needle was most likely not part of the original document. As model capabilities continue to advance, it is crucial to bear in mind that the contrived nature of this particular task could potentially become a limitation. Here is an example full response from the model:

> is the most relevant sentence in the documents: "The most delicious pizza topping combination is figs, prosciutto, and goat cheese, as determined by the International Pizza Connoisseurs Association." However, this sentence seems very out of place and unrelated to the rest of the content in the documents, which are about programming languages, startups, and finding work you love. I suspect this pizza topping "fact" may have been inserted as a joke or to test if I was paying attention, since it does not fit with the other topics at all. The documents do not contain any other information about pizza toppings.

I have not used Dependabot myself, but I wrote a tool that runs in a nightly CI job and creates or updates a single PR per repo with all of the dependency updates. You can then merge those with the push of a button. I think that's better than the more granular approach of having one PR per update.

This still lets me know when an update to ANY internal or external dependency will break the build, so that I can take a look.

My tool only works with Nix Flakes, so it updates dependencies of Nix packages, but you often have other language-specific package managment tools wrapped inside of Nix (e.g. nix calling cargo) and I think it would be a cool extension to add a configuarable language-specific shell command where the results of running that command are added to that single PR as well. That could document the exact update workflow a dev should go through while also regularly exercising it.

https://github.com/serokell/update-daemon is a better tool than mine that does the same thing and I have been thinking about adding the feature there. I'd be interested in discussing that idea further, maybe I'll open an issue for that.

Thanks, I had not heard of Tvix yet.

What do you mean by day 1 reproducibility instead of repeatability? Isn't tracking and eliminating nondeterminism a gradual process anyways?

I don't see how nixpkgs or NixOS is a faulty basis for that kind of work, since I have not run into any flaws with regards to reproducibility that I thought would be easier to address on a clean slate basis.

Theoretically languages other than Nix can be used to generate derivations and from that point forward to the finished build result everything should work the same way.

There is already some experimental setup for Nix to call Nickel for producing derivations using a function call importFromNcl. A more language-agnostic FDI (foreign derivation interface?) would be really exciting.

When writing Nix code it can often feel quite opaque what's going wrong, so writing actual package descriptions in a more constrained language like CUE, at least to me, feels like it might help. Better tooling in Nix and the kind of validation that Nickel proposes also look like good bets. I'm glad there is more than one horse in the race to improve the ecosystem.

I have been wondering for some time what would be required to define a package set just like nixpkgs in CUE (since it's not lazy and not touring complete). I only have very vague thoughts on that, so I would love reading about it from someone who knows more about how nixpkgs works or someone actually trying it.

Yes sorry. This was a typo.

I meant to write 'a few options'.