Tvix: We Are Rewriting Nix

by robto • 5 years ago

144 62 5 years ago

I don't know enough about Nix internals to know how much of a problem this is, or the right way to improve it.

However, it's good to see an ecosystem forming around the derivation data type, with the possibility of reuse across projects that use derivations.

Good stuff!

I'd be interested to hear a perspective from the Nix and Guix authors.

pxc5• 5 years ago

it's good to see an ecosystem forming around the derivation data type, with the possibility of reuse across projects that use derivations.

Guix used to use the same derivation format as Nix, but they've since diverged

thumbellina2• 5 years ago

TIL. Do you have any links that go into more detail about these differences?

johnisgood2• 5 years ago

Does Guix have any performance issues?

d3nj4l• 5 years ago

OT for nix, but that's an interesting approach to building an organisation on the internet. They seem to have a good sense of cohesion despite being just a group of people with common interests who hang out on an IRC and are distributed across the world?

southerntofu9• 5 years ago

They hang around on hackint, the hacking/hackerspace IRC network. So it's very likely they have some AFK connections as well, and also some common political motivations driving them.

darksaints• 5 years ago

Is tvix/nix pretty tightly coupled to the Linux kernel? I know there is some effort to use the nix package manager on OSX, but I'm not sure if that is a hack or an actual full port to a different kernel. Could it be used on, say, OpenBSD? SeL4?

tazjin9• 5 years ago

For the core parts of Tvix we won't have any OS-specific dependencies, but each platform will need a working builder and sandbox implementation. I suspect that OpenBSD has at least one OCI-compatible sandboxing/containerisation method so it should end up running fine on there.

akavel2• 5 years ago

Nix itself is in first approximation "just a package manager", so not particularly tied to a kernel AFAIU. As to NixOS, this one's a Linux distro, so it currently mostly assumes Linux kernel AFAIK. I tried exploring plugging in L4Linux in that place, but failed due to not being able to understand related areas of NixOS/Nixpkgs well enough. Guix OS on the other hand has some (experimental?) support for GNU Mach kernel I believe.

pxc5• 5 years ago

Guix System (the full OS distribution) supports both Linux and GNU Hurd as kernels

chriswarbo2• 5 years ago

Nix itself is in first approximation "just a package manager"

I'd go even further: Nix is just a build tool, like Make.

In fact, it assumes far less about the system than Make; e.g. Make does all sorts of shell-related things, whilst Nix simply executes a binary (with a list of arguments and a set of env vars).

(Nixpkgs definitions almost always use a bash binary as their executable, so it feels more like Make; but Nix doesn't care)

ReleaseCandidat• 5 years ago

Linux and OS X are officially supported

https://nixos.org/manual/nix/stable/installation/supported-p...

Valodim• 5 years ago

So this sounds like it is mostly concerned with the derivation part. Is it going to be based on Nickel then, or a new custom parser? Would be interested in the rationale there

tazjin9• 5 years ago

No, we are writing an implementation of the Nix language. Nixpkgs is written in Nix and we believe that's the most important part of the ecosystem, so Tvix will support it.

sidkshatriya• 5 years ago

There are noble goals to this project to rewriting Nix to improve performance. The reasons feel valid. However I wonder about certain issues:

- Nix itself has problems of compatibility with Nix! Let me explain...recently NixOS 21.11 released. It still comes packaged with Nix 2.3.16 instead of Nix 2.4 because of incompatibility issues. When the Nix project itself is not able to deliver backwards compatibility I feel uncertain that Tvix might be able to deliver their promise of full compatibility with all the deep architectural changes it plans. Nix does not have a formal specification: Nix is what its C/C++ implementation says. Nix also has deep architectural changes planned going ahead. So Tvix will need to re-architect Nix keeping in mind (and keeping up) with the future changes that are coming. It's going to be tough to build this "alternative" implementation. Users are going to run into weird compatibility issues, not have the patience to resolve them and then just go back to the original Nix implementation.

- A rewrite to a popular project will find it difficult to succeed. There is simply too much momentum in the parent Nix implementation (I'm not talking about the package definitions here) that very few people are likely to adopt this rewrite. This is because most people might not really need the speedup. However, it's possible that the pain around slow evaluation runs so deep that other companies might adopt this too. I don't know enough here.

- The rewrite needs to offer more reason to shift. Right now this rewrite seems to be mainly about performance. There is also a proposal to be able to integrate with GNU Guile. To me this is just avoiding one untyped language (Nix) to replace with another slightly better untyped language (Guile Scheme) (GNU Guix uses Guile also AFAIK). What about introducing typing? https://github.com/tweag/nickel is an interesting project and it would be great to have a system like Nix with optional typing to make writing the derivations more robust. What about rewriting in a safe programming language (e.g. Rust or a GC-ed language)? These things would be likely to excite would-be-contributors and would make your rewrite viable. In other words, the rewrite needs to offer more than just the promise of performance.

Perhaps I have not understood the project goals fully. Maybe the project is all about just being able to leverage the Nix package definitions but have a totally different implementation to instantiate them? But the issue here is that nix-the-language is very intimately tied up with the nix-the-platform and the rewritten implementation will still need to really be extremely compatible to capture all the implicit assumptions in nixpkgs.

Once again, improving nix through a rewrite is a noble goal. It's a tough problem and it would be awesome if the project succeeds!

I have obviously not thought about this as much as you guys and would love to learn how you wish to tackle some of these problems. Maybe some of these issues are not as big as them seem??

chme• 5 years ago

I would also like it if the language itself can be decoupled from the build and package manager system, so that it can be embedded in other software that would benefit from a language like nix. I think that might be a worthy goal and could attract people outside of the nix sphere.

axelf4• 5 years ago

Have you looked at Dhall? I do not think Nix the language would ever be preferable when compared to already existing configuration languages.

tazjin9• 5 years ago

Yep, we'd like that, too! Our current thinking around the evaluator design incorporates the idea of evaluation with/without a Nix store, where using the language itself (e.g. to generate JSON structures or whatever) should be perfectly possible without a store.

southerntofu9• 5 years ago

I would also like it if the language itself can be decoupled from the build and package manager system

I would also like that, but for the exact opposing reason. I love the concepts of nix/guix but i can't stand the dynamic languages with cryptic error messages.

tazjin9• 5 years ago

Nix does not have a formal specification: Nix is what its C/C++ implementation says.

Yep, and this causes a whole lot of issues. In tandem with evaluator development we are writing an initial language specification which will represent the language as it is used in nixpkgs. Our hope is that we can, in collaboration with the Nix developers, eventually find a fixed point between the two implementations here.

It still comes packaged with Nix 2.3.16 instead of Nix 2.4 because of incompatibility

Yep, we're very aware - in fact we had some debate on IRC yesterday about whether or not to touch on this in the blog post, but felt that for the majority of people it's a bit too deep into "current Nix politics" (for lack of a better term).

This is why we have explicitly committed to nixpkgs support - many of the experimental features in Nix (some of which are causes for the new incompatibilities) are not actually used in nixpkgs, and they might have a long road to go before getting there. We believe that this gives us enough time to get to a synchronisation point.

However, it's possible that the pain around slow evaluation runs so deep that other companies might adopt this too. I don't know enough here.

We'll see, on the corporate side of TVL (https://tvl.su) we've had some discussions with a variety of Nix-using companies and there are strong signals that some of our plans would be a big reason for them to switch (or possibly even just partially switch). We want to not worry about this too much this early, to be honest.

The rewrite needs to offer more reason to shift

Yes, there are more things that will become relevant but felt like they were outside of the scope of this initial announcement. For example:

* Implementation will be in memory-safe languages (evaluator in Rust; we will see about things like the sandbox & builder - Go might fit that reasonably well). This is a big point for companies that are worried about running current Nix (which is fairly easy to segfault) on production machines as root.

* We do intend to have more language-related tooling (including various kinds of static analysis, like dead-code analysis or maybe even going as far as an Erlang-style success typing approach)

* Streaming evaluation: We want to yield derivations while evaluation is still ongoing, essentially turning a whole build process into a sort of graph reduction. This has a huge impact on our ability to implement better Nix-native CI, which is very relevant for TVL's ambitions around better monorepo tooling.

It is also worth noting that a bunch of TVL members are nixpkgs committers, and as we proceed we would like to clean up bits of nixpkgs that use features which only exist "by accident". (An example of such a feature, which fortunately is not used in nixpkgs now, is __findFile: https://cl.tvl.fyi/c/depot/+/1325).

I appreciate that it's kinda difficult to serialise all of our thinking here in a handful of comments and a blog post, so we might not get the full point across. That's okay for now. If you're interested in following along more closely and discussing some bits of this, you're welcome to join our IRC channel :)

otabdeveloper42• 5 years ago

This is a big point for companies that are worried about running current Nix (which is fairly easy to segfault) on production machines as root.

Oh wow, wait till they find out what language the OS on their production machine is written in!

kohlerm• 5 years ago

With regards to the language. Is turing completeness really needed? google decided for Bazel that it is not desired to use turing complete language (Starklark). I guess the Nix language is turing complete and therefore for backward compatibility a Turing complete language would be needed.

throwaway9843932• 5 years ago

Every distribution has the exact same problems that are impossible to prevent. It's all about the inherent limitations of how software is developed and run in operating systems today. All this stuff about 'solving dependency hell' and dependency paths tied up with merkle trees doesn't change the fact that you can't use two completely different libraries in one application at the same time, nor shell out to two incompatible versions of a binary from the same application. It is only possible if the software itself is designed to do that, using the features of a dynamic linker that allow linking in multiple versions of the same function and specifying which version you're calling from what source function. We don't actually need a new package manager or distribution at all, because they can't solve one of the most fundamental problems - only developing software differently can do that.

Sorry, but Nix is a total waste of time.

Ericson23142• 5 years ago

You should spend more time in the Nix community, because actually people are very aware of these limitations. For C packages, yes the best we can easily do is ensure each binary has coherent library choices, but allow mixing libraries in different binaries. That is still useful, though!

That said, you underestimate the power of being able to hack on any dependency -- for real purpose of Nix is to defeat Conway's law, allowing any person to work on any codebase. It would be quite easy to fork the linker/loader to try out some experiment, or even just do some per-package symbol mangling, for example. I wouldn't want to try that experiment with any other distro, because it would be too hard to make my modification and then put everything together again!

throwaway9843932• 5 years ago

Do you find you need to make experimental changes to the linker often?

Back when I was a Gentoo user, I too bought the story that being able to re-emerge my whole system was some kind of performance or configuration boon. Eventually I realized that I just wanted to use my computer to get through life, that literally any operating system would work, and that any experimental hacks could simply be done when they were needed rather than trying to manage an over-complicated, poorly-supported hobby OS.

I use Ubuntu now. It sucks. But I spend my time on doing real world things now, rather than constantly tweaking my computer to eventually do a real world thing.

For work I use containers with any distribution at all. They all have the same result in a container.

ncmncm2• 5 years ago

What would seem most useful would be a compile-time-typed language (perhaps with some inference propagation, for convenience) that existing Nix code could be translated into, maybe even while emulating execution to discover the actual types to annotate with. You have the advantage that all existing production Nix code is known to terminate in short order.

Then, you could generate a new version of the entire 30,000-strong package collection automatically, with the new language swapped in in place of the Nix code, which could then be abandoned.

choeger2• 5 years ago

that existing Nix code could be translated into,

It is highly unlikely that this is feasible. I don't know nix in detail but I infer from your post that it is an untyped language. Untyped languages can, in general, not be translated to (statically decidable) typed languages.

which could then be abandoned.

It could not, unless you convince a majority of the contributors to the package repo to switch to your new language.

southerntofu9• 5 years ago

It is highly unlikely that this is feasible

Some more complex nix features would not be easy to transpile automatically, but many (i don't have stats sorry) packages are very simple declarations which could be ported automatically.

For example, all language-specific libraries imported from rust/python/etc could very easily be automatically ported, and in my experience, writing declarations for those many dependencies is what takes most time in making a small package (although admittedly i only tried once).

funcDropShadow• 5 years ago

Nix, is a dynamically typed language, i.e. the language gives every value at runtime a type, but there a no compile-time (i.e. static) checks whether any function or operator calls will match at runtime to their parameters. Programs can be translated from dynamically typed-languages to statically typed languages. Either by infering types at translation time or by exhaustively adding lots of cases to every function or operator call depending on the runtime. The first approach can in principle only translate a subset of all programs, the latter does not give you any additional guarantees, because you have encoded an interpreter for the dynamically-type language into the statically typed language. Whether the first approach could be used on a large codebase like nixpkgs is highly questionable, the second approach would not yield any advantages here.

tazjin9• 5 years ago

You have the advantage that all existing production Nix code is known to terminate in short order

You don't really know this - there are many code paths in nixpkgs that are never touched during Hydra evaluation (various overrides, user-facing flags, packages marked broken and so on). We think it might be possible to slowly move towards more static analysis of the existing Nix language, but this is a little bit further down the road.

(Note by the way that we have a "runtime type checker" for Nix, but it is mostly an experiment: https://code.tvl.fyi/about/nix/yants)

ncmncm2• 5 years ago

It has all terminated in short order whenever and wherever executed, except possibly dodgy packages no one tries to install.

chriswarbo2• 5 years ago

As sibling comments mention, this would struggle with edge-cases, and require too much buy-in from packagers.

A less ambitious approach would be creating/extending a static analyser/linter for Nix (there are already a few out there). This could nudge packagers towards certain styles, which language implementations could optimise for.

For example, an interpreter could have multiple implementations of the attrset interface (key/value mappings) with different performance characteristics, using some default for literals like '{ foo = "bar"; }' and a specialised version for known use-cases like nixpkgs.lib.nameValuePair ( https://github.com/NixOS/nixpkgs/blob/e9e53499b26ad98ac97f97... ) (mentioned in a sibling)

Linters could then spot when such functions are appropriate, e.g.

    Found literal name/value pair on line 123:
      { name = "x"; value = "y"; }
    This can be slow, consider using:
      nixpkgs.lib.nameValuePair "x" "y"'

kripke3• 5 years ago

I don't know much about Nix, although it has been on my radar for some time. The claim that the language performance is bad pops up time and again, and I'm sure many people would be happy to have a faster alternative --- so props to them for trying to address this clearly important problem.

That being said, in perusing related issues on GitHub such as [1], I have found no serious attempts to even profile the damn thing. The typical attempt seems to be throwing a random idea at the evaluator and crossing fingers. The example of a "performance experiment" [2] from the article is self-described as follows:

Add an alternative impl of the now-abstract Bindings base class that is backed by a std::vector, somewhat similar but stylistically a little superior to the array-backed implementation in upstream nix.

I fail to see how that is a performance experiment when it does not mention performance at all.

The post also mentions issues with the general Nix design and a desire to modernize it, which I am unqualified to comment on, but seems a worthwhile endeavor in and of itself. I do hope that they won't end up writing a new evaluator for the Nix language without a proper understanding of what makes the current one slow and a clear plan not to repeat its mistakes, or the new one risks being just as slow as the old one.

[1] https://github.com/NixOS/nix/issues/2652

[2] https://cl.tvl.fyi/c/depot/+/1123/

tazjin9• 5 years ago

The background on our side is that we did do quite a lot of profiling and investigation of the current evaluator - last year, informally, over video chats and IRC. We have a bunch of logs and various perf measurements spread out over some machines but right now the knowledge lives in our heads, mostly because at the time we did not expect this to spawn a serious project.

Once we get further along with the evaluator we will be incorporating some of that material into more specific blog posts, but it seemed a little too deep for the initial announcement.

Regarding that CL linked from the post: It was part of laying the groundwork for trying to have the same Nix language type backed by different implementations; the reason for this is that Nix's attribute sets (similar to dictionaries or k/v maps in other languages) have a wide variety of use-cases with wildly conflicting performance characteristics.

For example, one very common use-case is the data structure representing the top-level of nixpkgs: An enormous attribute set with many thousands of keys that is frequently modified & copied. An even more common one is related to the builtin function `listToAttrs` which deals with two-key attribute sets (name, value pairs). In current Nix, these are some of the biggest evaluator hotspots (apart from higher-level issues like cache-unfriendliness due to the tree-walking evaluation style) and optimising for one case vastly diminishes performance of the other.

The problem we ran into with all experiments of this form is that there is no clear interface separation in `libexpr` (the part of the Nix codebase that implements the evaluator): Everything is modifying and reading internal state of everything else, including in other parts of Nix that depend on libexpr. It made it basically impossible to really try anything without refactoring huge amounts of (memory-unsafe) code.

Now it's important to mention that the reason for this is understandable: Nix is an old project (18 years now I believe?) and I don't think anyone expected it to get to where it is now, so obviously different decisions were made in its implementation than would be made if it were started today knowing everything we know.

kripke3• 5 years ago

The background on our side is that we did do quite a lot of profiling and investigation of the current evaluator - last year, informally, over video chats and IRC. We have a bunch of logs and various perf measurements spread out over some machines but right now the knowledge lives in our heads, mostly because at the time we did not expect this to spawn a serious project.

It's good to hear that you did do some profiling, even if the results are not public! Of course I would not expect the details in the initial announcement, but I would have expected a mention of this work. At least I have a much higher confidence in the success of this project now than just after reading the announcement.

Switching between the proper implementation of high-level concepts is a staple of untyped (and sometimes typed) runtimes, so in this context the changeset makes more sense than simply "replacing an array-backed implementation with an std::vector-backed implementation" as described, thanks for the explanation!

I'm also not putting any blame on Nix --- the fact that the performance of the evaluator is such an issue is in a way a testament to its success. Unfortunately large, long-lived systems do require active care to enforce proper boundaries between subsystems, and it's a shame if it has got to a point where it's no longer possible to reinstate that separation.

di4na2• 5 years ago

Hey Tazjin, i see you are around that one.

Quick question. Can you expand on IFD and integrations? That is one thing i would love to know more about your plans.

Also i hope it works, the nox code base is imho the biggest impediment to the nix model being more used.

Small advise: please consider making it really easy to contribute to through wll the tooling. This is one of the reason Nix codebase rotted

southerntofu9• 5 years ago

please consider making it really easy to contribute

Yes, that's one area where Guix shines in comparison to nix. They use an established language with its standard library (Guile lisp) making it very easy for people to contribute without learning arcane incantations.

I would love to see nix/Guix implemented in a statically-typed, high-level language like Rust.

di4na2• 5 years ago

I mean nix is in Cpp so uh... technically...

MadeThisToReply• 5 years ago

How do you pronounce "Tvix"?

tazjin9• 5 years ago

Твикс - does that help?)

It seems that North Americans struggle with the word, but most Europeans (including Brits) don't. Don't know enough about linguistics to guess why!

Try saying "Twix" the way you'd imagine a German or Scandinavian accent to sound.

ogogmad• 5 years ago

Твикс - does that help?

Yes, definitely!!

I think it wasn't clear whether you were supposed to say "tee-vix" or /tvɪks/.

For non-Russians, try and figure out which of the above two is right. No one help them, because you'll be taking away a learning opportunity. https://www.youtube.com/watch?v=9hNphbAm_Hs

kungito• 5 years ago

I really wish there was a statically typed alternative to nix. Nix docs are horrible and incomplete and the language is called the same as the rest of tye ecosystem so it's hard to google. I don't think there can be a good declarative functional language without static types because how the hell do you debug? I'ts been really hard whenever I have to do something in a bigger nix environment. I get the same feeling like when I jump into a bigger Python project which isn't documented and I have to read around to see what is what, only goto definition doesn't work, the language is weird and everything is some specific function which I don't know is built in or where it's from and googling gives no answers except maybe link to source code

southerntofu9• 5 years ago

I strongly agree with your sentiment. Both nix and guix have terrible error messages and searching the web for answers if doomed to failure. Although i must point out both have very friendly communities and asking for help in chat is likely to yield constructive answers.

As guix has shown, nix principles can be applied as a DSL/library within your favorite language of choice. I would be curious to see what a "rustix" would look like!

aseipp• 5 years ago

For error messages I think you're ultimately a bit cornered. Even typed languages have serious problems with straightforward and reliable errors, everything else aside. The ones that do tend to have to make careful tradeoffs in the system to achieve that (HKDs probably just straight up make error messages worse, for instance.) That said I think having types would be useful for helping navigate code, but I think you'd need a fairly advanced type system to capture all the "open ended" parts of Nixpkgs, and that's where all the value really is, the tons of little ways people have built up to use it over time. So it's a mixed bag. I sympathize though; I've spent a lot of time on spurious Nix bullshit, even if I still love the tools.

As a contributor I actually think that the success of projects like TypeScript have shown that there should be a way to build a type checker for something like Nix, with incremental migration. But it probably isn't going to be the ideal solution we could all imagine and it will be very very difficult nonetheless. But it could work with the code we have now, and that's a very important bar to clear (I don't take some of the other suggestions like "auto translate 30,000 packages to some other better language that doesn't exist & isn't the one people are using now" all too seriously, I'm afraid.)

I do think having a store implementation (even a "virtual one") decoupled from the evaluator and builder tooling is a step in the right direction for experimentation and hope that it works out.

Tvix: We Are Rewriting Nix

Related Stories