cbrewster

Now you need to batch your requests to your intermediate server

You can talk to support about missing the notification about your reservation and they'll still let you put in an order.

Fair, but its still a real issue and solved in a similar way: Nix has finer grained reproducibility -- not only at the environment level but also at the derivation level. Being able to pick and choose which dependencies to update while ensuring other packages are left exactly the same is valuable to us.

For context, I'm referencing our (legacy) base image for projects on Replit: Polygott (https://github.com/replit/polygott/).

The image contains dependencies needed for 50+ languages. This means repls by default are packed with lots of commonly used tools. However, the image is massive, takes a long time to build, and is difficult to deploy.

Unfortunately, slimming the image down is not really an option: people rely on all the tools we provide out of the box.

What happens when you need to update some dependency within that image? Now you have to do an image rebuild. If you're lucky only the top-most layers will be rebuilt and the base layers stay the same, if you're unlucky nearly the whole image is rebuilt. Usually we just want to update some subset of dependencies, but during the rebuild some other dependencies may get updated unintentionally (eg if they aren't pinned to a particular version). For most, this may not be an issue but at Replit, everyone's projects use this base Docker image. Unintended updates can cause breakage for our users.

Author here. It's titled this way because Docker seems to come up in a lot of discussions around Nix and is often compared with Nix. Partly because of their overlap in functionality but also because people may not understand the difference between the two. The goal here was to a) highlight the different use-cases between the two tools b) compare the tools in the areas that they overlap and c) show how both tools can be used together.

Author here. In our case, we had a large base Docker image called Polygott (https://github.com/replit/polygott) it pulls in dependencies for 50+ different languages from various repositories. We would pin things where possible, but its still very difficult to ensure reproducible builds.

Additionally, docker builds have free access to the network to do anything it would like. Nix goes to great lengths to sandbox builds and limit network access. Anything accessed from a network requires a pinned sha 256 hash to ensure the remote data hasn't changed. (https://nixos.wiki/wiki/Nix#Sandboxing)

Author here. As with most things, its all about the trade-offs. Docker has certainly proved itself and that approach has worked on a massive scale. However, its not a silver bullet. For us at Replit, our Docker approach was causing issues: our base image was large and unmaintainable and we had almost no way of knowing what changed between subsequent builds of the base image.

We've been able to utilize Nix to address both of those issues, and others who may be in a similar scenario might also find Nix to be valuable.

Of course Nix comes with its own set of opinions and complexities but it has been a worthwhile trade-off for us.

While you can copy files from different stages, I wouldn't consider this to be the same thing as composing two base images together. Like the example in the post, you can't take the Rust and NodeJS images and tell Docker to magically merge them. You can copy binaries & libraries from one to the other but that seems extremely tedious and error prone.

Whereas Nix makes it rather trivial to compose together the packages you need (eg Rust, NodeJS) in your environment.

It was more of a silly demonstration showing what's possible and less about being practical. :)

Yeah, we are really excited about the future Nix will bring for us at Replit. Soon, language version incompatibility will be a thing of the past on Replit.

Yup, you can setup Haskell with Nix. I'm not too familiar with Haskell, but it looks like Nix even has a collection of Haskell packages (including lens). This looks like it is a good resource for getting started with Nix + Haskell https://notes.srid.ca/haskell-nix, you should be able to follow along in a Nix repl. I recommend forking on of the example repls from the blog post.

Sorry, building nix environments from scratch is still rough around the edges and we are working to improve that at the moment.

When hosting a web server, your app must listen on 0.0.0.0, adding that to your repl seems to make that work. I will make sure that is in our docs for web hosting.

Working Racket web app: https://replit.com/@ConnorBrewster/racket-server

This is coming soon!

The config files should be visible again, sorry for the delay!

Here's a template for Racket: https://replit.com/@ConnorBrewster/racket

Here's a template for nasm: https://replit.com/@ConnorBrewster/nasm

If they are in Nixpkgs they will likely work in a repl. Nixpkgs search: https://search.nixos.org/packages

Piet: https://search.nixos.org/packages?channel=unstable&show=hask...

Hi, one of the authors here. The Nix environment on Replit is under development right now, we hid the configuration files to clean up the filetree as most users shouldn't need to know about the config files unless they are customizing the repl environment.

However, we are reverting that change for now since we don't offer a good way to access those files yet. I'll reply here once they are visible again.

Neat, if it gets added to nixpkgs it will be available on Replit!

https://github.com/NixOS/nixpkgs

We don't have templates for every language, but if it's available in Nix, you can set this up in a repl. https://replit.com/@ConnorBrewster/fortran

We will be expanding our collection of templates over time, but Nix allows us to make language setup be a user-space concern instead of us internally maintaining a collection of languages configurations and docker images.

Nice catch! I'll get that fixed up. Thanks :)

Good question! One of the motivations for splitting our infrastructure into clusters was to separate paying and non-paying customer. There's various reason for this, reducing the size of our failure domains and running better hardware for paying customers. However, this means users can be transitioned between clusters as they become a paying customer or back to a non-paying customer. Since repl.co subdomains are tied to the user/repl and not to the particular back-end cluster, we'd have to maintain way too many records with our existing DNS provider. Even if we could, we'd need to make sure the records are always synchronized. In the end, it made more since to roll our own authority server that can just query our data store and generate the right DNS answers on the fly.

I'm not sure what you're referring to. You can use any web server library/framework you'd like, we open the web view whenever your program starts listening on a port. We can't open the web view any earlier since there would be nothing to load in the web view.

Hey anurag, big fan of Render! We currently don't have a great solution for this and we recommend that users use a DNS provider that has support for something like an ALIAS record.

Hi, author here! tptacek is right on the money. The authority server is really small and simple, and writing it in Go meant we had access to our existing internal packages that had the logic to fetch the data we need for each DNS query. This seemed like the most straight-forward path.

We do have an internal retry mechanism for certain kinds of errors, but after too many tries it will return an error to the client. Our clients already have to include a robust retry mechanism because we have to deal any sort of network instability.

Indeed, for our use-case we want an insta-kill so we can free up our global lock on the container. But the docker daemon will still get notified of the container death and run any sort of cleanup. This just gives us an opportunity to free up the global lock before getting stuck waiting on the cleanup.

Have you opened a bug report in Docker upstream to see if they can improve the situation (perhaps by putting networking cleanup jobs to a background goroutine)?

I have not yet, but I plan to! We figured we'd work around it ourselves instead of waiting for a potential fix upstream.

Hi! Author of the post here.

One of the most important invariants that we have to maintain is that there is only 1 container running per repl at any given time. We could determine if the machine is shutting down and not proxy the connection, but we wouldn't have a place to proxy it to since we can't be sure that the existing container has finished shutting down. So either way we end up returning an error and the client has to wait until the old container has been destroyed so a new one can be spawned.

Currently it will only be enabled on Windows machines with Nvidia GPUs. So this shouldn't be an issue for environments with no access to a GPU.