CompuIves

Very cool to see more use of userfaultfd, really powerful API because you can fully control how and from where memory is loaded during a pagefault.

This is really cool to see, reminds me of the early days of CodeSandbox. Though this API looks _fantastic_. I love that you do VM configuration using `with`.

I think this is very similar! Really cool to see.

The first version we launched used the exact same approach (MAP_PRIVATE). Later on, we bypassed the file system by using shared memory and using userfaultfd because ultimately the NVMe became the bottleneck (https://codesandbox.io/blog/cloning-microvms-using-userfault... and https://codesandbox.io/blog/how-we-scale-our-microvm-infrast...).

Yes, that's right. The Firecracker team has written a fantastic doc about this as well: https://github.com/firecracker-microvm/firecracker/blob/main....

It's important to refresh entropy immediately after clone. Still, there can be code that didn't assume it could be cloned (even though there's always been `fork`, of course). Because of this, we don't live clone across workspaces for unlisted/private sandboxes and limit the use case to dev envs where no secrets are stored.

I talk a bit about this here: https://codesandbox.io/blog/cloning-microvms-using-userfault.... Before VM A updates its data, the data is copied over to VM B if VM B hadn't written/read that data yet.

Oh wow! Unexpected and cool to see this post on Hacker News! Since then we have evolved our VM infra a bit, and I've written two more posts about this.

First, we started cloning VMs using userfaultfd, which allows us to bypass the disk and let children read memory directly from parent VMs [1].

And we also moved to saving memory snapshots compressed. To keep VM boots fast, we need to decompress on the fly as VMs read from the snapshot, so we chunk up snapshots in 4kb-8kb pieces that are zstd compressed [2].

Happy to answer any questions here!

[1] https://codesandbox.io/blog/cloning-microvms-using-userfault...

[2] https://codesandbox.io/blog/how-we-scale-our-microvm-infrast...

It's a block for non-US IPs, I think. The page is an application form for joining them, and it only accepts US citizens.

A new post in what is slowly becoming a series! Happy to discuss this or answer any questions here.

Exactly, the result would've been different if the author would not have disabled caching.

In this case it's because the iframes are loaded/unloaded multiple times, but we also spawn web workers where the same worker is spawned multiple times (for transpiling code in multiple threads, for example). In all those cases we rely on caching so we don't have to download the same worker code more than once.

If you want to be efficient in Amsterdam, you take the bike or public transport. That has been faster than cars even before this change, and now more so.

Hey all! I'm one of the co-founders of CodeSandbox. Happy to answer any (technical) questions about this release or CodeSandbox in general!

Yes! But I work on CodeSandbox, so that creates some bias :). We've been working on our own CDE solution, though we've taken a different spin to improve speed and cost.

Our solution is based on Firecracker, which enables us to "pause" (& clone) a VM at any point in time and resume it later exactly where it left of, within 1.5s. This gives the benefit that you won't have to wait for your environment to spin up when you request one, or when you continue working on one after some inactivity.

However, there's another benefit to that: we can now "preload" development environments. Whenever someone opens a pull request (even from local), we create a VM for it in the background. We run the dev server/LSPs/everything you need, and then pause the VM. Now whenever you want to review that pull request, we resume that environment and you can instantly review the code or check the dev server/preview like a deployment preview.

It also reduces cost. We can pause the VM after 5 minutes of inactivity, and when you come back, we'll resume it so it won't feel like the environment was closed at all. In other solutions you either need to keep a server spinning in the background, or increase the "hibernation timeout" to make sure you don't have the cold boot.

It's kind of like your laptop, if you close it you don't expect it to shut down and boot the whole OS again when you open it. I've written more about how we do the pausing/cloning here (https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-s...) and here (https://codesandbox.io/blog/cloning-microvms-using-userfault...).

The editor on CodeSandbox runs web VSCode, and in that version we emulate Node.js in the browser to run local extensions.

Thank you!

custom io_uring based driver for the VM block devices? or what do you mean here?

We're using the async io backend that's shipped with Firecracker for our scratch disks.

At CodeSandbox we use Firecracker for hosting development environments, and I agree with the points. Though I don't think that means you should not use Firecracker for running long-lived workloads.

We reclaim memory with a memory balloon device, for the disk trimming we discard (& compress) the disk, and for i/o speed we use io_uring (which we only use for scratch disks, the project disks are network disks).

It's a tradeoff. It's more work and does require custom implementations. For us that made sense, because in return we get a lightweight VMM that we can more easily extend with functionality like memory snapshotting and live VM cloning [1][2].

[1] https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-s...

[2] https://codesandbox.io/blog/cloning-microvms-using-userfault...

Yes, we use this at CodeSandbox for reclaiming memory to the host (and to reduce snapshot size when we hibernate the VM).

At CodeSandbox (https://codesandbox.io) we're also working on this! Main focus of us is that we're running the environment in Firecracker microVMs, which allows us to snapshot and clone environments very quickly. This enables us to create a VM for every branch, which comes with the added advantage that every branch automatically has a snapshotted preview environment that can resume in ~2 seconds.

Hey all! I'm super excited to share more of the inner workings of CodeSandbox. Happy to talk about the implementation or microVMs/infra in general.

Nice to see the post here! Happy to answer any questions or discuss about different approaches!

At CodeSandbox we use Firecracker to run our VMs (more info here: https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-s...).

To answer the questions:

what version of the kernel do you use (the github page says 5.10 but isn't that quite old?)

Right, they have tested with 5.10, but it also works with higher kernel versions. Our host currently runs 5.19 and we're planning to upgrade to 6.1 soon. The guest runs 5.15.63, we use a config very similar to the recommended config by FC team (it's in the FC repo). It's important to mention that we had to disable async pagefaulting (a KVM feature) with more modern kernel versions, as VMs could get stuck waiting for an PF resolve.

What do you use to build the 'micro' images

We created a CLI that creates a rootfs from a Docker image. It pulls the image, creates a container and then extracts the fs from it to an ext4 disk. For the init, we forked the open sourced init from the Fly team (https://github.com/superfly/init-snapshot) and changed/added some functionality.

How do you keep timesync of you're not using a timesync daemon?

IIRC we expose the time as a PTP device (handled by kvm) and run phc2sys to sync the time in an interval. Firecracker has some documentation on this, where it recommends chrony. It can also be done with vsock, but it would be more manual.

Handle kernel and app logs without adding an log daemon, and same through vsocks, etc?

The init forwards stdout/stderr of the command it runs to its own stdout, which Firecracker then logs out by itself. A supervisor reads these and writes the logs to files.

Hey all! I'm one of the co-founders of CodeSandbox. We just launched Python support on CodeSandbox, which allows you to quickly create Python prototypes or do full development of Python in CodeSandbox (together with our VSCode extension).

If you have any questions, either about the implementation or the product, don't hesitate to leave a message!

No, we've built our own web editor & iOS code editor. In our v1 editor we did run VSCode in the browser, but that was before code-server was released (in 2018). Even if we wanted to run code-server, that would be impossible as we allow for multiple users to open the same sandbox/branch, which wouldn't fit the model of VSCode server (which is single user per server).

Ah right, the main advantage I have here is that the dev server also stays running when I go to another branch (since it's a different VM). So I can work on one thing, share it with the team and in the meantime continue on another branch. They can see the devserver / running code, while I am working on something else.

It's especially useful for things like migrations or dependency management. In one branch I could work on something that has some database migrations, and then I can still switch to other branches without having to roll back the migrations.

I didn't know about git worktrees though! I'm going to read up on this.

Yeah Repl.it is one of the bigger ones.

I'd say that CodeSandbox has a strong focus on extending the existing workflow for developers. That's why we have a VSCode integration, a GitHub integration with a GitHub App that creates a running dev env for every branch/PR, and we make sure that generally all editor features you expect (autocomplete, go to definition, hover info, etc) are available for the languages that we support.

I'm using a web based IDE for my dev, but I'm biased in that sense. The main advantage for me is that I can easily switch branches, as every branch has its own VM. + I can easily share in-progress work. I do use the VSCode integration, because I'm very used to VSCode.

That said, I've also spoken with people who use a Web IDE next to their local environment. E.g. they use a Web IDE for reviewing PRs or making smaller changes, and they use their local editor for feature development.

Yep that works, you can either create a task in the editor for it (using the CLI), or you connect it to a GitHub Repo and install actions.

That said, it is something on our radar, being able to press a button for deploy from CodeSandbox would be valuable for many people!

Hey all! I'm one of the co-founders of CodeSandbox and a big fan of Rust. Rust support is something that we've been working on for a while, but the recent addition of Docker support in CodeSandbox really enabled it. Happy to answer any questions!

The online editor doesn't support rebinding yet, however, we have an integration with VSCode which allows you to open any branch/sandbox inside your local VSCode, with your own extensions and keybindings. That's how I've been doing most of my development with CodeSandbox!

Yes, I should've clarified that the opinionated part is mainly in the workflow.

When you open a repo on CodeSandbox, you automatically open the main branch (e.g. https://codesandbox.io/p/github/codesandbox/test-sandbox/mai..., which is the main branch of our test repo). To contribute to this branch, you can click "Branch" in the editor, and we'll automatically create a new branch, and redirect you to the URL of this branch.

They key here is that you can share this branch (with anyone who has access to the repo). If someone else opens this same branch in CodeSandbox, they will join the same workspace as yours and see your cursor. This allows you to share in-progress work with your team more easily, because you're sharing a link to a branch. When someone else opens that link, they will see a running environment instead of static code, even if you're not online.

So you could essentially say that dev environments on CodeSandbox are multi-user, which is a key difference compared to other cloud environments. With other cloud solutions, the link is tied to your personal environment (so the link is only valid as long as you are online). With CodeSandbox, the link is tied to a branch, which can be shared and opened any time by multiple users.

That's great to hear! We're now working a lot on making CodeSandbox a great experience for development, and we're now at the point that we're doing all our development of CodeSandbox on CodeSandbox as well. It's a quite opinionated experience, because every branch will have a URL that you can share, but it works really well when working together with a team. More info on that is here: https://projects.codesandbox.io/