ekzhang

I don’t think this is straightforward but it may be a skill issue on my part. It would require dockerizing headless Chrome with WebGPU support and dynamically injecting custom bundled JavaScript into the page, then extracting the results with Chrome IPC

Yes, we are actively working on it! The goal is to be a full ML research library, not just a model inference runtime. You can join the Discord to follow along

Hm okay, seems like an interesting set of benchmarks — let me know if there’s anything I can do to help make jax-js more compatible with your docker setup

Firefox doesn’t support WebGPU yet, you can run programs in the REPL through other backends like Wasm/WebGL: https://jax-js.com/repl

See: https://caniuse.com/webgpu

Since ONNX is just a model data format, you can actually parse and run ONNX files in jax-js as well. Here’s an example of running DETR ResNet-50 from Xenova’s transformers.js checkpoint in jax-js

https://jax-js.com/detr-resnet-50

I don’t think I intend to support everything in ONNX right now, especially quant/dequant, but eventually it would be interesting to see if we can help accelerate transformers.js with a jax-js backend + goodies like kernel fusion

jax-js is more trying to explore being an ML research library, rather than ONNX which is a runtime for exported models

I don’t think tf.tidy() is a sound API under jvp/grad transformations, also it prevents you from using async which makes it incompatible with GPU backends (or blocks the page), a pretty big issue. https://github.com/tensorflow/tfjs/issues/5468

Thanks for the feedback though, just explaining how we arrived at this API. I hope you’d at least try it out — hopefully you will see when developing that the refs are more flexible than alternatives.

For sure! It looks like this is benchmarking the autodiff cpu time, not the actual kernels though, which (correct me if I’m wrong) isn’t really relevant for an ML library — it’s more for if you have a really complex scientific expression

Wanted to share this library that I'm open-sourcing now after working on it for the past year.

You write code like in JAX/NumPy, but it’s fully interactive on the frontend and compiles down to shaders on the user’s GPU (with WebGPU). So far I’ve used it for purely frontend-only ML demos. https://jax-js.com/mobileclip

Thanks! I think that's a great idea. Containers are nice, but there's nothing quite like a "real" VM system in terms of learning + developing, and they can be very cheap if you're OK with the performance hit from nested virtualization :)

Another idea I had is to customize the rootfs. It boots Alpine right now, but should be trivial to change it to anything else (and it only takes ~5 seconds to build!).

Hello! Thrilled to be sharing something I've worked on for the past 8 months at Modal — a cloud, GPU notebooks product.

We wanted to improve the Jupyter Notebook experience while offering high-performance, cloud compute. So we sought out to fix some of the problems in cloud offerings for notebooks, and we ended up rebuilding a lot of the interface ourselves. This means you can do things like:

- Within the same notebook, swap from CPU to GPU, and the kernel will restart in seconds while keeping your work.

- Use _lots_ of hardware: up to 8x H100/B200 GPUs, not just one.

- Automatic idle shutdown (configurable timeout) so you don't get a huge bill / forget to stop your notebook.

- Real-time collaborative editing.

- Integrations with Pyright and Ruff for a better IDE-like experience.

Unlike Google Colab, we are a full cloud offering, so you can integrate with the rest of the Modal platform — use custom images, attach persistent volumes and so on. We've heard these are the actual, boring but really important features you need to do experimental work in notebooks.

Happy to hear any feedback! Would love for you to try it out.

Hello - I wanted to share a bit about an open-source project I’ve been working on, building a fundamental library for ML and numerical computing in the browser.

There are different tradeoffs here, but I’d like to replicate the speed and ergonomics of Python libraries with NumPy/PyTorch/JAX without the difficulty of distribution. jax-js is lightweight enough to be embedded in a website, but optimized to take advantage of tech like Wasm and WebGPU.

Let me know if you have any feedback on this compiler design. It’s working well and faster than tfjs at matrix multiplication, while having kernel fusion as well.

Just want to mention that these standalone python builds have been super important for simplifying a lot of dev tooling, even if users may not see it directly — I work for a cloud infrastructure company, and they’re what allow us to give users a one-line way of adding Python to any Docker image they desire. It’s helpful to have reproducible, standalone Python builds of this quality (and release transparency). Thanks y’all for taking this on.

Sorry, just realized the misunderstanding. To clarify Modal still uses the kernel WireGuard module. The userspace part that’s in Go and not in other languages that we use is the wgctrl library.

This is it. I like Rust a lot, but you gotta pick the right tool for the job sometimes

Right, vprox servers act as multiplexed NAT instances with a VPN attached. You do still need the VPN part though since our containers run around the world, in multiple regions and availability zones. Setting the gateway to a machine running fck-nat would only work if that machine is in the same subnet (e.g., for AWS, in one availability zone).

The other features that were hard requirements for us were multi-tenancy and high availability / failover.

By the way, fck-nat is just a basic shell script that sets the `ip_forward` and `rp_filter` sysctls and adds an IP masquerade rule. If you look at vprox, we also do this but build a lot on top of it. https://github.com/modal-labs/vprox

We have a native OIDC integration at Modal, as well! Every container gets a token. https://modal.com/docs/guide/oidc-integration

Yeah, you hit the nail on the head. We considered NAT64 as well and looked at some implementations including eBPF-based ones like Cilium.

Glad to know that IPv6-only is working well for you. "In a perfect world…" :)

Yep! This is something we have internal tests for haha, you have good instincts that it can be tricky. Here's an example of using that for multi-GPU training https://modal.com/docs/examples/llm-finetuning

Neat, thanks for sharing! Glad to know we're in good company here.

Thanks. We did check out Tailscale, but they didn't quite have what we were looking for: some high-availability custom component that plugs into a low-level container runtime. (Which makes sense, it's pretty different from their intended use case.)

Modal is actually a happy customer of Tailscale (but for other purposes). :D

We don't use Kubernetes to run user workloads, we do use gVisor. We don't use MIG (multi-instance GPU) or MPS. If you run a container on Modal using N GPUs, you get the entire N GPUs.

If you'd like to learn more, you can check out our docs here: https://modal.com/docs/guide/gpu

Re not using Kubernetes, we have our own custom container runtime in Rust with optimizations like lazy loading of content-addressed file systems. https://www.youtube.com/watch?v=SlkEW4C2kd4

Yeah, great question. This came up at the beginning of design. A lot of our customers specifically needed IPv4 whitelisting. For example, MongoDB Atlas (a very popular database vendor) only supports IPv4. https://www.mongodb.com/community/forums/t/does-mongodb-atla...

The architecture of vprox is pretty generic though and could support IPv6 as well.

I mentioned this in another comment thread, but we use gVisor to enforce isolation. https://gvisor.dev/users/

It's also used by Google Kubernetes Engine, OpenAI, and Cloudflare among others to run untrusted code.

That makes sense, mTLS is great. Some services like Google Cloud SQL are really good about support for it. https://cloud.google.com/sql/docs/mysql/configure-ssl-instan...

It's not quite a zero-trust solution though due to the CA chain of trust.

mTLS is security at a different layer though than IP source whitelisting. I'd say that a lot of companies we spoke to would want both as a defense-in-depth measure. Even with mTLS, network whitelisting is relevant. If your certificate were to be exposed for instance, an attacker would still need to be able to forge a source IP address to start a connection.

What would you suggest as an alternative?

Thanks for sharing. I'm interested in seeing what a global control plane might look like, seems like authentication might be tricky to get right!

Controlling our worker environment (like `net.ipv4.conf.all.rp_filter` sysctl) is a big help for us since it means we don't have to deal with the fullness of all possible network configurations.

We use gVisor! It's an open-source application security sandbox spun off from Google. We work with the gVisor team to get the features we need (notably GPUs / CUDA support) and also help test gVisor upstream https://gvisor.dev/users/

It's also used by Google Kubernetes Engine, OpenAI, and Cloudflare among others to run untrusted code.

Hi! This is a blog post sharing some low-level Linux networking we're doing at Modal with WireGuard.

As a serverless platform we hit a bit of a tricky tradeoff: we run multi-tenant user workloads on machines around the world, and each serverless function is an autoscaling container pool. How do you let users give their functions static IPs, but also decouple them from compute resource flexibility?

We needed a high-availability VPN proxy for containers and didn't find one, so we built our own on top of WireGuard and open-sourced it at https://github.com/modal-labs/vprox

Let us know if you have thoughts! I'm relatively new to low-level container networking, and we (me + my coworkers Luis and Jeffrey + others) have enjoyed working on this.

This is really neat! It looks like you implemented the parser and encoder entirely in Python, how does that compare to an approach where you load an extension module in C or Rust? (Possibly with the actual V8 source)

(Author) Modal tackles how to make FaaS work, but for actual _function calls_, and also with containers that have much higher resource caps (see article: 3 CPUs vs 64 CPUs, or 10 GB RAM vs 336 GB RAM).

EC2 isn't the same compute shape. We run fast-booting (think: seconds, not minutes), dynamic sandboxed containers on a single host (think: gVisor, Firecracker) and optimized file system lookups (FUSE, distributed caching, readahead, profiling). It also means we bill by the CPU cycle, scale rapidly, and bill you only for 100% utilization. You do not manage individual VMs.

This is why scaling the limits of functions-as-a-service is quite different from scaling VMs, and that's what the content of the article focuses on.