alexandercheema

lol bro there is already a big OSS project called exo: https://github.com/exo-explore/exo

Isn't Claude Code for Infrastructure just...Claude Code?

Is there one for Kimi K2.5?

Appreciate you checking back so often. We have some exciting plans. Keep checking and it won't be long before something pops up :)

Yes, these models are mostly compute-bound so benefit even more from the compute on the DGX Spark.

Blog author here. Actually, no. The model can be streamed into the DGX Spark, so we can run prefill of models much larger than 128GB (e.g. DeepSeek R1) on the DGX Spark. This feature is coming to EXO 1.0 which will be open-sourced soonTM.

exo maintainer here. tgtweak is correct.

This looks like potentially some promising research that I'm looking into reproducing now. We want to lower the barrier to running large models as much as possible so if this works, it would be a potential addition to the exo offering.

Not yet, should I make an issue for it?

This is fixed now, with these commits: - https://github.com/exo-explore/exo/commit/dbbc7be57fb1871d2b... - https://github.com/exo-explore/exo/commit/ce46f000591d8d59c1...

Please keep the bug reports coming, we're moving fast to get this stable on all platforms.

Do you mean with Apple Intelligence? You can already query models you host from Apple using exo or even just local on-device inference.

Try it out - don't trust me!

The way this works is that each device holds a partition of the model (for now a continuous set of layers). E.g. let's say you have 3 devices and the model is 32 layers. Device 1 could hold layers 1-10, device 2 holds 11-20 and device 3 holds 21-32. Each device executes the layers it's responsible for and passes on the output of its last layer (the activations) to the next device.

The activations are ~8KB for Llama-3-8B and ~32KB for Llama-3-70B (it's linear in the number of parameters in that layer and Llama-3-70B has more layers). Generally the larger the model gets (in terms of parameters), the more layers it ends up having, so we end up with sub-linear scaling so I expect Llama-3-405B to have activations on the order of ~100KB.

This is totally acceptable to send over a local network. The main issue you run into is latency, not bandwidth. Since LLMs are autoregressive (tokens are generated serially), additional latency limits throughput. However, over a local network latency is generally very low (<5ms in my experience). And if not, it's still useful depending on the use-case since you can get a lot of throughput with pipeline parallelism (overlapping requests): https://pytorch.org/docs/stable/pipeline.html

Repo maintainer here. It supports any device tinygrad does, which is a lot. We didn’t expect it to blow up so soon - the repo is still experimental. Internally we’ve mostly been testing on MacBooks and Mac Minis, and that’s where dev is happening. The swift implementation is outdated and currently broken, since Python has been changing so fast (over 20 commits in the last day). On my ToDo is CI/CD pipeline with integration tests for different device and network configurations, so we don’t randomly break stuff for certain devices.

We’re moving fast to get it stable and usable. The goal is for this to be as simple as running Dropbox. Bear with us :)

Thank you for the support! I agree on the cost point, and personally I don’t want to live in a world where all AI runs on H100s in a giant datacenter controlled by one company.

You’re right. The assumption right now is that you’re running on trusted devices on your own local network. I will add a section in the README.

That’s where we want to get eventually. There’s a lot of work that needs to be done but I’m confident we’ll get there. Give us 3 months and it’ll be as simple as running Dropbox.

Could you create a GitHub issue? There's a lot of work we'd like to do to improve this.

Thanks for pointing out that. Fixed https://github.com/exo-explore/exo/blob/main/LICENSE

Fascinating, will check this out! I wanted to focus on Python first to build this quickly, test out ideas and iterate.

This seems like a good option for a switch.

Do you know if any of these can run on Apple/Android devices?

Repo author here. This sounds interesting. Could you elaborate on the benefits of Bumblebee / Axon?

Issue link: https://github.com/exo-explore/exo/issues/11

It *should* but I haven't tried it. I will try it. Updated in this issue:

We could also try raspberry pi + coral usb tpu (https://coral.ai/products/) - that might be a killer combo for super cheap home ai cluster.

"Transformers are memory bandwidth bound" - this is the precise reason why this makes sense. If a model doesn't fit into memory on a single device, it needs to be incrementally loaded into memory (offloading), which is bottlenecked by memory bandwidth. Splitting the model over multiple devices avoids this, instead trading off for latency of communicating between nodes. The network bandwidth requirements are minimal since only the activations (intermediary embeddings) are passed between devices. For Llama-3-8B these are ~10KB, for Llama-3-70B these are ~32KB.

We didn't expect this to blow up so quickly. A lot of work needs to be done on getting different setups working. I have made an issue here: https://github.com/exo-explore/exo/issues/11

Yeah, unfortunately the autoregressive nature of these models slows it down significantly with added device<->device latency. However, you can still max out on throughput with pipeline parallelism, where you overlap execution. See: https://pytorch.org/docs/stable/pipeline.html

Repo author here. That's correct. The embeddings for Llama-3-8B are around 8KB-10KB. For Llama-3-70B they're around 32KB. These are small enough to send around between devices on a local network. For a SETI@home style network, latency will kill you if you go over the internet. That's why we're starting with local networks.

Repo author here. Tinygrad changes rapidly so wouldn't it say it's "more" stable, but it certainly supports more accelerators than llama.cpp. As George Hotz likes to say, it sits somewhere on the spectrum between llama.cpp and Mojo. No hand-written kernels, optimal kernels are generated and found by beam search.