Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

https://github.com/b4rtaz/distributed-llama/discussions/255

by b4rtazz • 11 months ago

347 135 11 months ago

geerlingguy2• 11 months ago

distributed-llama is great, I just wish it would work with more models. I've been happy with ease of setup and its ongoing maintenance compared to Exo, and performance vs llama.cpp RPC mode.

alchemist1e9• 11 months ago

Any pointers to what is SOTA for cluster of hosts with CUDA GPUs but not enough vram for full weights, yet 10Gbit low latency interconnects?

If that problem gets solved, even if for only a batch approach that enables parallel batch inference resulting in high total token/s but low per session, and for bigger models, then it would he a serious game changer for large scale low cost AI automation without billions capex. My intuition says it should be possible, so perhaps someone has done it or started on it already.

echelon• 11 months ago

This is really impressive.

If we can get this down to a single Raspberry Pi, then we have crazy embedded toys and tools. Locally, at the edge, with no internet connection.

Kids will be growing up with toys that talk to them and remember their stories.

We're living in the sci-fi future. This was unthinkable ten years ago.

striking• 11 months ago

I think it's worth remembering that there's room for thoughtful design in the way kids play. Are LLMs a useful tool for encouraging children to develop their imaginations or their visual or spatial reasoning skills? Or would these tools shape their thinking patterns to exactly mirror those encoded into the LLM?

I think there's something beautiful and important about the fact that parents shape their kids, leaving with them some of the best (and worst) aspects of themselves. Likewise with their interactions with other people.

The tech is cool. But I think we should aim to be thoughtful about how we use it.

bigyabai• 11 months ago

Kids will be growing up with toys that talk to them and remember their stories.

What a radical departure from the social norms of childhood. Next you'll tell me that they've got an AI toy that can change their diaper and cook Chef Boyardee.

manmal2• 11 months ago

An LLM in my kids‘ toys only over my cold, dead body. This can and will go very very wrong.

cdelsolar• 11 months ago

Why

fragmede2• 11 months ago

If a raspberry pi can do all that, imagine the toys Bill Gates' grandkids have access to!

We're at the precipice of having a real "A Young Lady's Illustrated Primer" from The Diamond Age.

9991• 11 months ago

Bill Gates' grandkids will be playing with wooden blocks.

1gn15• 11 months ago

This is indeed incredibly sci fi. I still remember my ChatGPT moment, when I realized I could actually talk to a computer. And now it can run fully on an RPi, just as if the RPi itself has become intelligent and articulate! Very cool.

dingdingdang2• 11 months ago

Very impressive numbers.. wonder how this would scale on 4 relatively modern desktop PCs, like say something akin to a i5 8th Gen Lenovo ThinkCentre, these can be had for very cheap. But like @geerlingguy indicates - we need model compatibility to go up up up! As an example it would amazing to see something like fastsdcpu run distributed to democratize accessibility-to/practicality-of image gen models for people with limited budgets but large PC fleets ;)

rthnbgrredf• 11 months ago

I think it is all well and good, but the most affordable option is probably still to buy a used MacBook with 16/32 or 64 GB (depending on the budget) unified memory and install Asahi Linux for tinkering.

Graphics cards with decent amount of memory are still massively overpriced (even used), big, noisy and draw a lot of energy.

ivape4• 11 months ago

It just came to my attention that the 2021 M1 Max 64gb is less than $1500 used. That’s 64gb of unified memory at regular laptop prices, so I think people will be well equipped with AI laptops rather soon.

Apple really is #2 and probably could be #1 in AI consumer hardware.

jibbers• 11 months ago

Get an Apple Silicon MacBook with a broken screen and it’s an even better deal.

giancarlostoro5• 11 months ago

You dont even need Asahi, you can run comfy on it but I recommend the Draw Things app, it just works and holds your hand a LOT. I am able to run a few models locally, the underlying app is open source.

croes2• 11 months ago

What about AMD Ryzen AI Max+ 395 Mini PCs with upto 128GB unified memory?

Aurornis2• 11 months ago

and install Asahi Linux for tinkering.

I would recommend sticking to macOS if compatibility and performance are the goal.

Asahi is an amazing accomplishment, but running native optimized macOS software including MLX acceleration is the way to go unless you’re dead-set on using Linux and willing to deal with the tradeoffs.

benreesman2• 11 months ago

If Moore's Law is Ending leaks are to be believed, there are going to be 24GB GDDR7 5080 Super and maybe even 5070 Super Ti variants in the 1k (MSRP) range and one assumes fast Blackwell NVFP4 Tensor Cores.

Depends on what you're doing, but at FP4 that goes pretty far.

nullsmack• 11 months ago

The mini pcs based on AMD Ryzen AI Max+ 395 (Strix Halo) are probably pretty competitive with those. Depending on which one you buy it's $1700-2000 for one with 128GB RAM that is shared with the integrated Radeon 8060S graphics. There's videos on youtube talking about using this with the bigger LLM models.

j45• 11 months ago

Connect a gpu into it with an eGPU chassis and you're running one way or the other.

trebligdivad2• 11 months ago

On my (single) AMD 3950x running entirely in CPU (llama -t32 -dev none), I was getting 14 tokens/s running Qwen3-Coder-30B-A3B-Instruct-IQ4_NL.gguf last night. Which is the best I've had out of a model that doesn't feel stupid.

codedokode2• 11 months ago

How much RAM it is using by the way? I see 30B, but without knowing precision it is unclear how much memory one needs.

vient• 11 months ago

For reference, I get 29 tokens/s with the same model using 12 threads on AMD 9950X3D. Guess it is 2x faster because AVX-512 is 2x faster on Zen 5, roughly speaking. Somewhat unexpectedly, increasing number of threads decreases performance, 16 threads already perform slightly worse and with 32 threads I only get 26.5 tokens/s.

On 5090 same model produces ~170 tokens/s.

kosolam• 11 months ago

How is this technically done? How does it split the query and aggregates the results?

magicalhippo• 11 months ago

From the readme:

More devices mean faster performance, leveraging tensor parallelism and high-speed synchronization over Ethernet.

The maximum number of nodes is equal to the number of KV heads in the model #70.

I found this[1] article nice for an overview of the parallelism modes.

[1] https://medium.com/@chenhao511132/parallelism-in-llm-inferen...

varispeed• 11 months ago

So would 40x RPi 5 get 130 token/s?

SillyUsername• 11 months ago

I imagine it might be limited by number of layers and you'll get diminishing returns as well at some point caused by network latency.

VHRanger• 11 months ago

Most likely not because of NUMA bottlenecks

reilly3000• 11 months ago

It has to be 2^n nodes and limited to one per attention head that the model has.

behnamoh2• 11 months ago

Everything runs on a π if you quantize it enough!

I'm curious about the applications though. Do people randomly buy 4xRPi5s that they can now dedicate to running LLMs?

ryukoposting• 11 months ago

I'd love to hook my development tools into a fully-local LLM. The question is context window and cost. If the context window isn't big enough, it won't be helpful for me. I'm not gonna drop $500 on RPis unless I know it'll be worth the money. I could try getting my employer to pay for it, but I'll probably have a much easier time convincing them to pay for Claude or whatever.

exitb• 11 months ago

I think the problem is that getting multiple Raspberry Pi’s is never the cost effective way to run heavy loads.

halJordan• 11 months ago

This is some sort of joke right?

numpad0• 11 months ago

MI50 is cheaper

rs186• 11 months ago

$500 gives you about 6 RPi 5 8GB or 4 16GB, excluding accessories or other necessary equipment to get this working.

You'll be much better off spending that money on something else more useful.

fastball• 11 months ago

Capability of the model itself is presumably the more important question than those other two, no?

amelius2• 11 months ago

I'd love to hook my development tools into a fully-local LLM.

Karpathy said in his recent talk, on the topic of AI developer-assistants: don't bother with less capable models.

So ... using an rpi is probably not what you want.

pdntspa3• 11 months ago

Model intelligence should be part of your equation as well, unless you love loads and loads of hidden technical debt and context-eating, unnecessarily complex abstractions

throaway920181• 11 months ago

It's sad that Pis are now so overpriced. They used to be fun little tinker boards that were semi-cheap.

hhh• 11 months ago

I have clusters of over a thousand raspberry pi’s that have generally 75% of their compute and 80% of their memory that is completely unused.

Moto7451• 11 months ago

That’s an interesting setup. What are you doing with that sort of cluster?

larodi• 11 months ago

Is it solar powered?

CamperBob2• 11 months ago

Good ol' Amdahl in action.

fragmede2• 11 months ago

That sounds awesome, do you have any pictures?

6r17• 11 months ago

I mean at this point it's more of a "proof-of-work" with shared BP ; I would deff see some domotic hacker get this running - hell maybe i'll do this do if I have some spare time and want to make something like alexa with customized stuff - would still need text to speech and speech to text but that's not really the topic of his set-up ; even for pro use if that's really usable why not just spawn qwen on ARM if that's cheaper - there is a lot of way to read and leverage such bench

ugh123• 11 months ago

I think it serves a good test bed to test methods and models. We'll see if someday they can reduce it to 3... 2... 1 Pi5's that can match performance.

giancarlostoro5• 11 months ago

Sometimes you buy a pi for one project start on it buy another for a different project, before you know it none are complete and you have ten Raspberry Pis lying around across various generations. ;)

dotancohen6• 11 months ago

Arduino hobbist, same issue.

Though I must admit to first noticing the trend decades before discovering Arduino when I looked at the stack of 289, 302, and 351W intake manifolds on my shelf and realised that I need the width of the 351W manifold but the fuel injection of the 302. Some things just never change.

Zenst• 11 months ago

Depends on the model - if you have a sparse model with MoE, then you can divide it up into smaller nodes, your dense 30b models, I do not see them flying anytime soon.

Intel pro B50 in a dumpster PC would do you well better at this model (not enough ram for dense 30b alas) and get close to 20 tokens a second and so much cheaper.

piecerough• 11 months ago

"quantize enough"

though at what quality?

dotancohen6• 11 months ago

Quantity has a quality all its own.

blululu• 11 months ago

For $500 you may as well spend an extra $100 and get a Mac mini with an m4 chip and 256gb of ram and avoid the headaches of coordinating 4 machines.

MangoToupe2• 11 months ago

I don't think you can get 256 gigs of ram in a mac mini for $600. I do endorse the mac as an AI workbench tho

mmastrac2• 11 months ago

Is the network the bottleneck here at all? That's impressive for a gigabit switch.

kristianp• 11 months ago

Does the switch use more power than the 4 pis?

mmastrac2• 11 months ago

Modern GB switches are pretty efficient (<10W for sure), I think a Pi might be 4-5W.

tarruda• 11 months ago

I suspect you'd get similar numbers with a modern x86 mini PC that has 32GB of RAM.

misternintendo• 11 months ago

At this speed this is only suitable for time insensitive applications..

daveed• 11 months ago

I mean it's a raspberry pi...

layer8• 11 months ago

I’d argue that chat is a time-sensitive application, and 13 tokens/s is significantly faster than I can read.

poly2it• 11 months ago

Neat, but at this price scaling it's probably better to buy GPUs.

rao-v3• 11 months ago

Nice! Cheap RK3588 boards come with 15GB of LPDDR5 RAM these days and have significantly better performance than the Pi 5 (and often are cheaper).

I get 8.2 tokens per second on a random orange pi board with Qwen3-Coder-30B-A3B at Q3_K_XL (~12.9GB). I need to try two of them in parallel ... should be significantly faster than this even at Q6.

jerrysievert• 11 months ago

a random orange pi board with Qwen3-Coder-30B-A3B at Q3_K_XL (~12.9GB)

fantastic! what are you using to run it, llama.cpp? I have a few extra opi5's sitting around that would love some extra usage

rao-v3• 11 months ago

Yup! Build and ignore KleinAI and Vulkan etc. I’ve found that a clean CPU only build is optimal

ThatPlayer2• 11 months ago

Is that using the NPU on that board? I know it's possible to use those too.

rao-v3• 11 months ago

It is possibly (superb subreddit) but painful to convert a modern model and takes ages for them to be supported. The NPU is energy efficient but no faster than CPU for generation (and has lousy software support).

I’m mostly interested in the NPu to run a vision head in parallel with an LLM to speed up time to first token with VLLMs (kinda want to turn them into privacy safe vision devices for consumer use cases)

ineedasername• 11 months ago

This is highly usable in an enterprise setting when the task benefits from near-human level decision making and when $acceptable_latency < 1s meets decisions that can be expressed in natural language <= 13tk.

Meaning that if you can structure a range of situations and tasks clearly in natural language with a pseudo-code type of structure and fit it in model context then you can have an LLM perform a huge amount of work with Human-in-the-loop oversight & quality control for edge cases.

Think of office jobs, white colar work, where, business process documentation and employee guides and job aids already fully describe 40% to 80% of the work. These are the tasks most easily structured with scaffolding prompts and more specialized RLHF enriched data, and then perform those tasks more consistently.

This is what I decribe when I'm asked "But how will they do $X when they can't answer $Y without hallucinating?"

I explain the above capability, then I ask the person to do a brief thought experiment: How often have you heard, or yourself thought something like, "That is mindnumbingly tedious" and/or "a trained monkey could do it"?

In the end, I don't know anyone whose is aware of the core capabilities in the structured natural-language sense above, that doesn't see at a glance just how many jobs can easily go away.

I'm not smart enough to see where all the new jobs will be or certain there will be as many of them, if I did I'd start or invest in such businesses. But maybe not many new jobs get created, but then so what?

If the net productivity and output-- essentially the wealth-- of the global workforce remains the same or better with AI assistance and therefore fewer work hours, that means... What? Less work on average, per capita. More wealth, per work hour worked per Capita than before.

Work hours used to be longer, they can shorten again. The problem is getting there. To overcoming not just the "sure but it will only be the CEOs get wealthy" side of things to also the "full time means 40 hours a week minimum." attitude by more than just managers and CEOs.

It will also mean that our concept of the "proper wage" for unskilled labor that can't be automated will have to change too. Wait staff at restaurants, retail workers, countless low end service-workers in food and hospitality? They'll now be providing-- and giving up-- something much more valuable than white colar skills that are outdated. They'll be giving their time to what I've heard, and the term is jarring to my ears but it is what it is, I've heard it described as "embodied work". And I guess the term fits. And anyway I've long considered my time to be something I'll trade with a great deal more reluctance than my money, and so demand a lot money for it when it's required so I can use that money to buy more time (by not having to work) somewhere in the near future, even if it's just by covering my costs for getting groceries delivered instead of the time to go shopping myself.

Wow, this comment got away from me. But seeing Qwen3 30B level quality with 13tk/s on dirt cheap HW struck a deep chord of "heck, the global workforce could be rocked to the core for cheap+quality 13tk/s." And that alone isn't the sort of comment you can leave as a standalone drive-by on HN and have it be worth the seconds to write it. And I'm probably wrong on a little or a lot of this and seeing some ideas on how I'm wrong will be fun and interesting.

drbscl• 11 months ago

Distributed compute is cool, but $320 for 13 tokens/s on a tiny input prompt, 4 bit quantization, and 3B active parameter model is very underwhelming

ab_testing• 11 months ago

Would it work better on a used GPU?

bjt12345• 11 months ago

Does Distributed Llama use RDMA over Converged Ethernet or is this roadmapped? I've always wondered if RoCE and Ultra-Ethernet will trickle down into the consumer market.

rldjbpin• 11 months ago

how would llm-d [1] work compared to distributed-llama? is the overhead or configuration too much to work with for simple setups?

[1] https://github.com/llm-d/llm-d/

Qwen3 30B A3B Hits 13 token/s on 4xRaspberry Pi 5

Related Stories