I regret building this $3000 Pi AI cluster

https://www.jeffgeerling.com/blog/2025/i-regret-building-3000-pi-ai-cluster

by speckx • 10 months ago

468 354 10 months ago

elzbardico• 10 months ago

Frankly, always thought about Pi Clusters as a nerd indulgence, something to play, not to do serious work.

devmor• 10 months ago

After a few years of experience with them I agree for the most part. They are great for individual projects and even as individual servers for certain loads, but once you start clustering them you will probably get better results from a purpose built computer in the same price range as multiple pis.

randomgermanguy2• 10 months ago

I think the only exception is specifically for studying network/communciation-topologies. I've seen a couple clusters (ca. 10-50 Pi's) in universities for both research and teaching.

ofrzeta2• 10 months ago

There are so many network emulators you can use, such as Mininet or GNS3.

NitpickLawyer• 10 months ago

It reminds me of the Beowulf clusters of the 90s-2000s, that were all the rage at some point, then slowly lost ground... I remember many friends tinkering with some variant of those, we had one in Uni, and there were even some linux distros dedicated to the concept.

gary_0• 10 months ago

Oh yeah, the "imagine a Beowulf cluster of these" Slashdot meme! I miss those days. At least the "can it run Doom?" meme is still alive and kicking.

unregistereddev• 10 months ago

Ditto! It reminded me of the time in college when I built a Beowulf cluster from recently-retired Pentium II desktops.

Was it fast? No. But that wasn't the point. I was learning about distributed computing.

dekhn• 10 months ago

Beowulf style clusters went on to dominate supercomputing.

fidotron• 10 months ago

If Pi Clusters were actually cost competitive for performance there would be data centres full of them.

phoronixrly3• 10 months ago

If they were cost competitive for ... anything at all really...

wltr2• 10 months ago

Well I have a Pi as a home server, and it’s very energy efficient, while doing what I want. Since I don’t need latest and greatest (I don’t see any difference with a modern PC for my use case), it’s very competitive for me. No need for any cooling is bonus.

ACCount37• 10 months ago

Prototyping and low volume.

They're good for long as the development costs dominate the total costs.

jacobr1• 10 months ago

They are competitive for hobbyist use cases. Limited home servers, or embedded applications that overlap with arduino.

magicalhippo• 10 months ago

I picked up several Rpi 4 2GB for $20 each just before covid-19. At that price point they've been quite competitive for small homelab workloads.

The current RPi 5 makes no sense to me in any configuration, given its pricing.

ssl-34• 10 months ago

It depends on the application.

If one just wants a cheap desktop box to do desktop things with, then they're a terrible option, price-wise, compared to things like used corpo mini-PCs.

But they're reasonably cost-competitive with other new (not used!) small computers that are tinkerer-friendly, and unlike many similar constructs there's a plethora of community-driven support for doing useful things with the unusual interfaces they expose.

rebolek• 10 months ago

They are cost competitive enough for Korg synthesizers which is pretty OK for me.

shermantanktop2• 10 months ago

Like the joke about the economists not picking up the $20 bill on the ground?

Faith in the perfect efficiency of the free market only works out over the long term. In the short term we have a lot of habits that serve as heuristics for doing a good job most of the time.

IAmBroom3• 10 months ago

Faith in the perfect efficiency of the free market only works out over the long term

... and even then it doesn't always prove true.

infecto• 10 months ago

Sure but for commodities, like server hardware, we can say it’s usually directionally correct. If there are no pi cloud offerings, there is probably a good economic reason for it.

ThrowawayR2• 10 months ago

There's been so much investigation into alternative architectures for datacenters and cloud providers, including FAANG resorting to designing their own ARM processors and accelerator chips (e.g. AWS Graviton, Google TPUs) and having them fabbed, that that comes off not as warranted cynicism but silly cynicism.

uncircle3• 10 months ago

Like the joke about the economists not picking up the $20 bill on the ground?

For those like me that don't know the joke:

Two economists are walking down the street. One of them says “Look, there’s a twenty-dollar bill on the sidewalk!” The other economist says “No there’s not. If there was, someone would have picked it up already.”

themafia• 10 months ago

It's quite the opposite when corruption becomes involved. There are definite financial incentives for middle men to deliver inefficient and wasteful experiences.

Competition is what creates efficiency. Without it you live in a lie.

miunau• 10 months ago

Mythic Beasts rents rpi servers: https://www.mythic-beasts.com/order/rpi/ - there is a niche for it

mayli2• 10 months ago

This is also a proof where you can get cheaper and more perf/$ if you buy x86 based vps.

rozab• 10 months ago

People said the same about PlayStations, to be fair

dbg31415• 10 months ago

Was it cost effective? Meh.

Was it a learning experience?

More importantly, did you have some fun? Just a little? (=

phoronixrly3• 10 months ago

Was it a learning experience?

Also no. The guy's a youtuber

On the other hand, will this make him 100+k views? Yes. It's bait - the perfect combo to attract both the AI crowd and the 'homelab' enthusiasts (of which the bulk are yet to find any use for their raspberry devices)...

aprdm4• 10 months ago

He is not a YouTuber. And even if he was - what's the problem ?

Jeff has many useful OSS software used by many companies around the world daily - including mine. What have you created ?

Coffeewine• 10 months ago

It's a pretty rough headline, clearly the author had fun performing the test and constructing the thing.

I would be pretty regretful of just the first sentence in the article, though:

I ordered a set of 10 Compute Blades in April 2023 (two years ago), and they just arrived a few weeks ago.

That's rough.

geerlingguy13• 10 months ago

That's the biggest regret; but I've backed 6 Kickstarter projects over the years. Median time to deliver is 1 year.

Somehow I've actually gotten every item I backed shipped at some point (which is unexpected).

Hardware startups are _hard_, and after interacting with a number of them (usually one or two people with a neat idea in an underserved market), it seems like more than half fail before delivering their first retail product. Some at least make it through delivering prototypes/crowdfunded boards, but they're already in complete disarray by the end of the shipping/logistics nightmares.

maartin0• 10 months ago

Not completely related, but do you know if hardware kickstarters typically have any IP protection? I'm surprised there haven't been any cases of large companies creating patents for ideas from kickstarter at least that I've seen

milesvp• 10 months ago

Seriously. Imagine the fractal nature of software problems (you do need firmware right?), and add that on top of a layer that has poorly documented errata and non standard peripheral variants. I'm currently making a board to solder onto an existing production run because we finally found the 1 sentence in a 1700page data sheet that explained why many of our MCUs are overheating. Had the datasheet simply labeled VREFP as VDDA, it probably would have triggered a review of the power schematic. Oh, and the CPU overheating was masked by component variation, and CPU binning, which means the overheating wasn't obvious on any of the early prototype boards...

And then, there's the sourcing problem. Components that looked like they were in big supply when the hardware was specced, can end up being in short supply, or worse end of lifed while you're trying to get all the firmware working.

bravetraveler• 10 months ago

Wow that's a lot of scratch for... scratch. Pays for itself, I'm sure: effective bait :)

'Worth it any more'? At this size, never. A Pi is a Pi is a Pi!

A few are fine for toying around; beyond that, hah. Price:perf is rough, does not improve with multiplication [of units, cost, or complexity].

xnx• 10 months ago

Fun project. Was the author hoping for cost effective performance?!

I assumed this was a novelty, like building a RAID array out of floppy drives.

leptons• 10 months ago

A lot of people don't understand the performance limits of the Raspberry Pi. It's a great little platform for some things, but it isn't really fit for half the use cases I've seen.

Our_Benefactors• 10 months ago

This was my impression as well, the bit about GPU incompatibility with llama.cpp made me think he was in over his head.

LTL_FTC4• 10 months ago

The author is a YouTuber and projects like these pay for themselves with the views they garner. Even the title is designed for engagement.

lumost2• 10 months ago

I don’t really get why anyone would be buying ai compute unless A) to your goal is to rent out the compute B) no vendor can rent you enough compute when you need it C) you have an exotic funding arrangement that makes compute capex cheap and opex expensive.

Unless you can keep your compute at 70% average utilization for 5 years - you will never save money purchasing your hardware compared to renting it.

2OEH8eoCRo0• 10 months ago

I don't get why anyone would hack on and have fun with unique hardware either /s

seanw444• 10 months ago

It's also not always just about fun or cost effectiveness. Taking the infrastructure into your own hands is a nice way to know that you're not being taken advantage of, and you only have yourself to rely on to make the thing work. Freedom and self-reliance, in short.

HenryMulligan• 10 months ago

Data privacy and security don't matter? My secondhand RTX 3060 would buy a lot of cloud credits, but I don't want tons of highly personal data sent to the cloud. I can't imagine how it would be for healthcare and finance, at least if they properly shepherded their data.

tern• 10 months ago

For most people, no, privacy does not matter in this sense, and "security" would only be a relevant term if there was a pre-existing adversarial situation

justinrubek• 10 months ago

At some point, the work has to actually be done rather than shuffling the details off to someone else.

causal• 10 months ago

1) Data proximity (if you have a lot of data, egress fees add up)

2) Hardware optimization (the exact GPU you want may not always be available for some providers)

3) Not subject to price changes

4) Not subject to sudden Terms of Use changes

5) Know exactly who is responsible if something isn't working.

6) Sense of pride and accomplishment + Heating in the winter

horsawlarway3• 10 months ago

There are an absolutely stunning number of ways to lose a whole bunch of money very quickly if you're not careful renting compute.

$3,000 is well under many "oopsie billsies" from cloud providers.

And that's outside of the whole "I own it" side of the conversation, where things like latency, control, flexibility, & privacy are all compelling reasons to be willing to spend slightly more.

I still run quite a number of LLM services locally on hardware I bought mid-covid (right around 3k for a dual RTX3090 + 124gb system ram machine).

It's not that much more than you'd spend if you're building a gaming machine anyways, and the nifty thing about hardware I own is that it usually doesn't stop working at the 5 year mark. I have desktops from pre-2008 still running in my basement. 5 year amortization might have the cloud win, but the cloud stops winning long before most hardware dies. Just be careful about watts.

Personally - I don't think pi clusters really make much sense. I love them individually for certain things, and with a management plane like k8s, they're useful little devices to have around. But I definitely wouldn't plan to get good performance from 10 of them in a box. Much better off spending roughly the same money for a single large machine unless you're intentionally trying to learn.

0xbadcafebee• 10 months ago

You could also spill a can of Mountain Dew over the $8,000 AI rig next to you. Oopsies can happen anywhere...

If it's for personal use, do whatever... there's nothing wrong with buying a $60,000 sports car if you get a lot of enjoyment out of driving it. (you could also lease if you want to trade up to the "faster model" next year) For business, renting (and managed hosting) makes more sense.

lumost2• 10 months ago

At the local/hobby scale, it’s very much a “do whatever” area. But I can rent a 4090 for a little under a dollar an hour, and I can rent a b200 for $6, it’s very hard to claim I’ll use 10k+ hours of gpu time on a b2000 I buy for myself.

a21282• 10 months ago

Why do people buy gaming PCs when it's much cheaper to use streaming platforms? I think the two cases share practically the same parallels in terms of reliability, availability, restrictions, flexibility, sovereignty, privacy, etc.

But also when it comes to Vast/RunPod it can be annoying and genuinely become more expensive if you have to rent 2x the number of hours because you constantly have to upload and download data, checkpoints, continuous storage costs, transfer data to another server because the GPU is no longer available, etc. It's just less of a headache if you have an always available GPU with a hard drive plugged into the machine and that's it

cellis• 10 months ago

And lest we forget! Forgetting to turn it off!

ripdog• 10 months ago

Because latency matters when gaming in a way which doesn't matter with AI inference?

Plus cloud gaming is always limited in range of games, there are restrictions on how you can use the PC (like no modding and no swapping savegames in or out).

bunderbunder2• 10 months ago

Reminds me a bit of one of my favorite NormConf sessions, "Just use one big machine for model training and inference." https://youtu.be/9BXMWDXiugg?si=4MnGtOSwx45KQqoP

Or the oldie-but-goodie paper "Scalability! But at what COST?": https://www.usenix.org/system/files/conference/hotos15/hotos...

Long story short, performance considerations with parallelism go way beyond Amdahl's Law, because supporting scale-out also introduces a bunch of additional work that simply doesn't exist in a single node implementation. (And, for that matter, multithreading also introduces work that doesn't exist for a sequential implementation.) And the real deep down black art secret to computing performance is that the fastest operations are the ones you don't perform.

Aurornis3• 10 months ago

I thought the conclusion should have been obvious: A cluster of Raspberry Pi units is an expensive nerd indulgence for fun, not an actual pathway to high performance compute. I don’t know if anyone building a Pi cluster actually goes into it thinking it’s going to be a cost effective endeavor, do they? Maybe this is just YouTube-style headline writing spilling over to the blog for the clicks.

If your goal is to play with or learn on a cluster of Linux machines, the cost effective way to do it is to buy a desktop consumer CPU, install a hypervisor, and create a lot of VMs. It’s not as satisfying as plugging cables into different Raspberry Pi units and connecting them all together if that’s your thing, but once you’re in the terminal the desktop CPU, RAM, and flexibility of the system will be appreciated.

bunderbunder2• 10 months ago

The cost effective way to do it is in the cloud. Because there's a very good chance you'll learn everything you intended to learn and then get bored with it long before your cloud compute bill reaches the price of a desktop with even fairly modest specs for this purpose.

aprdm4• 10 months ago

That really depends on what you want to learn and how deep. If you're automating things before the hypervisor comes online or there's an OS running (e.g: working on datacenter automation, bare metal as a service) you will have many gaps

Almondsetat3• 10 months ago

I can get a Xeon E5-2690V4 with 28 threads and 64GB of RAM for about $150. If you need cores and memory to make a lot of VMs you can do it extremely cheaply

montebicyclelo• 10 months ago

Yeah... Looks like can get about $1/hr for 10 small VMs, ($0.10 per VM).

So for $3000, that's 3000 hours, or 125 days, (if just wastefully leave them on all the time, instead of turning them on when needed).

Say you wanted to play around for a couple of hours, that's like.. $3.

(That's assuming there's no bonus for joining / free tier, too.)

nsxwolf• 10 months ago

That isn’t fun. I have a TI-99/4A in my office hooked up to a raspberry pi so it can use the internet. Why? Because it’s fun. I like to touch and see the things even though it’s all so silly.

bakugo• 10 months ago

It heavily depends on the use case. For these AI setups, you're completely correct, because the people who talk about how amazing it is to run a <100B model at home almost never actually end up using it for anything real (mostly because these small models aren't actually very good) and are doing it purely for the novelty.

But if you're someone like me who intends to actively use the hardware for real-world purposes, the cloud often simply can't compete on price. At home, I have a mini PC with a 5600G, 32GB of RAM, and a few TBs of NVME storage. The entire thing cost less than $600 a few years ago, and consumes around 20W of power on average.

Even on the cheapest cloud providers available, an equivalent setup would exceed that price in less than half a year. SSD storage in particular is disproportionately expensive on the cloud. For small VMs that don't need much storage, it does make sense, but as soon as you scale up, cloud prices quickly start ballooning.

newsclues• 10 months ago

Text and reference books are free at the library.

You don’t need hardware to learn. Sure it helps but you can learn from a book and pen and paper exercises.

mattbillenstein4• 10 months ago

LOL, no

dukeyukey• 10 months ago

It's good for the soul to have your cluster running in your home somewhere.

pinkgolem2• 10 months ago

For learning I feel much safer setting everything up locally, worst case I have to reinstall my system.

In the cloud, worst case I have a bill over 5-6 digits.

And I know my ADD, 2 is not super unlikely.

sam1r• 10 months ago

A great way to do this is… is with a brand new Aws account, which will give you 1 year free across all services with reasonable limits.

cramcgrab• 10 months ago

I don’t know, i keyed this into google Gemini and got pretty far: “ Simulate an AWS AI cluster, command line interface. For each command supply the appropriate AWS AI cluster response”

motorest3• 10 months ago

The cost effective way to do it is in the cloud.

This. Some cloud providers offer VMs with 4GB RAM and 2 virtual cores for less than $4/month. If your goal is to learn how to work with clusters, nothing beats firing up a dozen VMs when it suits your fancy, and shut them down when playtime is over. This is something you can pull off in a couple of minutes with something like an Ansible script.

glitchc2• 10 months ago

I did some calculations on this. Procuring a Mac Studio with the latest Mx Ultra processor and maxing out the memory seems to be the most cost effective way to break into 100b+ parameter model space.

Palomides• 10 months ago

even a single new mac mini will beat this cluster on any metric, including cost

randomgermanguy2• 10 months ago

Depends on how heavy one wants to go with the quants (for Q6-Q4 the AMD Ryzen AI MAX chips seem better/cheaper way to get started).

Also the Mac Studio is a bit hampered by its low compute-power, meaning you really can't use a 100b+ dense model, only MoE feasibly without getting multi minute prompt-processing times (assuming 500+ tokens etc.)

the8472• 10 months ago

You could try getting a DGX Thor devkit with 128GB unified memory. Cheaper than the 96GB mac studio and more FLOPs.

eesmith• 10 months ago

Geerling links to last month's essay on a Frameboard cluster, at https://www.jeffgeerling.com/blog/2025/i-clustered-four-fram... . In it he writes 'An M3 Ultra Mac Studio with 512 gigs of RAM will set you back just under $10,000, and it's way faster, at 16 tokens per second.' for 671B parameters, that is, that M3 is at least 3x the performance of the other three systems.

GeekyBear2• 10 months ago

Now that we know that Apple has added tensor units to the GPU cores the M5 series of chips will be using, I might be asking myself if I couldn't wait a bit.

teleforce2• 10 months ago

Not quite, as it stands now the most cost effective way is most likely framework desktop or similar system for example HP G1a laptop/PC [1],[2].

[1] The Framework Desktop is a beast:

https://news.ycombinator.com/item?id=44841262

[2] HP ZBook Ultra:

https://www.hp.com/us-en/workstations/zbook-ultra.html

llm_nerd2• 10 months ago

The next generation M5 should bring the matmul functionality seen on the A19 Pro to the desktop SoC's GPU -- "tensor" cores, in essence -- and will dramatically improve the running of most AI models on those machine.

Right now the Macs are viable purely because you can get massive amounts of unified memory. Be pretty great when they have the massive matrix FMA performance to complement it.

encom• 10 months ago

Mac
cost effective

lmao

moduspol2• 10 months ago

Also cost effective is to buy used rack mount servers from Amazon. They may be out of warranty but you get a lot more horsepower for your buck, and now your VMs don’t have to be small.

Aurornis3• 10 months ago

Putting a retired datacenter rack mount server in your house is a great way to learn how unbearably loud a real rack mount datacenter server is.

Y_Y• 10 months ago

If you're following this path, make sure to use the finest traditional server rack that money can buy: https://www.ikea.com/ie/en/p/lack-side-table-white-30449908/

allanrbo• 10 months ago

No, again, just run VMs on your desktop/laptop. The software doesn't know or care if it's a rack mounted machine.

vlovich123• 10 months ago

I’d say it’s inconclusive. For traditional compute it wins on power and cost (it’ll always lose on space). The inference is noted to not be able to use the GPU due to llama.cpp’s vulkan backend AND that clustering software in llama.cpp is bad. I’d say it’s probably still going to be worse for AI but it’s inconclusive where it could be due to the software immaturity (ie not worth it today but could be with better software)

tracker1• 10 months ago

But will there be a CM6 while you're waiting for the software to improve?

llm_nerd2• 10 months ago

If you assume that the author did this to have content for his blog and his YouTube channel, it makes much more sense. Going back to the well with a "I regret" entry allows for extra exploiting of a pretty dubious venture.

YouTube is absolute jam packed full of people pitching home "lab" sort of AI buildouts that are just catastrophically ill-advised, but it yields content that seems to be a big draw. For instance Alex Ziskind's content. I worry that people are actually dumping thousands to have poor performing ultra-quantized local AIs that will have zero comparative value.

philipwhiuk2• 10 months ago

I doubt anyone does this seriously.

ww520• 10 months ago

Now. Imagine a Beawulf of these...

TZubiri4• 10 months ago

Fun fact, a raspberry pi does not have a built in Real Time Clock with its own battery, so it relies on network clocks to keep the time.

Another fun fact, the network module of the pi is actually connected to the USB bus, so there's some overhead as well as a throughput limitation.

Fun fact, the Pi does not have a power button, relying on software to shut down cleanly. If you lose access to the machine, it's not possible to avoid corrupted states on the disk.

Despite all of this, if you want to self host some website, the raspberry pi is still an amazingly cost effective choice, from anywhere between 2 to 20000 monthly users, one pi will be overprovisioned. And you can even get an absolutely overkill redundant pi as a failover, but still a single pi can reach 365 days of uptime with no problem, and as long as you don't reboot or lose power or lose internet, you can achieve more than a couple of nines of reliability.

But if you are thinking of a third, much less a 10th raspberry pi, you are probably scaling the wrong way, way before you reach the point where a quantity matters ( a third machine), it becomes cost effective to upgrade the quality of your one or two machines.

On the embedded side it's the same story, these are great for prototyping, but you are not going to order 10k and sell them in production, maybe a small 100 test batch? But you will optimize and make your own PCB before a mass batch.

stuxnet79• 10 months ago

Fun fact, a raspberry pi does not have a built in Real Time Clock with its own battery, so it relies on network clocks to keep the time.
Another fun fact, the network module of the pi is actually connected to the USB bus, so there's some overhead as well as a throughput limitation.
Fun fact, the Pi does not have a power button, relying on software to shut down cleanly. If you lose access to the machine, it's not possible to avoid corrupted states on the disk.

With all these caveats in mind, a raspberry pi seems to be an incredibly poor choice for distributed computing

alias_neo3• 10 months ago

the raspberry pi is still an amazingly cost effective choice

It's really not though. I've been a Pi user and fan since it was first announced, and I have dozens of them, so I'm not hating on RPi here; we did the maths some time back here on HN when something else Pi related came up.

If you go for a Pi5 with say 8GB RAM, by the time you factor in an SSD + HAT + PSU + Case + Cooler (+ maybe a uSD), you're actually already in mini-PC price territory and you can get something much more capable and feature complete for about the same price, or for a few £ more, something significantly more capable, better CPU, iGPU, you'll get an RTC, proper networking, faster storage, more RAM, better cooling, etc, etc, and you won't be using much more electricity either.

I went this route myself and have figuratively and literally shelved a bunch of Pis by replacing them with a MiniPC.

My conclusion, for my own use, after a decade of RPi use, is that a cheap mini PC is the better option these days for hosting/services/server duty and Pis are better for making/tinkering/GPIO related stuff, even size isn't a winner for the Pi any more with the size of some of the mini-PCs on the market.

geerlingguy13• 10 months ago

The Pi 5 / CM5 / Pi 500 series does have a built-in RTC now, though most models require you to buy a separate RTC battery to plug into the RTC battery jack.

wccrawford• 10 months ago

Geerling's titles have been increasingly click-bait for a while now. It's pretty sad, because I like his content, but hate the click-bait BS.

jonathanlydall• 10 months ago

If it makes an appreciable difference to how much money he makes on YouTube then I can’t begrudge him for doing it.

Don’t hate the player, hate the game.

mrguyorama2• 10 months ago

Blame Youtube. They are the ones that run a purposely zero sum and adversarial system for directing attention at your videos. If he doesn't have a high enough click rate on his videos, Youtube will literally stop showing them to people, even subscribers.

Youtube demonstrably wants clickbait titles and thumbnails. They built tooling to automatically A/B test titles and thumbnails for you.

Youtube could fix this and stop it if they want, but that might lose them 1% of business so they never will.

They love that you blame creators for this market dynamic instead of the people who literally create the market dynamic.

kolbe2• 10 months ago

The author, Jeff Geerling, is a very intelligent person. He has more experience with using niche hardware than almost anyone on earth. If he does something, there's usually a good a priori rationale for it.

buildbot2• 10 months ago

Jeff is a good person/blogger and does interesting projects but more experience with niche hardware than literally anyone is a stretch.

Like what about the people who maintain the alpha/sparc/parisc linux kernels? Or the designers behind idk tilera or tenstorrent hardware.

amelius2• 10 months ago

Is a Pi still considered "niche" hardware?

AceJohnny2• 10 months ago

If he does something, there's usually a good a priori rationale for it.

I greatly respect Jeff's work, but he's a professional YouTuber, so his projects will necessarily lean towards clickbait and riding trends (Jeff, I don't mean this as criticism!) He's been a great advocate for doing interesting things with RasPis, but "interesting" != "rational"

randomNumber7• 10 months ago

What I think is strange with stuff like this that you should be able to come to that conclusion without technical knowledge. Just the fact that everyone runs AI on GPUs and NVIDIAs stock skyrocketed since the AI boom should tell you s.th..

Did OP really think his fellow humans are that moronic that they just didn't find out you can plug in together a cuple of rasperri pis?

rustyminnow• 10 months ago

Nobody thought an RPI cluster would ever be competitive, and Geerling never expected anybody would. But it's fun to play "what if" and then make the thing just to see how it stacks up and that's his job. Any implication or suggestion of this being a good idea is just part of the story telling.

Asraelite• 10 months ago

I don’t know if anyone building a Pi cluster actually goes into it thinking it’s going to be a cost effective endeavor, do they?

Some Raspberry Pi products are sold at a loss, so I could see how it's in the realm of possibility.

aprdm4• 10 months ago

Love Jeff's ansible roles/playbooks and his cluster building ! Quite interesting, I should reserve some time to play with a Pi cluster and ansible, sounds fun

noelwelsh• 10 months ago

I'd love to understand the economics here. $3000 purely for fun seems like a lot. $3000 for promotion of a channel? consulting? seems reasonable.

philipwhiuk2• 10 months ago

Jeff has a million YouTube subscribers, gets $2000 a month from Patreon and has 200 GitHub sponsors.

The economics of spending $3,000 on a video probably work out fine.

geerlingguy13• 10 months ago

It's definitely a stretch for my per-video budget, but I did want to have a 'maxed out' Pi cluster for future testing as well.

A lot of people (here, Reddit, elsewhere) speculate about how good/bad a certain platform or idea is. Since I have the means to actually test how good or bad something is, I try to justify the hardware costs for it.

Similar to testing various graphics cards on Pis, I've probably spent a good $10,000 on those projects over the past few years, but now I have a version of every major GPU from the past 3 generations to test on, not only on Pi, but other Arm platforms like Ampere and Snapdragon.

Which is fun, but also educational; I've learned a lot about inference, GPU memory access, cache coherency, the PCIe bus...

So a lot of intangibles, many of which never make it directly into a blog post or video. (Similar story with my time experiments).

markx2• 10 months ago

"But if you're on the blog, you're probably not the type to sit through a video anyway. So moving on..."

Thank you!

AlfredBarnes• 10 months ago

My pi's are just an easy onramp for me to have a functional NAS, PIHole, and webcam security.

Not at all the best, but they were cheap. If i WANTED the best or reliable, i'd actually buy real products.

hn_throw_250915• 10 months ago

I read through it and it’s amusing but along with the title being something I’d receive in email from a newsletter mailing list I’ve never subscribed to (hoping it has an unsubscribe link at the bottom), there’s nothing really of hacker curiosity here to keep me hooked. It’s shallow and appeals to some LCD “I did the thing with the stuff and the results will shock you because of how obvious they are now click here” mentality. Vainposting at its most average. The Mac restoration video was somewhat easier to sit through if only because the picture quality beats out a handful of other YT videos doing the exactly same thing as I’m holding back a jaw grating wince of watching someone butchering a board with poor knowledge of soldering iron practice, so YMMV? Back to hackaday for me I think. I’m not here to read submarine resumes of people applying to work at Linus Tech Tips.

paxys• 10 months ago

This is just the evolution of clickbait titles. The only thing missing is a thumbnail of an AI generated raspberry pi cluster with a massive arrow pointing to it and the words "not worth it!!"

Joker_vD• 10 months ago

There also needs to be a Face Screaming in Fear Emoji plastered on the other side of it.

imtringued• 10 months ago

Oh come on Jeff, you forgot to buy GPUs for your AI cluster. Such a beginner mistake.

All you needed to do is buy 4x xtx 7900 used on ebay and build a four node raspberry pi cluster using the external GPU setup you've come up with in one of your previous blog posts [0].

[0] https://www.jeffgeerling.com/blog/2024/use-external-gpu-on-r...

geerlingguy13• 10 months ago

More on that soon... ;)

cosarara• 10 months ago

Compared to the $8,000 Framework Cluster I benchmarked last month, this cluster is about 4 times faster:

Slower. 4 times slower.

teleforce2• 10 months ago

That's definitely a typo because I've to read the sentence 3 times from the article still cannot make a sense until I saw the figure.

TL;DR, just buy one framework desktop and it's better than the Pi AI cluster of the OP in every single performance metrics including cost, performance, efficiency, headache, etc.

geerlingguy13• 10 months ago

Oops, fixed the typo! Thanks.

And regarding efficiency, in CPU-bound tasks, the Pi cluster is slightly more efficient. (Even A76 cores on a 16nm node still do well there, depending on the code being run).

drillsteps5• 10 months ago

If he was building compute device for LLM inference specifically it would help to check in advance what that would entail. Like GPU requirement. Which putting bunch of RPis in the cluster doesn't help one bit.

Maybe I'm missing something.

deadbabe2• 10 months ago

I really don’t understand the hype over raspberry Pi.

It’s an overrated, overhyped little computer. Like ok it’s small I guess but why is it the default that everyone wants to build something new on? Because it’s cheap? Whatever happened to buy once, cry once? Why not just build an actual powerful rig? For your NAS? For your firewalls? For security cameras? For your local AI agents?

theultdev3• 10 months ago

I use mine for a plex server.

I don't need to transcode + I need something I can leave on that draws little power.

I have a powerful rig, but the one time I get to turn it off is when I'd need the media server lol.

There's a lot of scenarios where power usage comes into play.

These clusters don't make much sense to me though.

deadbabe2• 10 months ago

That’s insane, drawing very little power from an always on server is a solved problem.

jonatron2• 10 months ago

In the category of SBC's, it's pretty much the only one that has good software support, not outdated images made with a bunch of kernel patches for a specific kernel version.

qhwudbebd• 10 months ago

This is certainly the reputation but I'm not sure they deserve it. They've always had the horrible closed-source bootloader with threadx running on the gpu, without a free alternative. At least up to pi4 they weren't bad at linux mainlining, but progress on upstreaming pi5 support has been glacial.

Cf. the various Beagle boards which have mainline linux and u-boot support right from release, together with real open hardware right down to board layouts you can customise. And when you come to manufacture something more than just a dev board, you can actually get the SoC from your normal distributor and drop it on your board - unlike the strange Broadcom SoCs rpi use.

I'm quite a lot more positive about rp2040 and rp2350, where they've at least partially broken free of that Broadcom ball-and-chain.

heresie-dabord• 10 months ago

an overrated, overhyped little computer.

No, you are dismissive because you don't care about the use-cases.

The R.Pi 4 , 400, and the 500 are great models. Consider all the advantages together:

i= support for current Debian

ii= stellar community

iii= ease of use (UX), especially for people new to Debian and/or coding and/or Linux

iv= quiet, efficient, low power and passively cooled

v= robust enough to be left running for a long time

There are cheaper, more performant x86 and ARM dev boards and SOCs. But nothing compares to the full set of advantages.

That said, building a $3K A.I. cluster is just a senseless, expensive lark. (^;

hendersoon• 10 months ago

I mean, obviously it isn't practical, he got a couple of videos out of it.

nromiun2• 10 months ago

There is a reason all the big supercomputers have started using GPUs in the last decade. They are much more efficient. If you want 32bit parallel performance just buy some consumer GPUs and hook them up. If you need 64bit buy some prosumer GPUs like the RTX 6000 Pro and you are done.

Nobody is really building CPU clusters these days.

ted_dunning• 10 months ago

Well, El Capitan uses AMD CPUs (which have integrated GPU capabilities) and it is right on top of the rankings lately.

Frontier is right behind it with the same arrangement.

Having honest to god dedicated GPUs on their own data bus with their own memory isn't necessarily the fastest way to roll.

nromiun2• 10 months ago

They do not. The CPUs are only there to support and push data to the GPUs. Much like Nvidia GH200 systems. Nobody buys these APU chips for their CPU parts.

For comparison there are 9,988,224 GPU compute units in El Capitan and only 1,051,392 CPU cores. Roughly one CPU core to push data to 10 GPU CUs.

anematode• 10 months ago

Unfortunately even the RTX 6000 Pro has nerfed double-precision throughput at about 2 TFLOPS, 64x slower than single precision. For comparison an EPYC 9755 does ~10 TFLOPS, while drawing less power. An A100 -- if you can find one -- is in the same ballpark.

The best option for DP throughput for hobbyists interested in HPC might be old AMD cards from before they, too, realized that scientific folks would pay up the nose for higher precision.

deater• 10 months ago

as someone who has built various raspberry pi clusters over the years (I even got an academic paper out of one) the big shame is that as far as I know it's still virtually impossible to use the fairly powerful GPUs they have for GPGPU work

fgbarben• 10 months ago

sub hundred gigaflop counts as "fairly powerful" now?

zamadatix• 10 months ago

The article focuses on compute performance but I wonder if that was ever the bottleneck considering the memory bandwidth involved.

amelius2• 10 months ago

Ok, what are the back-of-the-envelope computations that he should have done before starting to build this?

geerlingguy13• 10 months ago

Pi memory bandwidth is less than 10 GB/sec, so AI use cases will be extremely limited. Network I/O is maximum of 1 Gbps (or more if you do some unholy thing with M.2 NICs), so that also limits maximum networked performance.

But still can be decent for HPC learning, CI testing, or isolated multi-node smaller-app performance.

bearjaws• 10 months ago

Am I the only one who looks at both the Pi Cluster and the Framework PC and wonders how they are both slower and less cost effective than a MacBook Pro M4 Max? 88 token/s on a 2.3b model is not exactly great, most likely you will want a 32 or 70b model.

Drblessing2• 10 months ago

The bee-link AI max+ is the best value AI pc right now.

stirfish• 10 months ago

I came here to ask about these. You like yours?

Drblessing2• 10 months ago

Yes. Incredible form factor.

Havoc3• 10 months ago

Have a bunch of Pis too, but realized I can use them to create a high availability control plane for a k8s cluster. Pi4s are entirely adequate for that

phito2• 10 months ago

Could you please elaborate? I find this interesting as I'm building a k8s cluster and also have a bunch of RPIs laying around...

Havoc3• 10 months ago

3x Pi 4s booting off ssd running Talos with virtual IP enabled

You know in k8s you've got worker nodes and control plane nodes? The control planes don't need much horsepower, but they're what you need to be online all to communicate with the cluster. Pis work just fine for that.

WhitneyLand• 10 months ago

“with 160 GB of total RAM, shared by the CPU and iGPU, this could be a small, efficient AI Cluster, right? Well, you'd think.”

No, I wouldn’t think.

owaislone• 10 months ago

What is the ideal or realistic setup for a home lab to power big enough models running locally and service ~2-3 users at a time?

hamonrye• 10 months ago

Strange quirks for language syntax. I read about a parsing system that was distilled towards being-towards this particular species of silver-tail that existed below Golden Gate bridge. Godwin's Law?

bee_rider3• 10 months ago

The first benchmark I ran was my top500 High Performance Linpack cluster benchmark. This is my favorite cluster benchmark, because it's the traditional benchmark they'd run on massive supercomputers to get on the top500 supercomputer list. […]
After fixing the thermals, the cluster did not throttle, and used around 130W. At full power, I got 325 Gflops

I was sort of surprised to find that the top500 list on their website only goes back to 1993. I was hoping to find some ancient 70’s version of the list where his ridiculous Pi cluster could sneak on. Oh well, might as well take a look… I’ll pull from the sub-lists of

https://www.top500.org/lists/top500/

They give the top 10 immediately.

First list (June 1993):

     placement  name            RPEAK (GFlop/s)
     1          CM-5/1024       131.00
     10         Y-MP C916/16256 15.24

Last list he wins, I think (June 1996):

     1          SR2201/1024     307.20  
     10         SX-4/32         64.00

First list he’s bumped out of the top 10 (November 1997):

     1          ASCI Red        1,830.40
     10         T3E             326.40

I think he gets bumped off the full top500 list around 2002-2003. Unfortunately I made the mistake of going by Rpeak here, but they sort by Rmax, and I don’t want to go through the whole list.

Apologies for any transcription errors.

Actually, pretty good showing for such a silly cluster. I think I’ve been primed by stuff like “your watch has more compute power than the Apollo guidance computer” or whatever to expect this sort of thing to go way, way back, instead of just to the 90’s.

m3kw9• 10 months ago

when i saw your tok/s on a shtty model, i said yah.

yalogin• 10 months ago

Can the LLMs be deployed split across multiple Pis? I thought it was not possible, may be I am not caught up.

bityard• 10 months ago

I can think of SOME decent--if suboptimal--reasons to build an RPi cluster, but it never would have occurred to me to try running an LLM on one. Even if you could cluster hundreds or thousands of Pis, it's still just entirely the wrong architecture for running an AI model as they are currently built.

asimpleusecase• 10 months ago

Great write up! This is nice nerd catnip. And sharing a failed project teaches so much. Please, more should share the projects that absorbed tons of time yet never really delivered.

numpad0• 10 months ago

goes round-robin style asking each node to perform its prompt processing, then token generation.

Yeah, this is a now-long-wide-known issue with LLM processing. This can be remediated so that all nodes split computation, but then you'll come back to classical supercomputing problem of node interconnect latency/bandwidth bottlenecks.

It looks to me that many such interconnect simulate Ethernet cards. I wonder if it can be recreated using the M.2 slot rather than using that slot for node-local data, and cost effectively so(like cheaper than bunch of 10GE cards and short DACs).

wcchandler• 10 months ago

I was just exploring Pi’s and AI hats, so this post is appreciatively timely.

I’m finally at the point where I can dedicate time for building an AI with a specific use case in mind. I play competitive paintball and would like to utilize AI for a handful of things. Specifically hit detections in video streams. Pi’s were my natural choice simply because of low cost of entry and wide range of supported products to get a PoV running. I even thought about reaching out to Jeff and asking his input.

This post didn’t change my direction too much, but it did help level set some realistic expectations. So thanks for sharing.

ComputerGuru2• 10 months ago

Guys, don’t take the claim so literally. He’s a successful tech poster. He makes good money showing off his purchases the good money complaining about how expensive they were.

But certainly don’t imitate his choices, his economics aren’t your economics!

system22• 10 months ago

I totally agree. A person with that kind of builder knowledge already knows a decent GPU could 10x that compute power.

esskay• 10 months ago

Thats pretty much a given, but what the real takeaway should be from most of the content is that whatever you are doing, these days the answer in all likelihood is not to buy a raspberry pi. It's specs to price just do not add up at all anymore, and it's looking like a pretty damn stagnant place these days.

Computer03• 10 months ago

What if you need ARM? What is the best sub $100 sbc that I am missing? Orange Pi hardware always looks good but I hear a lot of negativity about the software that I don't really experience with Raspbian.

brcmthrowaway• 10 months ago

If only Dan Luu pivoted to this style of content

dzhiurgis• 10 months ago

Yea but Jeff’s videos are refreshing.

A lot of others are stuck in a loop where they essentially review tech for making more youtube videos - render times, colour accuracy, camera resolution, audio fidelity.

iLoveOncall• 10 months ago

With 160 GB of total RAM, shared by the CPU and iGPU, this could be a small, efficient AI Cluster, right? Well, you'd think.

I don't know anyone who would think this actually.

geerlingguy13• 10 months ago

Most people who haven't actually self-hosted AI models would think this—for some reason people still think RAM is RAM, and they don't think about specs like memory speed, whether the full amount of RAM is shared with the GPU, etc.

You'd be surprised by the number of emails, Instagram DMs, YouTube comments, etc. I get—even after explicitly showing how bad a system is at a certain task—asking if a Pi would be good for X, or if they could run ChatGPT on their laptop...

WhereIsTheTruth2• 10 months ago

brand fanboyism, and vendor locking, the bane of our society

the common denominator is always capital gain

capitalism is the reason why we haven't been able to go back to the moon and build bases there

pmarreck• 10 months ago

and yet spaceX is for-profit and is pushing the boundaries again, after NASA (a not-for-profit government-created monopoly) stagnated.

blanket-blaming capitalism without good reasoning is becoming the new red-flag of "can't think critically"

WhereIsTheTruth2• 10 months ago

NASA didn't stagnate, NASA landed humans on the Moon with 1960s technology

private space companies, despite decades of hype and funding, have stagnated by comparison

the fact that SpaceX depends heavily on government contracts just to function is yet another proof: their "innovation" isn't self sustaining, it's underwritten by taxpayer money

are you denying that NASA landed on the Moon?

Elon psyop doesn't work on me, i know who is behind it all, they need a charismatic sales man for the masses, just like Ford, Disney, Reagan and all, masking structural power with a digestible story for the masses

blanket-blaming capitalism without good reasoning is becoming the new red-flag of "can't think critically"

it's quite the opposite, people unable to take criticism of capitalism, talk about "critical thinking", how is China doing?

system22• 10 months ago

Look at it this way, you had fun playing with an expensive toy. A single $1000 GPU could 10x that, even when you were building those (not today's GPUs). You probably already knew it. But I gotta admit, the rig looks very nice.

densh• 10 months ago

For anyone interested in playing with distributed systems, I'd really recommend getting a single machine with latest 16-core CPU from AMD and just running 8 virtual machines on it. 8 virtual machines, with 4 hyper threads pinned per machine, and 1/8 of total RAM per machine. Create a network between them virtually within your virtualization software of choice (such as Proxmox).

And suddenly you can start playing with distributed software, even though it's running on a single machine. For resiliency tests you can unplug one machine at a time with a single click. It will annihilate a Pi cluster in Perf/W as well, and you don't have to assemble a complex web of components to make it work. Just a single CPU, motherboard, m.2 SSD, and two sticks of RAM.

Naturally, using a high core count machine without virtualization will get you best overall Perf/W in most benchmarks. What's also important but often not highlighted in benchmarks in Idle W if you'd like to keep your cluster running, and only use it occasionally.

bee_rider3• 10 months ago

Tangentially related: I really expected running old MPI programs on stuff like the AMD multi-chip workstation packages to become a bigger thing.

le-mark• 10 months ago

I actually worked with some MPI code way back. What MPI programs are you referring to?

cyberpunk• 10 months ago

Honestly why do you need so much cpu power? You can play with distributed systems just by installing Erlang and running a couple of nodes on whatever potato-level linux box you have laying around, including a single raspberry pi.

globular-toast3• 10 months ago

I've been saying this for years. When the last Raspberry Pi shortage happened people were scrambling to get them for building these toy clusters and it's such a shame. The Pi was made for paedogogy but I feel like most of them are wasted.

I run a K8s "cluster" on a single xcp-ng instance, but you don't even really have to go that far. Docker Machine could easily spin up docker hosts with a single command, but I see that project is dead now. Docker Swarm I think still lets you scale up/down services, no hypervisor required.

motorest3• 10 months ago

I've been saying this for years. When the last Raspberry Pi shortage happened people were scrambling to get them for building these toy clusters and it's such a shame. The Pi was made for paedogogy but I feel like most of them are wasted.

You're describing people using RPis to learn distributed systems, and you conclude that these RPis are wasted because RPis were made for paedogogy?

I run a K8s "cluster" on a single xcp-ng instance, but you don't even really have to go that far.

That's perfectly fine. You do what works for you, just like everyone else. How would you handle someone else accusing your computer resourcss of being wasted?

qmr2• 10 months ago

No need for so much CPU power, any old quad core would work.

subscribed• 10 months ago

Old quad core won't have all the virtualisation extensions.

anaganisk• 10 months ago

Aren’t newer CPUs especially AMDs more energy efficient?

malux85• 10 months ago

Yeah this is how I practiced Postgres hot standby and read replicas,

It was also how I learned to setup a Hadoop cluster, and a Cassandra cluster (this was 10 years ago when these technologies were hot)

Having knowledge of these systems and being able to talk about how I set them up and simulated recovery directly got me jobs that 2x and then 3x my salary, I would highly recommend all medium skilled developers setup systems like this and get practicing if you want to get up into the next level

user432678• 10 months ago

Yes, but this is boring. Saying this as an owner of home server with ProxMox.

teaearlgraycold• 10 months ago

Having seen someone else build and tinker with a 7 node Pi cluster it seems like an absolute waste of time. 1Gb/s networking, PCIe 3.0 x1, slow RAM, slow CPU. And with all the hats and accessories needed it’s not that good of a deal.

Getting some NUC-like machines makes a lot more sense to me. You’ll get 2.5Gb/s Ethernet at the least and way more FlOPS as well.

nathan_douglas• 10 months ago

You might really dislike the ESP32 cluster I'm planning right now :P

leakycap• 10 months ago

I dipped my toe into cluster computing when rendering a MAXON Cinema 4D project across a lab of ~30 dual G5 Power Macs.

Quickly learned that there is so much more to manage when you split a task up across systems, even when the system (like Cinema 4D) is designed for it.

nodesocket• 10 months ago

I feel like AI is not the right workload for this. I run a 4x node Kubernetes cluster (1x control and 3x workers) and it’s surprising good. While performance is certainly not state of the art by any stretch of the imagination it works great for HomeLab’in. Portainer, Glance, OpenSpeed Test, Semaphore UI, Uptime Kuma, and Home Assistant. Many small and lightweight containers is where a cluster such as this can shine.

djhworld• 10 months ago

I watched the video and enjoyed it, I think the most interesting part to me was running the distributed Llama.cpp, Jeff mentioned it seems to work in a linear fashion where processing would hop between nodes.

Which got me thinking about how do these frontier AI models work when you (as a user) run a query. Does your query just go to one big box with lots of GPUs attached and it runs in a similar way, but much faster? Do these AI companies write about how their infra works?

geerlingguy13• 10 months ago

ServeTheHome has a few videos covering AI servers and interconnects.

And yes, they basically have 1 Tbps+ interconnects and throw tens or hundreds of GPUs at queries. Nvidia was wise to invest so much in their networking side—they have massive bandwidth between machines and shared memory, so they can run massive models with tons of cards, with minimal latency.

It's still not as good as tons of GPU attached to tons of memory on _one_ machine, but it's better than 10, 25, or 40 Gbps networking that most small homelabs would run.

MomsAVoxell• 10 months ago

I have a SOPINE cluster board that I'd quite like to get booted up and pressed into some sort of useful service.

I think the biggest problem with cluster products is that they just don't work out of the box. Vendors haven't really done the "last 2%" of development required to make them viable - its left to us purchasers to get the final bits in place.

Still, it'll make a fun distributed computing experimental platform some day.

Just like the Inmos Transputer I've got somewhere, sitting in a box, waiting for a power supply ..

dragontamer• 10 months ago

Why build a cluster?

I believe the Rasp Pi cluster is one of the cheapest multi node / MPI machines you can buy. That's useful even if it is t fast. You need to practice the programming interfaces, not necessarily make a fast computer.

However, NUMA is also a big deal. The various AMD Threadrippers with multi-die memory controllers are better on this regards. Maybe the aging Threadrippers 1950x, yes it's much slower than modern chips but the NUMA issues are exaggerated (especially poor) on this old architecture.

That exaggerates the effects of good NUMA and now you as a programmer can get more NUMA skills.

Of course, the best plan is to spend $20,000,000++ on your own custom NUMA nodes cluster out of EPYCs or something.

-------

But no. The best supercomputers are your local supercomputers that you should rent some time from. You need a local box to see various issues and learn to practice programming.

zeristor• 10 months ago

If it worked out well he would be trumpeting the glory.

Not so good, and this is the sort of title. you need to bring the punters in for YouTube.

I don't mean to sound too cynical, I appreciate Jeff's videos, just wanted to point out that if you've spent money and time on content you can either ditch it or make a regret video.

Just so long as the thumbnails don't have an arrow on them I'm happy.

computersuck• 10 months ago

RPIs have always had shit cpu performance - the initial idea for the pi was to use very low cost broadcom crap chips that no one would buy and market it for "educational" use.

They are essentially for kids to play around with learning computers by blinking LEDs and integrating with circuit boards. The idea of building a high performance cluster with pis is dumb from day one

trhway• 10 months ago

They are essentially for kids to play around with learning computers by blinking LEDs and integrating with circuit boards.

Where a "kid" may be a 53 years old with 30+ years softdev experience who ultimately got to get to the stuff he wanted to for quite some time, and the "blinking LEDs" are a bunch of servos programmatically controlled based on input from a bunch of sensors. While there are definitely better alternatives based on various narrow metrics, especially when it may come to actual productization, the ease (and cheapness, so you don't think much about that spending) of starting with all those easily available for RPi servo array drive boards and various IO ports array boards and all the available software - it is hard to imagine how it can be more easy/cheaper/available than it already is with all that actual compute power and full-featured Linux environment.

yellowapple• 10 months ago

Incredible to me that you can buy a 10-pack of 256GB NVMe drives for less than $200. Seems like yesterday when those ran for $200 each.

OrangeMusic• 10 months ago

Imagine a Beowulf cluster of these!

I regret building this $3000 Pi AI cluster

Related Stories