The state of silicon and the GPU poors

https://www.latent.space/p/semianalysis

by swyx • 3 years ago

80 40 3 years ago

A flood of giant H200s and MI300s is great and all, but what about personal accelerators?

The current state of local inference/finetuning is insane, where the hardware that makes any financial sense is ancient Nvidia GPUs, like the 3090 (2020) or the RTX 8000 (2018). Or maybe the rare used AMD Radeon Pro, if you can actually find one.

The only hope seems to be the rumored large APUs Intel/AMD APUs and maybe Intel Battlemage, since AMD is seemingly complicit in preserving the low VRAM status quo with Nvidia.

treesciencebot• 3 years ago

VRAM is not the main constraint, is it? The computational power of any of the new graphical cards (beside from the highest end models, where the VRAM is the actual constraint, like RTX 4090s) is absurdly low on the stuff that actually matters (tensor cores, cuda cores, etc.). They are graphical cards, equipped with consumer grade VRAM (instead of HBM) and IMHO it will take a very big shift before we see them being used as real AI accelerators.

sp332• 3 years ago

Well if you don't have enough VRAM to hold the whole model, you have to swap out to main RAM for every iteration, which for most home/local setups means once for every single token generated. So having enough is kind of a minimum.

Me10002• 3 years ago

Nah, vram is definitely the main constraint for most people trying to do local inference of LLMs. If you look at a lot of the local LLM communities, for people who aren't super interested in training, many people suggest the M2 Ultra or M3 Max with > 128GB of unified RAM just because it has so much memory. Again, you might not be able to train or fine tune as well, but the first step is just being able to keep the weights in memory. Inference isn't that computationally intensive relative to just how much RAM you need.

brucethemoose27• 3 years ago

VRAM is everything.

The more VRAM you have, less aggressively you have to quantize models for inference, which in turn has huge speed/quality implications. You can run higher batch sizes, or draft models, or more caching, which increases efficiency. For LLMs specifically, you can load bigger models into VRAM in the first place. You can load more of a multimodal pipeline in VRAM without having to constantly swap everything out.

This is all 10x true for finetuning. Quality and speed is essentially determined by VRAM capacity, as long as you are not on truly ancient GPU like a P40 than't can't even do fp16.

As for architecture... TBH, many operations are heavily bandwidth bound these days. Sometimes a 3090 and a 4090 are essentially the same speed. And waiting a little longer for a finetune is no big deal vs not being able to do it at all, or doing it at low quality.

They are graphical cards, equipped with consumer grade VRAM (instead of HBM) and IMHO it will take a very big shift before we see them being used as real AI accelerators

I think its important for users to break away from the cloud and APIs, and try to run stuff themself, lest we get locked into an OpenAI monopoly.

But setting that aside, running local is also extremely useful for prototyping and testing. You can see if something works without burning dollars every second you spend debugging on a big cloud instance. Even if that's affordable, just feeling like I am under the clock when debugging/optimizing is stressful to me.

kimixa2• 3 years ago

Looks like you can get an MI60 with 32gb vram for ~$500 on ebay - if you're vram limited might be a good option for less than a 3090 seems to be going for, and seems supported by rocm (being an mi50 just with more hbm).

Not sure how well it will perform as it lacks some of the smaller data type acceleration available on newer cores, though.

superkuh• 3 years ago

AMD has a bad habit of dropping ROCm support for their devices after 4 years. Trying to work with an old dependency chain is something I wouldn't wish on an enemy. I don't know about the MI60 but much of the MI* line has already had support for compute under ROCm dropped.

brucethemoose27• 3 years ago

Its kinda funky, and AMD dropped the MI50 already. See this very interesting reddit discussion:

https://old.reddit.com/r/LocalLLaMA/comments/17vcsf9/somethi...

Another thing that jumps out:

I also have a couple W6800's and they are actually as fast or faster than the MI100s with the same software...

That's insane. The MI100 should be so much faster than the W6800 (a ~6900XT) that its not even funny.

65a2• 3 years ago

I grabbed a 32gb mi60 on a lark, because at the time it was ~$300. It works fine with recent ROCm etc, but it has the following potentially severe drawbacks:

1. It is not a recent GPU, so don't expect blazing inference speeds

2. It is designed for external forced air, so you will need to push air through it somehow

3. The one I got is actually not a normal card, I think it was a firmware testbed, and has no VBIOS and reports its product name as "TBD"

It does work and have 32gb of VRAM though.

wmf• 3 years ago

Apple Silicon

coldcode• 3 years ago

I wonder if we can afford the energy cost of the ever-increasing need to compile more and more information into ever larger AI baskets. Is there a point when the cost exceeds even what a Zuckerberg can pay?

fermuch• 3 years ago

I am hopeful developments like Mistral (a 7B model with the performance of ChatGPT for most of the tasks I've used it for) keeps happening. Seems like we can still compress a lot of knowledge.

ReactiveJelly• 3 years ago

I'm GPU poor. Sick of buying hardware that is not only not fast enough after a year, but not even supported by software that just multiplies and adds numbers in fancy ways.

Debian runs like a champ on my 8 year old CPU. When will gpus last that long?

amelius• 3 years ago

In the early days of computing this was the same situation with CPUs.

However, I do want to complain about the poor tooling around GPUs, particularly package managers, and closedness of drivers. Getting stuff to work on your GPU is often a shitty and time-consuming experience.

vGPU• 3 years ago

I guess it just depends on what you want to do. Training AI? Probably better off renting computer at this point. Everyday use? I still have a 1070 in my main workstation that has survived three computer upgrades. I am looking to upgrade, but I’m not in a hurry.

giobox• 3 years ago

The 1070/1080 are absolute champions of longevity if using a GPU... for graphics/gaming. I still have a gaming machine with my old 1080ti from 2017 in the house, for the most part it will play most of the latest games like Cyberpunk (albeit with no raytracing, but heyho) at 1080p/1440p at good frame rates. Every year that passes now I genuinely marvel at the useful lifespans of those parts, which has been completely unprecedented for me as someone who has been building gaming PCs since the 90s.

The MacBook Pro I bought in the same year is officially classed as "obsolete" by Apple.

calamari40652• 3 years ago

I ran a used GTX570 until it died in 2017. Then a used GTX780 until it died in 2021. Now I'm on a used Radeon 5600. I paid less than $200 for each of them.

I've also been running the same i5-3470 CPU since it was new.

Computers can last a long, long time if you can bear the thought of not being on the bleeding edge. Otherwise, yeah, you're going to be paying a premium for brand new tech. That's the trade you make chasing the newest and shiniest thing.

FirmwareBurner2• 3 years ago

Now I'm on a used Radeon 5600.

Well AMD just ended support for Vega and Polaris architectures so you're shit out of luck there, proving the point of this thread.

And AMD was selling Vega APUs all the way in 2022. So imagine a year later you get the news that the chip in your system is not getting updates because it's architectures is too old. Lol.

I wish I couldn't forget AMD for this.

tedunangst2• 3 years ago

Can I theoretically buy a 4090 and rent it out to some cloud?

petercooper• 3 years ago

Yes - https://vast.ai/ is one option.

tedunangst2• 3 years ago

Interesting, although the host requirements are a bit stringent for a casual. https://cloud.vast.ai/host/setup

Raydovsky• 3 years ago

Huh, if your electricity prices are low enough, it can quite worth it

codetrotter• 3 years ago

I’m not a cloud but I’d be happy to rent a 4090 from you now and then :D

lostmsu2• 3 years ago

We are working on it. If you have Windows Pro machine, see https://borg.games/setup

It is currently tailored to gaming, but ML workloads are coming shortly. Everything runs in an invisible ephemeral VM.

ronsor• 3 years ago

I love the idea, but the nature of running on untrusted user PCs brings some security concerns. If a game is multiplayer/requires a login, it would be trivial for a host to extract those credentials.

darklycan51• 3 years ago

Ever since the 3000 gen of Nvidia GPUs have simply gone out of being a possibility for anyone in the 3rd world, they are way too expensive.

Add in the increase in energy costs... I don't know, really.

brucethemoose27• 3 years ago

Yeah, CPUs too. Power use is unacceptable, desktops and big laptops turbo like mad to achieve really tiny performance boosts.

gumby• 3 years ago

I've been experimenting with using fixed-point weights using the integer vector hardware (AVX) with pretty good results, though haven't built enormous models yet. The GPU may turn out to be a spur line.

baz00• 3 years ago

It only matters if you give a fuck. You don't have to give a fuck to make an impact in the technology market.

The state of silicon and the GPU poors

Related Stories