Ask HN: What local machines are people using to train LLMs?

https://news.ycombinator.com/item?id=39028370

by Exorust • 3 years ago

56 24 3 years ago

news.ycombinator.com

How are people building local rigs to train LLMs?

bearjaws3• 3 years ago

I doubt many people are using local setups for serious work.

Even fine tuning Mixtral is 4xH100 for 4 days. Which is a ~$200k server currently.

To fully train, not just fine tune a small model, say Llama 2 7b you need over 128GiB of vram, so still multiple GPU territory, likely A100s or H100s.

This is all dependent upon the settings you use, increase the batch size and you will see even more memory utilization.

I believe a lot of people see these models running locally and assume training is similar, but it isn't.

YetAnotherNick2• 3 years ago

With qlora, 7b models could be fitted in 24 GB VRAM. And I have seen few folks doing it locally in discord servers I am part of.

bearjaws3• 3 years ago

For inference and fine tuning yes, not training.

malux853• 3 years ago

I don’t train LLMs from scratch, but I have:

3x4090s 1xTesla A100

Lots of fine tuning, attention visualisation, evaluation of embeddings and different embedding generation methods, not just LLMs though I use them a lot for deep nets of many kinds

Both for my day job (hedge fund) and my hobby project https://atomictessellator.com

It’s summer here in NZ and I have these in servers mounted in a freestanding server rack beside my desk, and it is very hot in here XD

2099miles• 3 years ago

Can you talk about the attention visualization a bit. What’s the benefit. I have been plotting attention masks and even visualizing them over sentences but it obv doesn’t result in a ton of insight due to the sentences changing from words to much more complex vectors that represent those words after being processed through attentions 32x.

malux853• 3 years ago

I am using attention weighted pooling strategy to generate cross-attentions across N transformer heads, this allows us to visualize which parts of an output were caused by which part of the input -- this is used to debug summarization. Only works for encoder/decoder networks - i.e. ones where you have cross attentions

bavell• 3 years ago

Very cool website, love the aesthetic! Would love to know how you made the animations and the algos behind them.

malux853• 3 years ago

Thank you! The animation at the top is a GPU shader of the transition between 1s, 2p and 3d electron wave functions in a hydrogen atom, the animated crystal of sulpher I made using Maya and then animated into/out of wireframe by loading it and spinning it in WebGL, the other ones further down are just screen recordings.

You might like this too - this last weekend I finished writing a topological contour GPU shader for visualizing electron density grids, here is a molecule of buckminsterfullerene - the contour generator is running on the GPU, which is how I can keep the framerate so high.

https://drive.google.com/file/d/1xZkXsWDZtpe0H3BKSkdzvjOA3UE...

rgbrgb2• 3 years ago

Some people have been fine-tuning mistral 7B and phi-2 on their high-end macs. Unified memory is a hell of a thing. The resulting model here is not spectacular but as a proof of concept it's pretty exciting what you get in 3.5 hours on a consumer machine.

- Apple M2 Max 64GB shared RAM

- Apple Metal (GPU), 8 threads

- 1152 iterations (3 epochs), batch size 6, trained over 3 hours 24 minutes

https://www.reddit.com/r/LocalLLaMA/comments/18ujt0n/using_g...

justanotheratom• 3 years ago

Which one would you recommend between these two for fine-tuning OSS LLMs?

M2 Ultra Mac Studio vs M3 Max Macbook Pro

https://www.apple.com/shop/buy-mac/mac-studio https://www.apple.com/shop/buy-mac/macbook-pro/16-inch

petercooper• 3 years ago

There's little between the M2 and M3 at this point with regards to LLM work, and given no M3 Ultra exists yet, the M2 Ultra would be the way to go considering this task in isolation, and indeed the only option if you wanted to go up to 192GB RAM. If you needed to be portable or had budgetary concerns, then your choice may differ.

myself248• 3 years ago

Unified memory is a hell of a thing.

Does that also apply to stuff like AMD APUs? Honestly asking, I have no clue.

jeswin• 3 years ago

No, they have far less bandwidth. However, I am not sure if getting Apple CPUs would be a good idea. You can get some toy projects running with them, but a pay-as-you-go cloud solution would have been cheaper and faster. I don't find my M1 Max to be particularly competitive.

kube-system• 3 years ago

AMD and Intel allow their customers to choose lots of different memory configurations and so they prioritize compatibility over bandwidth. Apple's architecture has no support for external memory, they put their memory directly on package and can prioritize for bandwidth.

ProllyInfamous2• 3 years ago

What can a bone-stock (16GB) M2Pro Mini do? Would it be correct to assume this takes 8x longer (on account of 1/4 memory and 1/2 GPU), or is training not even possible?

rgbrgb2• 3 years ago

i'm not sure, but I bet you could get an answer to a more specific hardware question like this over at /r/locallama

bearjaws3• 3 years ago

Watch out for the memory bandwidth limit on M3, you have to buy the top of the line 16c CPU to get the same memory speed as an M2 Pro.

buildbot2• 3 years ago

A self built machine with dual 4090s, soon to be 3x. Watercooled for quieter operation.

Did the math on how much using runpod per day would be, and bought this setup instead.

Using Fully sharded data parallel and bfloat16, I can train a 7b param model very slowly. But that’s fine for only going 2000 steps!

mrkeen• 3 years ago

Watercooled for quieter operation.

How is that these days? I water-cooled my system with the intention of having a quiet system sometime around 2008, but it just used liquid to transfer heat to a sink further away, which had a noisy fan.

godelski• 3 years ago

Much better these days and easier to do. Fans are aso much quieter these days too. A few years ago I was twisted into building a water cooled GPU system because it was the only 30 series card I could get in a 6 month period. Wasn't too hard to build into a system that doesn't even have enough room for the water pump to stand vertically. Soft tubing is annoying though and will fatigue with time, restricting flow rates. So I suggest if you do that to take more time to ensure you have angled fittings where you need them and to zip tie straight paths where you can (and thicker tubing is better). Might just be easier to go hardline instead tbh.

Thermals are great though. But tbh now what I do is put my machine behind my TV and I have the side open and place the pump outside to stand vertically like old systems that had them outside the case lol. That's because if I'm not playing games on it then there's no difference in just sshing into the system because I live in the terminal anyways.

buildbot2• 3 years ago

Pretty straightforward these days! Though I've been doing it for awhile. There a lot of little gotcha's like tubing choice and fittings but once you get the hang of the options it's pretty fun.

Though cooling 4090s even with water is hard - you need a good flow rate to dissipate 450W. They'll hit near 70C during training if the fans are kept below audible, which works for me. 55-60C with fans at 50%. This is with 2x420mm (3x 140mm fan) rads and one 280mm rad (2x 140mm fans).

The stock heatsinks actually were actually able to cool them just as well, just they where much louder doing so!

Power is the other tricky bit. The computer is living on the dedicated AC circuit for now... 1600W is a lot. When it was on the single outlet circuit in my house, I could see each iteration as the lights dimmed in sync with the power draw surging.

Ask HN: What local machines are people using to train LLMs?

Related Stories