Running Llama3 Locally

https://news.ycombinator.com/item?id=40096199

by parthi2929 • 2 years ago

1 2 2 years ago

news.ycombinator.com

I am newbie to AI, want to run local LLMs, greedy to try LLama 3, but my old laptop is 8 GB RAM, I think in built Intel GPU.

What would be system requirement to comfortably run Llama 3 with decent 20 to 30 tokes per second at least? RAM, GPU, CPU, motherboard?

A formula if could be derived can help a lot (so I could plugin 7B or 70B LLM size, and out it comes, the sys requirements)..

Thank you in advance for your kind help...

kashaudhan• 2 years ago

Follow this: https://ollama.com/library/llama3

slimesli• 2 years ago

for llama 3-8b (smaller model) you need 8GB/16GB of memory: minimum/recommended || for llama 3-70b (larger model) you need 64GB/96GB of memory

what kind of memory? there are 3 types of memory: GPU memory: medium capacity | high speed || Apple Unified memory: high capacity | high speed || CPU memory: high capacity | low speed ||

rtx 3090 has 24GB of memory at 935GB/s memory bandwidth || rtx 4060ti has 16GB of memory at 288GB/s memory bandwidth ||

m2 mac mini has 8GB/16GB of memory with 100GB/s memory bandwidth || m3 macbook pro has 32-128GB of memory with 400GB/s memory bandwidth ||

new x86 consumer cpu have up to 256GB of DDR5 at 100GB/s memory bandwidth || old cpu with ddr4 and ddr3 are much slower ||

Remember, there is a simple formula for tokens/s: you need to divide memory bandwidth by model size, so for example:

mac mini runs llama3-8b at 18.8 tps using 5GB of memory 100GB/s/5GB = 20 https://twitter.com/awnihannun/status/1781345824611680596

rtx 3090ti runs llama2-7b at 179 tps also using around 5GB 1000GB/s/5gb = 200 https://github.com/turboderp/exllamav2

Running Llama3 Locally

Related Stories