Benchmarks show even an old Nvidia RTX 3090 is enough to serve LLMs to thousands

https://www.theregister.com/2024/08/23/3090_ai_benchmark/

by mfiguiere • 2 years ago

45 11 2 years ago

metadat2• 2 years ago

How does 12 tokens/sec equate to satisfactorily severing thousands of end-customere?

I did enjoy the headline, and the register has formerly been an incredibly good news outlet. Prior to the founder passing away last week.

atherton33• 2 years ago

From the article, it's not 12 total, but 12 per user for 100 concurrent generations.

metadat2• 2 years ago

Thank you atherton friend.

washadjeffmad• 2 years ago

Capacity doesn't equate to quality, but I could easily see an 8B finetune with exl2 at low context working for short, simple customer interactions, akin to oversubscribing a 1Gbit uplink for 100 customers at 50Mbps.

Cordiali• 2 years ago

"even an old Nvidia RTX 3090"

Saying that like it's mediocre... Maybe I'll have to benchmark my old 1050, see what it can do!

fooblaster• 2 years ago

Guess what Nvidia won't let you deploy in a data center!

stevenhuang• 2 years ago

This is wildly misleading as the benchmarks make use of batching. It will entirely fall apart in real workloads where each prompt is different. If you're doing batch processing with a fixed prompt, the results will be more applicable.

PeterStuer• 2 years ago

It depends. For batching to be viable each prompt has to share some similarities of context/intention, which quite often is the case in specific applications (as opposed to say general chats)

iAkashPaul• 2 years ago

Pretty sure this was never questioned for batched requests, sg-lang/lmdeploy/tensorRT-LLM will have nearly twice as reported speeds with INT8 (fp16 A100 benched here https://github.com/sgl-project/sglang?tab=readme-ov-file#ben...)

Havoc• 2 years ago

Bought a 3090 because they are good value for this, but this logic is frankly a little ridiculous:

Since only a small fraction of users are likely to be making requests at any given moment

So what if 5 out of the thousands happen to coincide?

kristoo• 2 years ago

The benchmark result is for 100 concurrent users

Benchmarks show even an old Nvidia RTX 3090 is enough to serve LLMs to thousands

Related Stories