Show HN: A GPU/VRAM filter for finding LLMs that will run on your hardware

https://www.whichllmmodel.com/app/text?local=true

by mzubairtahir • 27 days ago

2 12 27 days ago

I kept seeing people ask "Which model i can run on my gpu", "will model X fit on my GPU". Thats why I built a filter on whichllmmodel that lets you search models by what will actually fit on your hardware (8GB, 16GB, 24GB, etc.) at a given quantization level.

necovek• 27 days ago

Very broken: "live minimums" do not allow me to remove 512 token limit and put a bigger number easily.

No unified or shared memory scenarios (like Apple's M platform or AMD's integrated GPU platform).

johng• 27 days ago

Was going to mention this. I'm on an M1 Max and wanted to see what the site suggested.

mzubairtahir6• 27 days ago

actually that input is broken. and sorry for that. and I am adding shared memory features in next iterations.

mzubairtahir6• 27 days ago

that broken input is fixed

CRSilkworth• 27 days ago

very nice idea. Would be nice if you could also keep desired context as a free parameter and let the models tell you what maximum context you could have.

mzubairtahir6• 27 days ago

actually that's free by design, it is just broken. fixing it in next sprint. And really thanks for your feedback!!!

mzubairtahir6• 27 days ago

now that is fixed, please try it

xlr8_track2• 27 days ago

Awesome, how do I contribute to this? A gihub link or smthg?

mzubairtahir6• 27 days ago

actually, currently it is not open source, but I am thinking about making it open source so that other developers can also contribute in it(espeically data layer). what do you think?

xlr8_track2• 26 days ago

Yes, many people like myself are willing to contribute. It'll take the load off you and give you time to work on other features or projects.

GreyOcten• 26 days ago

handy, but the gap most of these filters have is that "fits in VRAM" doesn't mean usable. context length blows up the KV cache fast, a 7B that fits at 2k tokens will OOM at 32k. factoring context len + quant into the estimate is where it'd actually save people from getting burned.

mzubairtahir6• 26 days ago

i think you did not check app properly, it is actually taking required context window from the user and then caluclate kv cache size and then count it along with size of model itself. it also reserves some more memory to avoid oom....

Show HN: A GPU/VRAM filter for finding LLMs that will run on your hardware

Related Stories