HN user
wanna chime in, recently tried vLLM to consume a NVFP4 Gemma4 safetensor model and see how the batching can show up in nice t/s numbers. it's slow to start, it's Linux only, it doesn't like WSL much, ended up with either old or nightly container builds, I more or less have given up. Appreciate how llama.cpp simply works and does things fast and obvious
my latest experiments with local LLM (mistral coder variations) fitting in older 6 GB GTX1060 were disappointing as long as you try to hook Copilot (CLI or VScode) to it and are used to provide a lot tooling. this seems to bloat initial prompt to 20k and more which seems the bottleneck if I did not completely misconfigured things. output tokens/s are more than fine, but PP is frustrating / unusable.
both 5.4 were best bang (-mini even more as I found it usually same well performing!!!) for the buck before. Now they face cost reality it seems.
e.g. if on an annual plan? 0x will be gone, but there are okay 1x and 0.3x models left. I am pretty much curious how the early may test invoicing will look like. current setup of tools etc. is way too chatty eats up 1+M token per PRU easily. not sure how much is cached.
not sure how much resources it took from my WinARM Win11 as store app, but it's buggy in either release or beta since months as it loses network connectivity after some time...
I think it's more Dave that "sucks"...click baiting titles, no need to and ever and ever repeating that he was at MS. He should do Podcasts as there is no value in videos only speaking (feels like 1.5x btw)
Stackoverflow usage down to 10%
at least the naming implies factor ~20 less :)
this paper is going through mass media meanwhile as it alledgely shows how Skynet started to take over...
found my glasses and there is a candidate AVX optimized to 1 GB/s
do I need glasses, or not a single c#/.net implementation?
nice one:
https://www.youtube.com/watch?v=ejA0wKTRHH4
Presentation shown to new Mercedes salsman in the late 1980s and early 90s. Covers lots of sales tips and techniques to help put the customer in the driver's seat of a new Mercedes.
great!