kpw94

Yeah definitely. I've recently commented on that: https://news.ycombinator.com/item?id=48557890

In the context of local LLMs on limited hardware I've ran to the exact same conclusion: "tok/s" isn't the most useful metric when my personal North star metric, given my fixed hardware is: Model smart enough to execute my goals _in the minimum amount of time_.

Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that they generated many more tokens compared to a Qwen model of same caliber to answer the same thing.

So even though it had better generated tok/s, because so many more were generated, the clock time was longer.

And this compounds over mutli-turns: more generated token means more context used in the next turn (until some compaction or something runs)

What it does:

--jinja for tool calling support

Pretty sure this flag hasn't done anything for a while. It's enabled by default since ~November of last year

The huge spike of "lk-99" in science & frontier tech is amusing...

This is cool concept, would love a positive/negative sentiment computed for each comment that refers to a given word, so you can see trends of "cloudflare (positive)" vs "cloudflare (negative)" where first one counts comments only if sentiment confidence is greater than say 0.6 and the other one counts comments only if sentiment is less than 0.4 (assuming [0,1] sentiment score)

Thanks! Super helpful.

I do use it the same way as you're describing on personal projects at home, in a very crude manner (pasting code snippets in llama server web UI prompt. Next will attempt OpenCode)

At work I use it in similar manner with more mature tools, but the vast majority of token spend comes from a totally different workflow: "pretend the AI is a fleet of junior/intern engineer you're delegating work to", where the agent will on its own do the implementation, commit the changes etc.

It does indeed spend a lot of tokens wandering the codebase, talking to MCPs, loading skills etc.

About the generation speed: ~100-150 t/s on the RTX 5090 and ~40 t/s on the Mac

Curious if you can share the prefill speed too?

I run locally on a crappy desktop (some AMD iGPU with Vulkan llama.cpp, 32 GB DDR4 RAM) for experimentation. I get 15 tok/s on generation for the qwen & gemma4 MoE models. I get around 150 tok/s prefill speed.

Reason I'm asking about the prefill is looking at my stats at work, I use between 20M to peaks of 300M input tokens daily. Some of those token are cached but in general, I seem to have roughly 500x more input tokens than output. So interested in prefill tok/s stats.

Huge Thank you for llama.cpp btw!!

gemma (unsloth/gemma-4-26B-A4B-it-GGUF) models

Since you're running quantized (at UD-Q4_K_XL) , check out the "qat" models (unsloth/gemma-4-26B-A4B-it-qat-GGUF) !

- https://huggingface.co/unsloth/gemma-4-26B-A4B-it-qat-GGUF (With "Jun 9 Update: Added MTP support.")

- https://blog.google/innovation-and-ai/technology/developers-...

I did the opposite switch:

In ~2015 got an Xbox one, as a media center it was an awesome experience:

Kinect voice control to play/pause and other things way before Google home/Amazon echo ecosystem were mature.

Free OTA channels via TV tuner and well designed OneGuide (with ability to pause and rewind).

And of course all the Netflix and other apps, Plex server etc.

But strategically it seems Microsoft decided they wanted to look more like Playstation, focused on gaming (at that time paid Xbox live subscription vs free Playstation)

And as gaws points out, they seem to recently announce to double down on the gaming stuff.

So when they discontinued OneGuide. I picked Roku since they seem to be focused on the media experience primarily... but unsure how I feel about this acquisition news.

What's hard to figure out here?

Negative externalities are hard to figure out.

Since parent mentions "toxic byproduct": Say you're the company that invented Teflon pans. you made billions. You saved billions in time for all the users of the pans... A true entrepreneurial success.

But, by how many billions did you fuck up the environment, people's health etc with the spread of PFAS everywhere?

That seems a very risky assumption for any car (self driving or human driver) during flash floods. "Turn around don't drown":

You think you know how deep it is under because you've taken that road many times before (or in your case you have historical laser measurement)

But you don't know:

- Maybe the road under fully collapsed

- Maybe the flow of water is extremely strong, so you need to accurately estimate that too.

And the author is correct (while the phrasing is a bit weird.)

Right, that's just a description of the https://en.wikipedia.org/wiki/Baumol_effect

Speculative execution techniques in software & hardware exist everywhere,

- Speculative multi threading

- Data Value Speculation

- Speculative Memory Disambiguation

- Runahead Execution

- Speculative Prefetching

- Multi-path (Dual-path) Execution (goes beyond branch prediction by computing both paths)

- Optimistic Concurrency Control (for database transactions etc)

My non-controversial theory: It's all the attention-span-shortening stuff.

- tech apps starting with infinite scroll (facebook, 9gag, Instagram, etc.)

- media/tech shortened content: shorter tv shows, short video content, etc.

(Tiktok is the "state of the art" of those 2 trends pushed to the max)

Specifically, we're getting more & more addicted to things that increase the dopamine spikes frequency, making it increasingly difficult to go in deep focus work.

But isn't the prefill speed the bottleneck in some systems* ?

Sure it's order of magnitude faster (10x on Apple Metal?) but there's also order of magnitude more tokens to process, especially for tasks involving summarization of some sort.

But point taken that the parent numbers are probably decode

* Specifically, Mac metal, which is what parent numbers are about

When you say tok/s here are you describing the prefill (prompt eval) token/s or the output generation tok/s?

(Btw I believe the "--jinja" flag is by default true since sometime late 2025, so not needed anymore)

Right, they're not the only FAANG company for which we know they're doing it: https://news.ycombinator.com/item?id=46318494

Some might be tempted to brush aside that Server Linux threat model is very different from Desktop Linux (to snarkily reply "we'll it's powering a vast majority of GDP via all of AWS, Azure, etc.").

However comparing apples to apples, what makes you say this isn't ready for government usage, when it's ready for trillion dollar big tech companies' majority of their workforce? (Aside from Microsoft, Apple obviously). Large employers like IBM etc also must be using red hat or some other distro

I don't know how to force this issue as a European. There are just too many levels of abstraction between me and Brussels.

EU moves so much faster when it comes to regulations like forcing all of us in Denmark to use timesheets, annoying lids on our bottles, and invasive surveillance laws.

Rediscovering the principle of subsidiarity from first principles...

I'll need to investigate further but it doesn't seem promising.

That's what I meant by "waiting a few days for updates" in my other comment. Qwen 3.5 release, I remember a lot of complaints about: "tool calling isn't working properly" etc.

That was fixed shortly after: there was some template parsing work in llama.cpp. and unsloth pulled out some models and brought back better one for improving something else I can't quite remember, better done Quantization or something...

coder543 pointed out the same is happening regarding tool calling with gemma4: https://news.ycombinator.com/item?id=47619261

Wild differences in ELO compared to tfa's graph: https://storage.googleapis.com/gdm-deepmind-com-prod-public/...

(Comparing Q3.5-27B to G4 26B A4B and G4 31B specifically)

I'd assume Q3.5-35B-A3B would performe worse than the Q3.5 deep 27B model, but the cards you pasted above, somehow show that for ELO and TAU2 it's the other way around...

Very impressed by unsloth's team releasing the GGUF so quickly, if that's like the qwen 3.5, I'll wait a few more days in case they make a major update.

Overall great news if it's at parity or slightly better than Qwen 3.5 open weights, hope to see both of these evolve in the sub-32GB-RAM space. Disappointed in Mistral/Ministral being so far behind these US & Chinese models

That's not using tech that you're describing here. You're talking about literally learning some basic computer skills (such as word processor, excel, reading email, some basic website building, use printer, and some amount of programming)

For those, obviously you need a computer and completely agree that those are important skills to learn... But you maybe need to spend 1h/week during last 2 years of middle school on those at the computer lab (as it's been done since the 90s in many schools around the world)

But for any other course such as Math, English (or whichever primary language in your country), second languages, history, etc. : that's where using tech is a mistake

A bit of tech is ok, but it cannot be "everyone does their homework and read lesson on a iPad/Chromebook"

The options from big companies to run untrusted open source code are:

1) a-la-Google: Build everything from source. The source is mirrored copied over from public repo. (Audit/trust the source every time)

2) only allow imports from a company managed mirror. All imported packages needs to be signed in some way.

Here only (1) would be safe. (2) would only be safe if it's not updating the dependencies too aggressively and/or internal automated or manual scanning on version bumps would catch the issue .

For small shops & individuals: kind of out of luck, best mitigation is to pin/lock dependencies and wait long enough for hopefully folks like Fibonar to catch the attack...

Bazel would be one way to let you do (1), but realistically if you don't have the bandwidth to build everything from source, you'd rely on external sources with rules_jvm_external or locked to a specific pip version rules_pyhton, so if the specific packages you depend on are affected, you're out of luck.

The Autoboxing example imo is a case of "Java isn't so fast". Why can't this be optimized behind the scenes by the compiler ?

Rest of advice is great: things compilers can't really catch but a good code reviewer should point out.

People complaining about how hard to get simple answer is don't appreciate the complexity in figuring out optimal models...

There's so many knobs to tweak, it's a non trivial problem

- Average/median length of your Prompts

- prompt eval speed (tok/s)

- token generation speed (tok/s)

- Image/media encoding speed for vision tasks

- Total amount of RAM

- Max bandwidth of ram (ddr4, ddr5, etc.?)

- Total amount of VRAM

- "-ngl" (amount of layers offloaded to GPU)

- Context size needed (you may need sub 16k for OCR tasks for instance)

- Size of billion parameters

- Size of active billion parameters for MoE

- Acceptable level of Perplexity for your use case(s)

- How aggressive Quantization you're willing to accept (to maintain low enough perplexity)

- even finer grain knobs: temperature, penalties etc.

Also, Tok/s as a metric isn't enough then because there's:

- thinking vs non-thinking: which mode do you need?

- models that are much more "chatty" than others in the same area (i remember testing few models that max out my modest desktop specs, qwen 2.5 non-thinking was so much faster than equivalent ministral non-thinking even though they had equivalent tok/s... Qwen would respond to the point quickly)

At the end, final questions are: are you satisfied with how long getting an answer took? and was the answer good enough?

The same exercise with paid APIs exists too, obviously less knobs but depending on your use case, there's still differences between providers and models. You can abstract away a lot of the knobs , just add "are you satisfied with how much it cost" on top of the other 2 questions

That's a big flaw of LLMs, not limited to RAGs: it lacks the fundamental understanding of "good and bad", like Richard Sutton said in that Dwarkesh podcast.

So if you flood the Internet with "of course the moon landing didn't happen" or "of course the earth is flat" or "of course <latest 'scientific fact' lacking verifiable, definitive proof> is true", you then get a model that's repeating you the same lies.

This makes the input data curating extremely important, but also it remains an unsolved problem for topics where's there's no consensus

"water is wet" kind of study, as tariffs are precisely supposed to increase price for consumers for imported goods... But the last 3 paragraphs are interesting:

- Importers raised the price more than needed (i.e. blame tarifs to increase their profit margin)

- Price increases took one year to fully reflect to the customers, and persisted nearly one year after the tariffs expired.

- chicken-tax-like loopholes implemented wherever possible (for wine apparently it's raising the ABV to more than 14%)

On my 32GB Ryzen desktop (recently upgraded from 16GB before the RAM prices went up another +40%), did the same setup of llama.cpp (with Vulkan extra steps) and also converged on Qwen3-Coder-30B-A3B-Instruct (also Q4_K_M quantization)

On the model choice: I've tried latest gemma, ministral, and a bunch of others. But qwen was definitely the most impressive (and much faster inference thanks to MoE architecture), so can't wait to try Qwen3.5-35B-A3B if it fits.

I've no clue about which quantization to pick though ... I picked Q4_K_M at random, was your choice of quantization more educated?

Basically, I was told to make it so that my phone's camera could see something on the screen and my desk at the same time without washing out

+1. The low-tech version of this I've heard and I've been doing is:

Hold a printed white paper sheet right next to your monitor, and adjust the amount of brightness in monitor so the monitor matches that sheet.

This of course requires good overall room lightning where the printed paper would be pleasant to read in first place, whether it's daytime or evening/night

Per https://github.com/QwenLM/Qwen3.5, more are coming:

News

2026-02-16: More sizes are coming & Happy Chinese New Year!

However Germany and it's infrastructure can not be compared to the Netherlands. I refuse to take trains through that country anymore.

In which country are the trains bad? Netherlands or Germany? Do you care elaborating why? is that punctuality? strikes? decaying infrastructure?