tarruda

You can't. Not if you're in the minority

Is that a bad thing?

It would be fundamentally wrong for you to ask what's the value of solving the Erdős–Hajnal conjecture; the value is that it's solved.

I suspect the value is in showing the potential that LLMs have in developing new breakthroughs.

Assuming steady 1 tok/second generation (which seems to be the case for M5 Max macbook), wait 1 day for a 86400 token response. In some configurations it can be as slow as 0.1 tok/s, so be prepared to wait for 10 days.

Recently tried the pelican test on GPT-OSS which was probably one of the best local models of 2025. So cool to see how models have improved in the SVG pelican!

given they are pretty close in size

One thing that might not be obvious about about DSV4 is how much innovation the Deepseek team implemented in its architecture. When llama.cpp fully supports its lightning indexer, the full 1M context will only require about 6G of RAM. So even though they are similar in size, I believe Deepseek will be much more efficient in that regard.

I wonder if Hy3 can compete there

Highly depends on how well Hy3 is resilient to quantization. DSV4 is useful even at 2-bit quants.

Hopefully this distillation will lead Alibaba to release more powerful open weights LLMs, contributing to the democratization of AI.

Vibe thinker also beats Opus 4.5

If your framework desktop is the 128G Strix Halo, I recommend giving Qwen 3.5 122B-A10B a shot.

This Q5_K_M quant should be near lossless and fit with full 256K context in about 100GB of RAM: https://huggingface.co/AesSedai/Qwen3.5-122B-A10B-GGUF

I don't feel like AI coding has ruined my skills, and I could go back to manual coding any time.

However, I cannot build a good mental model of a software component that I didn't write myself, and that can affect future maintenance if that component is not properly decoupled from the rest of the system.

What I find fascinating is the idea that there might be a set of "secret" tweaks that when applied to those weights (or even smaller models) could result in an intelligence simulation that could vastly surpass even something like Fable.

Not as much as Qwen, since apparently 3.6 35B surpassed Opus 4.7 https://x.com/simonw/status/2044830134885306701

I don't think there's any incentive for Nvidia to make this a Windows-only device, so most likely it will be fully supported on Linux, just like their GPUs are.

This also means that, according to our plans, Zig will have to propagate "stackless-ness" upwards in the call chain while analyzing the code (thus making Future.await not special per-se).

Very interesting, will be following Zig development more closely. Thanks for sharing!

there's an accepted proposal to bring them back, in which case any function that calls await, or that otherwise has a suspension point, would have to be transformed into a stackless coroutine by the compiler, yes. The plan is for that to happen transparently without requiring an `async` annotation in the function signature, like we already did in the past.

If the compiler will treat functions that call a library function (Future.await) as special and change how the call site is compiled, why not just have an `await` keyword that when present will convert it into a state machine that can be suspended/resumed?

In other words: What is gained by not having a keyword that changes how a function is emitted if the compiler will change it anyway based on detection of a library call?

Fixed it. thanks!

especially with the new IO mechanism which allows supper efficient code that looks good whether it's implemented single-threaded, multi-threaded or just via an event loop!

I had some trouble understanding how the async/await mechanism works:

  var foo_future = io.async(foo, .{args});
  defer if (foo_future.cancel(io)) |resource| resource.deinit() else |_| {}

  var bar_future = io.async(bar, .{args});
  defer if (bar_future.cancel(io)) |resource| resource.deinit() else |_| {}

  const foo_result = try foo_future.await(io);
  const bar_result = try bar_future.await(io);

The part that I don't understand is what happens when you call future.await(io). Will the IO implementation somehow suspend the current function and resume once the future is resolved? If so, does that mean that every function in zig is a stackless coroutine?

The official Q4_K_S gguf is quite good and has very good 35 tps generation on a M1 mac studio. Should be much faster on recent Macs, especially M5.

One of the most prominent improvements in Opus 4.8 is its honesty.

Does that mean it no longer deletes or changes tests to make it pass?

safer bet as a dependency.

The recent 1 million line vibe coded PR suggests it is not so reliable as a dependency.

That's impressive getting a 397B down to <110GB

It is higher than 110GB. MacOS allows up to 125G of the RAM to be shared with GPU, so it is certainly less than that!

HF link is broken though!

Doesn't seem broken to me, but you should be able to search for tarruda/Qwen3.5-397B-A17B-GGUF on huggingface.

I'm questioning ROI

If by ROI you mean saving more money than using paid APIs, then I don't think it is worth it. All you gain is full sovereignty over your AI usage.

300B models at least fit in a single maxed out Mac Studio or a small stack of DGX Sparks or AMD Strix Halo boxes.

I run 2.54 BPW 397B Qwen 3.5 GGUF on a 128G mac studio at 20 tokens/second generation and 200 tokens/second processing. I'm not suggesting it matches the performance of the full BF16 model, but I did run some benchmarks locally and the results were pretty good:

- MMLU: 87.96%

- GPQA diamond: 86.36%

- IfEval: 91.13%

- GSM8k: 92.57%

So I think we have been at the "frontier capabilities at home" for a few months now.

I only tried a very early version of that when it was just a llama.cpp fork and Qwen was certainly better in my tests.

But I was not super impressed with deepseek 4 flash using it from the official API either, so it doesn't seem quantization fault. It is a good model, but nothing out of the ordinary in the few benchmarks I ran on it (with full awareness that benchmarks are biased).

What’s the price point for getting into that sweet spot?

In October/2024 I got my Mac studio M1 ultra with 128G, IIRC it was ~$2500. With recent prices explosion, it has certainly gotten more expensive. https://frame.work/ is selling 128G strix halo mainboard for $2700, but you have to add storage and case.

I have a 128G mac studio and even 397B was a happy surprise to me due to its high quantization resilience.

I've created a 2.54BPW quant that fit on my hardware with 128k context, 20 tps tg and 200tps pp, while maintaining high scores on many benchmarks: https://huggingface.co/tarruda/Qwen3.5-397B-A17B-GGUF/discus...

Looking forward to more open weight releases from Qwen, especially 122B and 397B.

And as long as Bun doesn't break Claude code, which only uses a subset of it's APIs, this might just pay out.

I started looking at the commits, and it's basically solving the ,,tests not pass'' problem by changing the tests themselves

Not sure if these decisions were made by the LLM, but I've always felt that Claude is more prone to doing "shady stuff" like modifying tests than finding correct solutions to problems.

GPT/Codex is more honest in this regard.

Mailbox.org (also from Germany) seems to be experiencing issues too.

They also published draft models for E4B and E2B. For those, the draft models are only 78m parameters: https://huggingface.co/google/gemma-4-E4B-it-assistant