sosodev

Realistically you can't prevent distillation. OpenAI / Anthropic are slowly moving towards hiding the steps in-between input and output (hidden thinking), but that only helps so much. Imagine you put a file into Claude and say "do X to this" and it returns it to you without showing any of its internal reasoning. That's harder to distill, but the simple mapping of input to output still creates very valuable training data. It is reflective of all the training the model did to learn how to do that transformation.

Model distillation can't be stealing at all if you rationally apply copyright law to it. Anthropic is not deprived of Fable so there is no theft. At best it would be infringement, but even that might not hold up in the courts given the current position that model outputs can't be subject to copyright.

Distillation is a very vague term. It can mean anything from training exclusively on a model's output to using it for a very small portion of the training. In this case it is almost certainly towards the very small portion side of the spectrum.

A month seems plenty long enough. They're not rebuilding the entire model from scratch. It's just getting Fable to act as a teacher model for some of the final reinforcement learning on the base that Kimi already had.

When? It literally says on the page for Gemini 3.6 Flash "Artificial Analysis Coding Index represents the weighted average of coding benchmarks in the Artificial Analysis Intelligence Index (Terminal-Bench v2.1, SciCode)"

What inference server are you using? They have a custom branch for llama.cpp, but I wouldn't be surprised at all if it still needs fixing.

Because AA Coding "Index" consists only of two benchmarks (Terminal-Bench v2.1, SciCode) and generally fails to be meaningfully representative of agentic coding capabilities.

That’s the worst thing about sycophancy. You can never tell when it’s warranted or not. I imagine that many AI chats have had real gold in them and yet we’ll never know.

The same was obviously true historically. People have had many revolutionary thoughts or even writings that nobody noticed. I think in this case it just feels worse because the AI could be the perfect tool for discerning the diamonds in the rough.

Of course efficiency matters, but a lot of people either have cheap electricity or efficient hardware. My AMD strix halo home server can serve Gemma4-26B at like 70 TPS (rough estimate, I don’t remember the exact speed buts its fast af) while only using 100W.

They’re exaggerating or have a very simple way of using these models. The Gemma 4 series, even at 31B, is nowhere near the frontier. They’re great models, but you will notice a huge difference for complex tasks.

The best local agentic coding experience I’ve had so far is Qwen3.6-27B with Pi.

Qwen3.6-27B is the best model in that range that I’ve used for agentic coding by far. I think it’s kinda mid at everything else.

The Gemma models are so good at vision. It seems particularly important for phones. Also, they write in a much more pleasant manner than Qwen imo.

I think the difference is asking what happens if all contexts collapse. In the 2000s or 1960s context collapse was something that would happen occasionally when opted into. These days everybody is hyper connected and living in a collapsed context every day. That's the uni-context and the implications of it feel very different than that of a wedding or old-school forum. A wedding wouldn't push people to permanently change who they are or how they perceive themselves for more favor in the collapsed context. They would only do it momentarily.

Also, the wedding or forum had very limited scope. As mentioned in the interview, the uni-context is about context collapse on a global scale.

The thing we must keep in mind with any of the AI-slop style writing is that it was reinforced behavior because humans wanted it.

I suspect it would depend on the task. DS4-flash does, as previously mentioned, handle quantization very well. Even at 2-bit it's still very coherent.

It's almost as if HN users aren't all the same.

He got his name in the credits. The question was if he is owed anything else. The contract he created says he was not. I’m simply suggesting he might need a different contract.

That’s a fine perspective, but the whole point of law is to guarantee outcomes. The license could easily say “if you make more than $500M, you must pay me $1M”. Why is that not an acceptable solution here?

I find it odd how we frame fairness in regards to open source software. He licensed his software as MIT. It says anyone can you use it without owing the author anything. So how is it unfair?

To be clear, I think that open source maintainers deserve much more, but I don't understand why we rarely inspect the licenses as the source of the problem.

In my experience, even with basic project concepts the small models struggle to spin up greenfield stuff. There's just too many decisions to be made and they're not good at that.

Modifying existing code is way easier if you don't expect it to be smart about it. Don't say "add X feature" and let it explore the codebase and build its own understanding. Point it at the relevant files and say "the goal is to add X feature to this code, follow Y guidelines". Now you've done the hardest part of making the decisions and it just has to follow instructions while coloring within the lines.

The license scheme makes me sad. It reads like a subscription pretending to be a one-time purchase.

I think it really just depends on your goals. Slow tokens per second is fine by some people if they cost a fraction of a single node setup that can run a trillion param model. If you’re actually running a small business and want to have multiple users getting a good experience in parallel then yeah I think you need a single node. At that point you can afford it I suppose.

I don’t know what the scaling for multiple strix halo boards looks like in practice. From what I understand each server has to process the model in serial. Meaning server A has 1/4 the weights and sends server B the results to process and so on. So you don’t get compute scaling just memory scaling.

I read the comment, thanks. I just disagree with your cost estimate. Even for a small business that needs high throughput they could probably do it for far less than $300k if they aren’t just blindly buying the first big nvidia setup they can.

I don’t have any particular model in mind, sorry. My data is just rough estimates based on my experience with a single node setup. You might need to opt for a 2 or 3 bit model to get the full context window. The KV cache memory consumption as well overall performance will be heavily dependent on the model’s architecture. The performance too will depend a lot on the inference server chosen and its configuration. I suspect a sub-agent running a much smaller model would be the ideal way to get the latest knowledge via web search and summarization.

I’m not trying to say that this would be a great experience or really compete with just buying a subscription to the top models. Rather I just wanted to point out that $300k is an absurd estimate for a trillion param model meant for personal use.

It depends entirely on what you want to do and think is feasible. Small models can almost certainly run on the computer that you already have. They can do good tool calling.

You can run a trillion parameter model with decent quality for far less than $300k. A cluster of 4 AMD AI Max 395+ boards with 128GB unified memory each can be had for around $15k. That would run the 4-bit quant of a trillion param model well enough for personal use. At full use the cluster would only be consuming around 400-500W of power too. That's about the same as one high end graphics card.

That's still a lot of money, but most people don't really need a trillion parameter model. If privacy is more valuable than the frontier capabilities then they could almost certainly get by with much less.

They have it, we just haven’t enabled them. The smart model with a chat box is the wrong abstraction for local. Ideally we would have it built into applications as a clear and easy to use opt-in feature. Like allowing a user to index a folder on their hard drive and then search it semantically via embeddings. You could do that on fairly low end hardware these days. Like 2GB of RAM with any processor made within the last 10 years.

Note that AA's coding index is only made up of two benchmarks: Terminal-Bench Hard and SciCode. I'm skeptical that it makes a good coding index. It ranks Gemma 4 31B above Deepseek V4 Flash. Having used both of those models for a broad variety of coding tasks I would choose Deepseek every day.

Petsitter's default tricks doesn't seem to do much for Qwen3.6, right? JSON mode could be useful I suppose, but that's not really going to make it better at writing code. Do you have any other example tricks? I'm having a hard time understanding how I would apply them.

Meh. My server can run these models for neglible power draw (like ~130W fully maxed out). That's with ~30 tok/s which isn't that bad. I do agree that they're still nowhere near as good as the frontier models though. I do lean on those when I need to get something done with better quality or at a faster speed.

I've also been using Deepseek V4 pro/flash for some work stuff and I do find them to be much closer to frontier capability. I may try running flash at home soon for very patient edits. :)