liuliu

training an LLM takes more resources and expertise than distilling from an existing LLM

This is not automatically true. Training and distillation use the same underlying infra and method and there is no intrinsic differences in between.

It is disabled because it doesn't work :) Try it and see the doom loop it gets itself in.

Note that 3.5 9B cannot do thinking (while 3.6 27B can, pretty effectively, quite verbosely).

You also need to pay close attention to BFCLv3 multi-turn result, that helps you to get a sense how frequently these quants will be in a doom loop.

The problem, of course, is if you run the UD_Q2 variant (Unsloth) which does only post-training, the number is pretty close to 1-bit model here and the 5% drop in tool-call is significant than it suggests in real-life use cases.

This should be the first comment here.

Too many replies here are done before reading it. It is not "just another agent does the agent thing". It is a deliberate choice of the Grok Build team to have a toggle from the server to let the program to upload your entire codebase to a Google Cloud Storage bucket. It is not an agent decision, the program is written by the Grok team, can be dissembled and seeing the logic wild-open.

Realistically, you need to experiment with any user prompt + a good amount of system prompt (at least > 1000 tokens, but realistically, in the range of 3000 tokens probably good).

llama.cpp includes tools for that, what you are looking at is to have a prefill before token generation to measure it properly. Increasingly also, measuring token generation speed at longer context (32k or 64k) is important too.

It is a “research note”. It might not pan out, and you might say it doesn’t deserve the attention on the internet. But it did suggest something that resembles of compression, just no experiment done for that.

I actually agree. At some point, a RSI system has to interact with real-world, and that imposes serialization constraints. It is harder to know how much that slow-down would be and how much speed-up we will get before that. But a RSI cannot simply be a exponential growth forever.

Except the two (GPT-Image-2 and Nano Banana Pro), anything displayed here can run on the 16 GiB MacBook (including the FLUX.2 [dev]): https://tests.drawthings.ai/generate

To our knowledge, Bonsai Image 4B is the first image model in its parameter class to run directly on an iPhone.

This is wrong. But they worded it carefully to be not entirely wrong.

FLUX.2 [klein] 4B (the same parameter class, basically the same model) runs on iPhone through Draw Things app, with 8-bit or 6-bit quantization (hence not "directly", I guess, but that is the technicality that sounds fishy enough).

It solves part of the download issue if they actually delivers a 1-bit whole package (currently their download is around 3.5GiB, still not ideal since FLUX.2 [klein] 4B you can get a package including text encoder ~6 GiB).

For speed, no. Draw Things runs on iPhone just fine and generally faster than their implementation on the same model (FLUX.2 [klein] 4B).

One thing people seems not to acknowledge, and this post made it super clear is that NVIDIA kept their lead extremely well in a few years of very high growth. The TFLOPs, the bandwidth, the interconnect mentioned in this post continues to grow at exponential rate with no sign of stopping yet. This is a 30-year-old incumbent reminding you. The willingness to compete from NVIDIA is just simply remarkable.

Probably not really. For gaming, I think probably just need to have a better way to explain visual and what the problem is (collision not done correctly, ways to feedback to LLM's experimentation loop how that should be checked and why etc).

Models usually is broken if there is no feedback loop. Well, websites might be exception since they can one-shot pretty well. But there are plenty of things they can do well without one-shot that just requires a good feedback loop to be built.

Only if you think B is an important thing. He is easily > $100M from Tesla.

Since the frontier is only 8-month ahead of DeepSeek, it is hard to see how model training can be a moat as all the tricks are available from open labs in China. You really just need <100m to bootstrap at this point.

DSv4 generates much faster on NVIDIA class hardware. It is just a very efficient model.

I am not sure where this comment is from (possibly without looking at this project?). This project is running quasi-frontier model at reasonable tps (~30) with reasonable prefill performance (~500tps) with a high-end laptop. People simply project what they see from this project to what you optimistically can expect.

You can argue whether the projection is too optimistic or not, but this project definitely made me a little bit optimistic on that end.

Thanks. I think it is a good explanation, but also suggests a gap. QAT to me, if done right, is the only way to recover performance for extreme quantization regime. The only thing matters of course, if whether it can work. My confidence in QAT comes from the LoRA can recover most quality misses in quantization, and that is still different from QAT for extreme quantization, so it could be very wrong. I need to try it anyway.

I am actually getting interested in QAT these days, especially for LSQ+ type, but it doesn't seem like people have done that enough in open-source world at least, for 2-bit / 3-bit OPD with LSQ+ basically.

The competition is on DeepSeek v4 Flash for similar size / deployment target.

Agreed. It is nonsensical to argue that a 3B transformer that hard-capped to decode 100 tokens is "intelligent". Of course when we are evaluating whether "transformers" is intelligent or not, we are talking about taking transformers as a core part of the system in some ways and enhance it with some other means (as you said, it is pretty trivial to making transformers a Turing machine, hence can carry out any compute, including intelligence (if you are in the camp that intelligence is computable, I don't think it makes sense to argue with anyone who otherwise believes intelligence is not computable)).

but transformers are not AGI, and they will never be AGI

Like the claim "transformers are AGI", this needs proof, otherwise should be prefixed "I think". And honestly, positive proof is easier than negative proof (you just need to make one transformer model that is a AGI, whereas the never claim requires you to enumerated all possibilities).

I think the parent comment is specifically addressing why the black box (or stochastic?) optimizer he used not working.

Like you said, yes, it is about courage. I just felt that I won't have that courage when I were in his shoes. We can just be different.

That's a lawful FBI. This is a lawless executive branch. As we all know by now, executive branch has a lot a power that cannot be limited by Congress nor the Courts and erasing a few zeros from 4T market valuation is a piece of cake (as we witnessed daily how they moved billions around the market to their favorite inside traders).

I honestly don't know. tim@apple.com is unavailable for quite some time now (since I tried a few years ago), while lisasu@amd.com still works around that time frame.

He also donated to Kamala Harris campaign. He would also donate to the next Democratic president for their inauguration if they still choose to do this corruptive thing. And your point is?

Trump is the president. People voted him into the Office. Tim Cook didn't give him the golden statue before he is in the Office.

Everyone in the United States is complicit to the horrible things done by the Trump administration by your logic. I partially agree, but I also think burning Apple to the ground will not be Tim Cook's legacy and he is in no place to go against the executive branch.

It is not about Trump, it is about the corrupted executive branch. Tim didn't do any crime against humanity in his act.

I32 are 8 4-bit value packed into one int32.