EnPissant

"I'm selling an AI security product and want to establish my brand. I'll post several scare-mongering posts on my blog every week and people like solid_fuel will eat it up because it's what they want to hear."

I'm guessing all the "censored" boxes are not actually censoring anything and are placed there to make you imagine something much worse.

Rust gives you statically linked binaries as well.

So your argument boils down to having to add `reqwest = "0.13.3"` to your `Cargo.toml`.

I thought MTP wasn't very useful on MoE models because the expert overlap for 2 tokens was too small.

When running on a GPU, dense models are shaping up to be the best way due to two things:

- Maximum intelligence per VRAM (you dont have much)

- Dense models can benefit from MTP to get an almost 2x speedup in decode (ie, a 27b dense model with mtp decodes at about the same speed as a MoE model with 14b active param model would). This is important because local llm rarely has parallel streams to batch together.

When running on large unified memory like Strix Halo or Spark Dgx, MoE models are usually best:

- You can get similar intelligence as a smaller dense model with fewer active params (to compensate for the slower memory) by throwing ram at the problem.

Pay? This is the best marketing they could have hoped for.

It's not real. It's like naming your movement "The Good People". It sprouted from the "Rationalist" community, which is even more self-aggrandizing.

Neither has any hope of doing any good for the world as they don't understand evolutionary pressures. They are set up to reward making members feel smart, not accomplishing anything.

And if they ever gain any real power, they will be corrupted immediately.

Over the past five months, our team has been running an experiment: building and shipping an internal beta of a software product with 0 lines of manually-written code.

This is such a common thing among software engineers nowadays that I was very surprised that OpenAI would open with that line as if it were mind blowing.

But then I saw it was published in February and OP is just reposting it to farm karma.

They did not submit the full log because this is fake.

You are correct. I should have said "increased to 200%".

But you won't see 2x expert re-use, the speedup with 5 streams will be tiny.

You don't need "very much" expert overlap to see aggregate gains at scale, you just need some of it

I'm not sure what you are claiming. Decode is bottle-necked by memory bandwidth. To see a speed up of 2x, you have to ensure each expert weight memory fetch can be used by 2 parallel streams. What exactly is the average factor you are claiming for 5x parallel streams (due to "birthday paradox" factors)? The Birthday paradox isn't really relevant here. It's about coverage, not parallelism.

Memory for context is an issue, but recent models like DeepSeek V4 use very little of it even at relatively large contexts.

This is not true.

This is just wishful thinking.

For prefill, it's really easy to batch MoE and get really good tk/s, even on a single stream.

For decode, you will run into the problem that:

1) you need more parallel requests which means more memory for context

2) 5 requests will not give you very much expert overlap on parallel requests

Even if you could fit a 500B model's expert weights in very fast system RAM, it would run so slow as to be useless.

Also, electricity isn't free.

There was only a very brief time it was selling for MSRP (last fall for $2000). Even if you use that as the previous data point, it's only 200% increased.

I hear "I'm not anti immigrant, I'm anti illegal immigrant" a lot. To which there is an easy solution: increase the number of legal immigrants we allow.

Being "anti illegal immigrant" doesn't have to imply you let in whoever wants as long as they follow some process. You are taking away the agency of the people to select its immigrants.

The following things are not in contradiction:

1) Someone can be against illegal immigration and for legal immigration.

2) That same person's idea about who should immigrate to the country may exclude most or all of the people who are currently immigrating illegally.

It's not like you can only be against illegal immigration because they forgot to fill out some form. Legal immigration has a component of deciding who gets in.

Now try applying this logic to elevators.

1. SWE-bench Verified is now saturated at 93.9% (congrats Anthropic), but anyone who hasn't reached that number yet still has more room for growth.

But if some or all players are bench-maxing it, then it becomes a much less useful metric for comparison.

Also, this doesn't address what OpenAI says about the test suite disallowing valid solutions.

But you aren't trying out the model. You quantized beyond what people generally say is acceptable, and reduced the number of experts, which these models are not designed for.

Even worse, the github repo advertises:

Pure C/Metal inference engine that runs Qwen3.5-397B-A17B (a 397 billion parameter Mixture-of-Experts model) on a MacBook Pro with 48GB RAM at 4.4+ tokens/second with production-quality output including tool calling.

Hiding the fact that active params is _not_ 17B.

Yeah, I am mostly only talking about the SSD bottleneck being too slow. No way Apple gets 17GB/s sustained. SSDs thermally throttle really fast, and you have some random access involved when it needs the next expert.

I don't mean to be a jerk, but 2-bit quant, reducing experts from 10 to 4, who knows if the test is running long enough for the SSD to thermal throttle, and still only getting 5.5 tokens/s does not sound useful to me.

This was perfect.

Clearly the top comment should be "Imagine a beowulf cluster of Deepseek v4!"

Streaming weights from RAM to GPU for prefill makes sense due to batching and pcie5 x16 is fast enough to make it worthwhile.

Streaming weights from RAM to GPU for decode makes no sense at all because batching requires multiple parallel streams.

Streaming weights from SSD _never_ makes sense because the delta between SSD and RAM is too large. There is no situation where you would not be able to fit a model in RAM and also have useful speeds from SSD.

Those are quants, not distills.

This should not be the top comment on every model release post. It's getting tiring.

Are nvfp4 / mxfp4 even useful without QAT?

And yet when you look at corruption perception indexes, they largely track the Hajnal line. As do so many other graphs.