mediaman

None of this matters, this is the judge approving a voluntary settlement reached between the parties last year.

If you think it should be different then you have to make a cogent argument why the public should get to interfere with a settlement the two sides mutually agree on.

He's saying they just plug your USB stick into some gizmo sold to the state, and it's going to find enough to escalate it.

If the idea is that there's no point in encrypting anything on a USB stick, and you just hope they don't look at it at all, sure.

But there is no threat model here that includes "check the USB stick" and does not automatically lead to finding and breaking "hidden" content.

Because they don't have a lot of parameters to store general Wikipedia knowledge. They're small. Use big models that have high parameter capacity to store general information. Or build a harness around the small model that searches a knowledge base/internet.

Use the right tool for the job. It's like asking why a screwdriver isn't good at sawing wood, or calling C a terrible language because it's hard to make CRUD apps with it.

Small open source models shouldn't be used for world knowledge, that's not their purpose.

Gemma 4 was released in April. It's a good series of multimodal models.

There was some recent reporting that a July release of the Pro model got pushed back for exactly that reason. Its performance was not good compared to the OpenAI/Anthropic big models. They are having a lot of problems with posttrain.

You're correct, but that's a different market segment and not the market GLM 5.2 and its peers compete in.

The labs are not interested in the small, fast, single purpose end of the market. Google increased their pricing on Flash so much that it stopped becoming a cheap model; instead, they released Gemma 4 open source, which is actually easier to use from a third-party inference provider than from Google.

From a total token volume perspective, these "utility" models (classifiers, simple summarizers, small OCR models) will absolutely drive enormous volumes of tokens, at low prices and margin and modest overall market size. Because the models are small and the performance requirements are modest, and because their use cases are specialized rather than general, there are poor economies of scale: they can run cost effectively on rented small GPUs, and a big player doesn't get a structural cost advantage. These models are usually 1b - 30b in size, and can run on a rented 5090. I've productized these myself: I run millions of pages through a fine tuned 1b OCR language model that runs on 5090s at a cost far lower than commercial providers.

But that's not the segment of the market where GLM 5.2, Kimi 3, etc., play. They compete with frontier capabilities, and they are not particularly cheaper than OpenAI models at a cost per task. (I do actually think they compete well with Anthropic, because Anthropic's model efficiencies are poor compared to OpenAI.) And although this part of the market may not be the bulk of the token volume, it is the bulk of the market value.

That's because a lot of human knowledge work is too generalized and fuzzy for dedicated, fine-tuned models, so they are almost entirely different markets that don't particularly compete with each other. (Though if SaaS companies successfully build around verticals that can use small models applied against well-defined jobs, there may be opportunity to push the small/big capability boundary to subsume marginally more valuable tasks that today would require mid-grade reasoning.)

The (quite excellent) article discusses several of your points. If you haven't read it, I recommend it.

  - Commodity market profitability is determined by marginal cost of production. LLMs have marginal cost; traditional software does not.

  - Models are not free. Downloading them is free. Running them is not. This has manufacturing economics, not software economics; the idea that they are "free" is an economic category error as it relates to their actual use

  - The highest tier Chinese models are not more economical than US frontier models. Try GLM 5.2 and see how much it costs to do real work. I did, and it was more expensive than GPT 5.6.

  - This is because US labs are leading on cost efficacy of inference ($/task)

  - Training will decline as a percentage of costs as inference expands compute share due to agentic workloads. A big part of training now is optimizing token efficiency. It's hard to distill token efficiency; that is perhaps why Chinese LLMs are so inefficient.

  - With increasing inference as % of total compute, if labs create efficient models -- which they can, because they can create highly optimized models amortized over very high inference loads -- they can be low cost producers, and be competitive at $/task rates

Convergence in coding makes them highly substitutable. But I could see harnesses configured for different purposes -- let's say, a harness for creating teaching plans -- being able to cater to its audience better than a coding harness. Maybe it's got tools to plug into standardized curricula, what the lesson books will be, what other lesson plans the district's teachers have made, etc., which could be done in a clunky way in a regular harness but could be streamlined.

This happens all the time. The government can decide legislatively that certain commercial terms are simply unenforceable. Making distillation clauses unenforceable in tort law would be straightforward. They can decide what customers they want to have, but they do not have unfettered rights as to the enforceability of terms governing the relationships between the parties.

The problem with many of these kinds of works is that they are a product of a provincial SV mindset that is largely unaware of the broader world or anything at all outside of its cultural and economic ecosystem, so it winds up being a super narrow slice of conventional, mostly boring topics that everyone in SV rehashes over and over.

Wouldn't it be interesting to learn an influential work that changed how health care professionals run hospitals? Or a document that changed how mining works? A paper on Wright's theory of manufacturing scaling that explains the solar revolution? A thesis on how the world's factories moved from pneumatics to servo systems and why? How a policy thesis changed federal regulators' approach to approving rare disease drugs? Maybe Hayek's views on socialism and information theory, or perhaps an influential thesis on how antitrust monopoly regulation should work?

But instead it's a bunch of crypto and AI stuff that everyone in tech already knows about, rewarmed again for its hundredth serving.

Because quality of writing matters.

Good communicators learn to use the written word. Bad ones rely on mental crutches.

Good communicators get an audience, and bad ones won't.

You think it's a lost cause, but it's not, because people don't like this junk, because it is low quality and, on average, lacks substance.

The best minds in AI that I've seen all write their own words. They use AI to help them research or ideate, but what they write is their own.

Before assuming this is a "lost cause," consider why the smartest people in the room don't do it.

That's annoying, but it's not why scaled manufacturing is lowering unit costs of panel production. Look at bare panel prices, they've followed the same cost curve down.

The same problem exists in the airline market. Airline ticket prices are historically very low, but people complain about seats, fees, and so on. But then they keep buying the absolute cheapest ticket.

What consumers say they care about, and what they actually care about, are not the same. Otherwise they'd pay more for the less irritating product.

I have seen the SWIFT thing happen for $100k. I think AI could actually be better for this, because it's often easier to implement hard rules for the AI.

With the SWIFT incident I saw, there was a rule that no payment can go to a vendor's bank that isn't a current, approved vendor. But the rule was not enforced in software: it was an internal accounting rule that humans were supposed to follow. The AP person "thought it had been approved" because there was a similar transaction with a different company that was a new vendor at a similar time. The other transaction was legitimate, the fraudulent spoofer wasn't. The wire got sent to a party in China.

With AI agents, if you approach it from the perspective that it will be gullible and trickable by fraudsters, you build in these hard guardrails. With humans, it's much easier to believe that "we trained Lucy on this procedure" will work in all circumstances, even if Lucy still has the technical ability to bypass the official procedure.

In these cases, it starts looking a lot more like traditional software, with your little AI chaos monkeys constrained in little boxes within the software chain.

You can fix this simply by using normal controls.

That's why we have purchase orders that can only be entered by buyers. Product is received and approved by buyer. Invoice goes to accounting, who can't approve it unless there's a matching purchase order and receiver.

Yes, letting agents do whatever they want leads to disaster. But humans are gullible stochastic token generators as well. And that's why the problem is already solved.

Looks like it's planned.

https://github.com/orgs/chattocorp/projects/1?pane=issue&ite...

They did use RLHF at the time, at which point it is not a pure probabilistic representation of the training corpora. Bizarrely, RLHF never came up in the paper.

Bender's paper had this to say about stochastic parrots:

"Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot."

This was not even a correct criticism in 2021. She is right that, at the time, the pretraining -- where it learns to predict missing words in pre-existing corpuses of text -- is basically a stochastic parrot.

But nowhere in her paper does the term "reinforcement" come up. At the time, this was done mainly through RLHF (reinforcement learning from human feedback) - after the initial training is done, you then tune the model's responses based on human grading. Humans imbue their own meanings into the parameter weights through their judgment.

At this point, they aren't really stochastic parrots anymore, because parameter weights have been shaped beyond the text corpus. It's not purely probabilistic in the sense of using the probabilities of the underlying text sequences. (It still is probabilistic in its output, but that is a pointless claim, because all events in the universe are also probabilistic; it is not enough to merely claim that probability is involved in some way in the outputs.)

RLHF was already in use prior to the paper, and was written about by Christiano in 2017 "Deep reinforcement learning from human preferences," so it's surprising that Bender apparently didn't know about this well-known paper.

RLHF was also, of course, a precursor to a more advanced form of parameter shaping - reinforcement learning with verified rewards, or RLVF, which has driven a lot of the gains in verifiable domains lately. That was not done in 2021 when she wrote the paper. But if you knew about RLHF -- and knew how Alpha Zero worked, with training neural nets on game rollouts -- you could squint and see that it might be useful for language models.

So after being proven to not only having a limited understanding of the field at the time, but also not being able to forecast the field, she's now walking back what she meant by "stochastic parrot," I assume because she believes readers will not read what she wrote. But despite the protests, her original claim was that it is a parrot because the text has no meaning -- a direct quote from the paper, which only really makes sense if training stops at the pretrain.

Regulatory capture doesn't necessarily mean the regulated get to decide what the regulators do in precise steps. It can simply mean they support and exist within a regulatory regime that greatly benefits the regulated.

In fact, you generally don't want them directly telling the regulators what to do. Instead, the regulators make complex, costly rules that only large establishment players can follow. The regulators look like they're doing their job; the regulated enjoy higher margins and protection from disruption.

He has this wealth as a function of having sold the equity in the company he co-founded, so in fact virtually all his wealth has already gone through a tax event at the federal and state levels.

On-prem versus cloud inference doesn't matter for concentration of power.

Concentration of power exists when the model makers are the same as (or control) the inference providers. Making a model is capital intensive, so there aren't many of them. Providing inference is not: I don't even need to own GPUs; I can rent them from those who do and then sell by the token. B300s cost less than $4 an hour currently.

Cloud can even be more effective at lowering concentration of power than on premise. Asking people to individually buy $20,000 of compute equipment plus power and cooling equipment to run a frontier model is not something they're going to do if they can just pay four-tenths of a cent per output token. If the only cloud inference providers are the big proprietary US titans, that means you're going to get far more power concentration than if open source inference providers are an alternative, because then I can just switch my API endpoint.

The tools aren't owned by a cabal of state-adjacent parties. Specific implementations are. But open-source models have saved the day here.

Yes, people conflated progressive disclosure as a method with skills as a particular implementation because skills became the first widely adopted use of progressive disclosure.

But progressive disclosure is just a method that you can apply to lots of things to reduce context bloat. Any time you provide some kind of limited index or search to an AI and then let it expand that based on the circumstances of the request, it's progressive disclosure.

And one of the things you can apply it to is MCPs.

Why? That doesn't make any sense.

The government would be far better off figuring out how to take commodity models and applying them to government functions where they can, with deterministic scaffolding and guardrails, to make government more efficient, optionally using RL on traces from their use to improve their performance.

Imagine taking models and fine-tuning them / doing RL rollouts to help automate permit application approvals, as applied specifically to Dutch permit processes. That would be a real help to Dutch businesses!

That type of applied AI is more interesting and effective now than just trying to make another foundational model that isn't going to work well or do anything of economic value.

It's right that they steer toward the center of their distribution. But I would offer a different view on whether that's a step up for half the population.

Writing isn't a distribution on a single dimension that goes from "bad" to "good". It's a lot of dimensions that encompass everything from "funny" to "formal", "precise" or "hysterical". They may be filled with metaphors, or use allegory; they may use math or logic to explain. The allegories could be from science fiction or they could be Biblical or 19th century Victorian novels. None of these are right or wrong, but they are opinionated ways to express an idea.

Writing feels better when it has real texture and character to it. That character is not the monodistribution of "bad" to "good". It's whether it inhabits pockets of out-of-distribution thought in the thousands of dimensions of "thought-space."

An LLM pushing to the center of distribution means it pushes the writing out of inhabiting any of the interesting pockets that create the feeling of texture. The middle of the distribution does not mean it is average quality: it means it's not good at all. The median of the distribution can be far worse than the median writer if you accept that the median writer has out-of-distribution thoughts on at least something, and that it is this which makes their thoughts interesting.

That's why a rough, perhaps not-totally-grammatical article written by someone with interesting thoughts is vastly better than a "correct" LLM revision, even if the human writer isn't a 'good' writer. Their article occupies an opinionated stance on some dimension that matters; it sits in a pocket of interestingness that LLMs seem almost totally unable to inhabit.

The exact middle of the distribution across thousands of dimensions may actually be one of the very worst places of them all.

I can't stand it when LLMs tell you to tone it down. Their writing advice is almost universally awful. They want you to write the most cliched bland content possible.

Sometimes I see technical people who feel they aren't good writers, but who have good ideas. They then turn to LLMs, believing that the LLM will help them express their good ideas. They're often right that they have good ideas. But the LLM just turns them to sawdust.

Kudos to spurning the mediocrity conversion machine and hitting publish.

Anthropic is, I believe, fully pursuing the idea that you shouldn't use their model with anything but their own products. They don't care whether it generalizes.

I agree it's very frustrating to use with custom tools/harnesses that can speed up the process for domain specific purposes.

Double check your math. All of their posts in this thread are correct.

1/30,000 * 100 = .003

That is not what their policy states. It specifically says they will sabotage even non-distillation attempts, such as distributed training pipeline design. And given that they are so far very nonperformant in classification accuracy, expect it to randomly include far more topics wide of the mark.

The fun part is that you will never know if your neural net classification project is getting silently sabotaged because their classifier doesn't work!

The funny thing about this comment is that neural networks are universal function approximators.

The most fundamental essence of what they do is exactly what you say they don't: estimate.