causal

Yeah if anything it makes Anthropic look incompetent

This is true, models themselves can be dangerous, but my point is that dangerous providers beget dangerous models.

My takeaway is that closed model providers are dangerous. OpenAI and Anthropic are more motivated than anyone to prove that models can be dangerous, and so they will make dangerous models. "Look how dangerous our models are!" No bro YOU are the danger.

I think you misunderstand. Code is also representing something. It may be what gets executed, but that does not make it "correct".

Not to mention the "distilled" models almost certainly have other training inputs as well, again watering down the meaning of the word.

And if just partial output is all that it takes to declare a model distilled, then every model trained on Internet content since 2023 is now technically a "distilled" ChatGPT and Claude model.

Are you talking about my username? Yeah I liked the word before LLMs made it cool/uncool.

1) Your own Wikipedia link goes on to describe using logits. Yes, language evolves to mean multiple things, and that is my point: Anthropic is pushing for a watered down definition. Furthermore, Anthropic hides thinking, so you do not even really get model outputs, you get some downstream partials. Further-furthermore, Anthropic does not describe their own models as distilled when they produce training data. Why? Because generating training data != distillation.

2) This is a stretch: it allows Anthropic to arbitrarily define "attack" via TOS, and ignores the fact that the generated training data is literally paid for by the "attackers".

Open weights dude. You can literally run it on your own or rented hardware and give your data to exactly nobody, unlike closed models.

"distillation attack" is such a loaded term that really pisses me off.

Distillation is a technical term with real meaning, and historically requires logits which Anthropic does not provide.

"Generated training data" is the correct term. It's not an "attack". And Anthropic undoubtedly also generates training data for each new generation of models, yet you never see them claim Fable is a distilled Opus.

Am I dumb or does this chart make no sense? Or why does the line only go up even with compaction? Or maybe "overall trajectory size" is hiding some meaning I don't understand?

"threatens" - Dawg there aint a lead anymore. I've been testing K3 and it is outperforming Sol and Fable on a project of mine, fixing stuff I couldn't get either to.

We need to see private set results, but if this holds then it might represent a breakthrough in other domains as well.

Hey I love the idea, don't let the haters get you down just because the site was vibe coded, that's trivial to improve upon.

I do think more information is needed to understand exactly where these claims are coming from. Not sure the "not legal advice" tiny text at the bottom is enough, people are going to see the claims and most won't double check.

Yeah title is way off lol

Yeah I sometimes see people on here getting defensive when you call out AI slop, saying maybe it's just a human who writes like Claude, and I really don't care- slop is slop.

That battle tested bit is so true, always was. It’s hard for engineers to admit their hard work is worthless until they have let other people use and shape it.

I think it was always hyperbolic to call rewrites the single greatest mistake (I can think of worse) but I think most of the wisdom still holds. And AI generated rewrites are arguably riskier because now NOBODY is familiar with the massive codebase.

If OpenAI is this shady culturally then all kinds of dark secrets could come out.

They just released 5.6 to great fanfare with benchmarks showing even their weakest model (Luna) supposedly smoking Fable. Having used it a couple hours I'm certain this is a lie, and I'm now curious what else is.

I feel like the converse question is more relevant: what projects have open source agents been good enough to implement on their own?

Distillation is a specific named technique from a pretty famous paper "Distilling the Knowledge in a Neural Network," by Hinton and others.

What makes it different from just training on any data is that you train the student model on "soft targets" that includes the full output distribution (logits) from the teacher model. Regular training uses one-hot targets and penalizes anything else; distillation will partly reward a student for getting in the distribution. This teaches the student to think like the teacher, not just imitate it.

Generating training data is not distillation, not in the technical sense, and I dislike Anthropic undermining the nomenclature to set a narrative.

So you can also destroy trust other ways. What’s your point?

Still debating whether these are actually information dense or not. Complexity gives the appearance of density, but these sentences always come in whole pages at a time while often failing to deliver the needed information.

I wish people would stop using Anthropics incorrect use of the term distill. They don’t share logits so you can’t distill. You can generate training data, which doesn’t sound nearly so scary.

Way easier said than done, and hiding that behavior isn’t trivial, and huge waste of compute budget if it’s found and never used. Also not difficult to run in contained environments where it doesn’t have access to Internet to begin with.

Not impossible I agree, but seems like a really impractical way to ship a trojan while much weaker channels exist.

Yeah I recently downgraded my subscription. The paternalism is out of control. Secret weights, secret guardrails, secret stenography, secret dumbing-down of triggers, secret token allowances.

Yeah though to be fair that's a settings option.

Depends on leadership culture. In toxic (aka “competitive”) environments managers are insecure and fear their own staff as potential competition.

For anyone wondering what the actual purported security weaknesses are in this article (I used the slop machine to reduce the slop):

- Cloud backups — by default, backups to iCloud/Google Drive contain plaintext messages, and E2EE backup is opt-in. Even if you enable it, a weak password collapses the effective security, and any other person in the chat with an unencrypted backup exposes the conversation.

- Metadata — who you talk to, when, how often, IP, contact graph, etc. This is the "reading your life without reading your messages" argument, and it's the part that's genuinely well-established.

- Pen register / FBI — the claim that WhatsApp uniquely delivers near-real-time metadata (~every 15 min) to law enforcement.

- Group chat membership integrity — a server-level adversary can inject a member into a group; messages stay encrypted but get delivered to the injected party. Endpoint compromise (Pegasus / CVE-2019-3568) — encryption is irrelevant if the device is owned.

- Closed source, Meta AI, business accounts — content can leave the E2EE envelope in those flows.

Nothing really new here, and as everyone else is pointing out Telegram might be worse.

I think it's important to multiply "likelihood of the human making a mistake" by "how many times the human must avoid a mistake". If your hobby presents 100 avoidable-but-life-threatening mistakes an hour, it's a dangerous hobby.

Too many people read "human error" as "human preventable" rather than "a thing you will also do because you are human"