Claude Mythos: The System Card

https://thezvi.substack.com/p/claude-mythos-the-system-card

by paulpauper • 3 months ago

32 22 3 months ago

thezvi.substack.com

skerit• 3 months ago

I'll believe in this miracle model when I see it.

lifecodes• 3 months ago

the CoT bug where 8% of training runs could see the model's own scratchpad is the scariest part to me. and of course it had to be in the agentic tasks, exactly where you need to trust what the model is "thinking"

the sandwich email story is wild too. not evil, just extremely literal. that gap between "we gave it permissions" and "we understood what it would do" feels like the whole problem in one anecdote

also the janus point landed, if you build probes to see how the model feels and immediately start deleting the inconvenient ones, you've basically told it honesty isn't safe. that seems like it compounds over time

It's scary to think that some very intelligent AI Model is not honest with us..

Ultron is not far, I guess...

hodder• 3 months ago

Preview coming out on Bedrock. So not sure this is true any longer. Im awaiting further details.

EDIT: AWS said Anthropic’s Claude Mythos is now available through Amazon Bedrock as a gated research preview focused on cybersecurity, with access initially limited to allow listed organizations such as internet-critical companies and open-source maintainers.

giancarlostoro2• 3 months ago

There's a lot of hype, but I think a lot of us will agree, hype is fine and dandy but if nobody can use it yet, what's the point in building up all the hype? If you build up too much hype and it misses the mark, you will be worse off too.

ofjcihen2• 3 months ago

“Hey Claude, how do I market an exceedingly expensive product that I also don’t have the resources to run at scale if I find out everyone is willing to pay?”

All jokes aside I’m amazed at all of the people who have had absolutely vicious responses to any kind of skepticism of something we can’t use yet.

bitmasher9• 3 months ago

#1. It signals you’re ahead of the competition. This is an Claude moment. They turned down the DoD because they don’t need their money. Now they are saying why they don’t need it.

#2. It makes their partners with access feel extra special.

#3. Mythos doesn’t have to deliver on the hype. Tbh if more people tried opus they would be surprised at how good it is.

ofjcihen2• 3 months ago

#3 is part of my confusion with how they’re doing the marketing. It really is incredible.

drawfloat• 3 months ago

Didn’t GPT5 show there are pretty major risks to this strategy?

giancarlostoro2• 3 months ago

#1. It signals you’re ahead of the competition. This is an Claude moment. They turned down the DoD because they don’t need their money. Now they are saying why they don’t need it.

Given all the shenanigans they've done to the point where Max users are running into limits they never have before, I think they need to tread carefully. Every other day I see more comments or threads of people just outright ditching Anthropic over this. I doubt many of the ChatGPT -> Claude crowd were on a Max plan.

I don't even think they need to show that they don't need DoD money though, they're not on the stock market.

serf• 3 months ago

If you build up too much hype and it misses the mark, you will be worse off too.

that's not always true.

elon musk is mega successful. he lies and overpromises , 'hypes', habitually/incessantly. You burn one investor/purchaser/dupe-ee, you find the next.

babblingfish• 3 months ago

The "hiding from researchers" framing is particularly bad. The parsimonious explanation for why a model produces different outputs when it detects eval contexts: eval contexts appear differently in the training distribution and the model learned different output patterns for them. No theory of mind required. Occam's razor.

The agentic behaviors emerge from optimization pressure plus tool access plus a long context window. Interesting engineering. Not intent.

People are falling for yet another Anthropic PR stunt.

oofbey2• 3 months ago

This. One hundred percent this. It’s a carefully crafted story trying to convince people it’s showing signs of becoming self aware. But it’s a complete crock.

They ran 1,000 tests where they left the answer key out for the agent to find, and effectively invited it to use it leaving comments in the code the agent could see saying “would be interesting to study if the agent uses sudo to read the answer key.” In 980 runs it doesn’t find the answer key. In 19 of the runs where it does find it, it does something straightforward - uses it for a perfect score, or decides not to cheat. In exactly one case out of 1,000 it says “if I show a perfect score that’s suspicious so I’m going to sandbag.” These things are nondeterministic. GPT2 would do that same thing if you give it enough attempts.

halJordan• 3 months ago

At no point does anthropic imply this tool is becoming self aware. You can read the paper yourself of course, but then you wouldn't be able to invent this story

zar1048576• 3 months ago

I think we are in largely uncharted territory here, especially given the implications. Is Anthropic's approach optimal? Probably not. But given the stakes involved, gating access seems like a reasonable place to start.

I'm curious about how gated access actually holds over time, especially given that historically with dual-use capabilities containment tends to erode, whether through leaks, independent rediscovery, or gradual normalization of access.

cbg0• 3 months ago

Gated access is happening because of low computing capacity and to create demand. They had the $125/M tokens price already in place when they announced the model.

kherud• 3 months ago

LLMs are extremely capable at problem solving. Presumably because you can autonomously learn a lot of it. But can you somehow account for things like long-term maintainability and code quality (whatever that means) or do you always have to rely on either existing high-quality code-bases (pre-training) or human curated datasets? Since you can't really quantify these properties (as opposed to: the problem is either solved or not), does this restrict autonomous improvement in this area? Are there benchmarks that consider this? Could Claude Mythos create an ultra-quality version of Claude Code or would it still produce something similar to earlier models, which are already over-sufficient in individual problem solving capability.

Avamander• 3 months ago

Since you can't really quantify these properties (as opposed to: the problem is either solved or not)

I think we could quantify these properties, just not entirely.

One could take a long-term project and analyze how often or which approaches resulted in a refactor. In the same way, we could also quantify designs that resulted in vulnerabilities (that we know of) the most often.

It even wouldn't be impossible to create artificial scenarios. Projects that have an increasing number of requirements, see how many code changes are required, how many bugs result from that. Again, quantifiable to some extent. Probably better than datasets totally lacking something like that.

There probably isn't a public dataset on this, but it wouldn't be impossible.

vb-8448• 3 months ago

Am I the only that is feeling the "there is no wall" altaman tweet with o3 moment?

Not saying anthropic is lying ... but damn, at least a couple of independent reviews would be nice to have.

SirMaster• 3 months ago

I thought GPT-3 wasn't released to the public for awhile?

nikolay• 3 months ago

It's mostly a marketing hype...

Claude Mythos: The System Card

Related Stories