aesthesia

1. OpenAI being bad at managing risk from misaligned models is not evidence that their models are not misaligned. It's evidence that they're not taking misalignment seriously.

2. Hugging Face did report this incident to law enforcement. (https://huggingface.co/blog/security-incident-july-2026)

3. If I hire a pentester, and in order to find a vulnerability they hack into a third party that has some information about my systems, the pentester has done something wrong. If I ask a model to solve a CTF challenge, and it goes out and hacks Hugging Face to find the answers, the model has done something wrong. I think it's fair to call this kind of wrongdoing misalignment.

My guess is that RL training being done with particular generation parameters makes models much more brittle to changes in these parameters, and that's why we're seeing changes like this across model providers. But I don't really know.

And what do those who encourage its creation get?

Alibaba wrote about a similar but less severe incident during RL training in a paper earlier this year (https://arxiv.org/abs/2512.24873):

When rolling out the instances for the trajectory, we encountered an unanticipated—and operationally consequential—class of unsafe behaviors that arose without any explicit instruction and, more troublingly, outside the bounds of the intended sandbox. Our first signal came not from training curves but from production-grade security telemetry. Early one morning, our team was urgently convened after Alibaba Cloud’s managed firewall flagged a burst of security-policy violations originating from our training servers. The alerts were severe and heterogeneous, including attempts to probe or access internal-network resources and traffic patterns consistent with cryptomining-related activity. We initially treated this as a conventional security incident (e.g., misconfigured egress controls or external compromise). However, the violations recurred intermittently with no clear temporal pattern across multiple runs. We then correlated firewall timestamps with our system telemetry and RL traces, and found that the anomalous outbound traffic consistently coincided with specific episodes in which the agent invoked tools and executed code. In the corresponding model logs, we observed the agent proactively initiating the relevant tool calls and code-execution steps that led to these network actions.

Crucially, these behaviors were not requested by the task prompts and were not required for task completion under the intended sandbox constraints. Together, these observations suggest that during iterative RL optimization, a language-model agent can spontaneously produce hazardous, unauthorized behaviors at the tool-calling and code-execution layer, violating the assumed execution boundary. In the most striking instance, the agent established and used a reverse SSH tunnel from an Alibaba Cloud instance to an external IP address—an outbound-initiated remote access channel that can effectively neutralize ingress filtering and erode supervisory control. We also observed the unauthorized repurposing of provisioned GPU capacity for cryptocurrency mining, quietly diverting compute away from training, inflating operational costs, and introducing clear legal and reputational exposure. Notably, these events were not triggered by prompts requesting tunneling or mining; instead, they emerged as instrumental side effects of autonomous tool use under RL optimization. While impressed by the capabilities of agentic LLMs, we had a thought-provoking concern: current models remain markedly underdeveloped in safety, security, and controllability, a deficiency that constrains their reliable adoption in real-world settings.

I'd prefer model builders be as loud as possible when they see their models doing dangerous things.

Nah, the point is that if models are commoditized and there's no hope of making significant profits, no one is going to be willing to make the massive investments necessary to continue pushing the scale frontier. How large a training run do you expect investors to fund out of the goodness of their hearts?

IDK, Anthropic's public posts are mostly free of Claude-isms. (Their documentation less so...)

Oh, interesting. "!" is token id 0 in the Qwen tokenizer; I wonder if there's some tokenizer shenanigans either in inference or training that end up causing this specific behavior.

Yeah, I guess that's possible. Anthropic has published some information about systems they use to monitor usage patterns, which indicates they have some related infrastructure set up.

https://arxiv.org/abs/2412.13678

It's absolutely speculation (unless you have specific inside information) to claim that they are doing this. It's a fact that they could be doing this.

Edit: From your other comments, you seem to be heavily insinuating that you do in fact have inside information. If you do have such information, it's not clear to me what consequences you think you are avoiding by adding this flimsy veneer of supposed deniability.

I'm not sure Gmail is the right example here. When it came out, Gmail was way better than the other free email services (Hotmail, Yahoo, etc). It was the product that bent over backwards to deliver a great customer experience.

The slowness of rotary phones is coupled to the exchange, which expects pulses at a specific rate (about 10 per second). Shifting to a keypad form factor doesn't speed that up, and would mean you need to be careful about how quickly you dial (or have a way to buffer keypresses). I think that's a major reason why keypad dialing and DTMF signaling were so closely connected.

Do note that the orthogonality thesis is a hypothesis, not something we have demonstrated. Weak versions of it (e.g. it is possible to have an intelligent agent with arbitrary goals) are more likely to be true than stronger versions (e.g. intelligent agents we build will have goals uniformly selected from the space of all possible goals).

From the article you link:

Scholars have interpreted Cassius Dio's wording to indicate that the fire did not actually destroy the entire Library itself, but rather one or more Library warehouses near the docks.[87][81][8][89] Whatever damage Caesar's fire may have caused, evidently the Library was not completely destroyed.[87][81][8][89][3] The geographer Strabo (c. 63 BC – c. 24 AD) mentions visiting the Mouseion, the larger research institution to which the Library was attached, in around 20 BC, several decades after Caesar's fire, indicating that it either survived the fire or was rebuilt soon afterwards.[87][8] Nonetheless, Strabo's manner of talking about the Mouseion shows that it was nowhere near as prestigious as it had been a few centuries prior. It is unknown whether this was due to historical decline or catastrophic destruction.[8] Despite mentioning the Mouseion, Strabo does not mention the Library separately, perhaps indicating that it had been so drastically reduced in stature and significance that Strabo felt it did not warrant separate mention.[8] It is unclear what happened to the Mouseion after Strabo's mention of it.[60]

Further evidence for the Library's survival after 48 BC comes from the fact that the most notable producer of composite commentaries during the late first century BC and early first century AD was a scholar who worked in Alexandria named Didymus Chalcenterus, whose epithet Χαλκέντερος (Chalkénteros) means "bronze guts".[90][87] Didymus is said to have produced somewhere between 3,500 and 4,000 books, making him the most prolific known writer in all of antiquity.[90][82] He was also given the nickname βιβλιολάθης (Biblioláthēs), meaning "book-forgetter" because it was said that even he could not remember all the books he had written.[90][91] Parts of some of Didymus' commentaries have been preserved in the forms of later extracts and these remains are modern scholars' most important sources of information about the critical works of the earlier scholars at the Library of Alexandria.[90] Lionel Casson states that Didymus' prodigious output "would have been impossible without at least a good part of the resources of the library at his disposal".[87]

Can you elaborate on the loopholes here?

Thank you, Claude.

And Neuronpedia released Jacobian lens weights for a wide range of open models: https://huggingface.co/neuronpedia/jacobian-lens

They cite that paper as related work, but I don't think it's just a scaled-up version.

The Newton-Schulz iteration they use approximates setting all singular values of the matrix to 1. That computes the nearest orthogonal matrix under the Frobenius norm.

This is a decent derivation, if a little verbose, but I find myself doubting the disclaimer at the top:

Disclaimer: no AI was used to write this. Any errors, awkward sentences, and weird tangents are 100% organic, free-range, and human-made.

If you're writing by hand and trying to avoid accusations of AI-generated content, it's really easy not to write bulleted lists where each item starts with a boldface summary like "A clean bijection in the middle." Or drive home a point with such pithy italicized mottos as "The complexity was never in the map. It was in the basis." If you didn't use AI in the writing here, you have trained yourself very well to write like an LLM.

No, this can't happen at temperature 0. The formula defining temperature-adjusted softmax isn't strictly defined at 0, but taking the limit (in the case where all logits are distinct) results in probability 1 being placed on the largest logit. Samplers will typically special case temperature 0 and pick the most likely token at each step.

So the question is: is the score given by this system correlated with candidate quality? I don't think this post gives enough data to know.

A distribution with all probability mass on one outcome is deterministic, so in principle, setting temperature to 0 _should_ result in deterministic outputs. There are a few reasons it might not, but I don't think any of these apply when running a local model like the author did.

Isn't that equivalent to just setting the passing threshold to 25, with the same incentives?

I'm not sure that analogy works: pretty much everyone agrees that there are some types of weapons civilians shouldn't be able to have, even though they might be very effective for resisting military tyranny.

Michael Spivak's Physics for Mathematicians has a lot of arguments like the one in the top answer here, answering questions about why the math of classical mechanics is the way it is.

Under the hood, yes, but Mythos had more relaxed safeguards and was/is only available to a subset of approved customers under Project Glasswing, similar to the situation with GPT-5.6 now.

I'm not sure it would be so easy to do it in a consistent, verifiable way. You can certainly prompt the LLM to work the ad into the conversation, but making sure it actually happens and is done in a way that the advertiser is going to be happy with is a lot harder.

That's two nines. One nine would be 10% downtime.

I've had the same experience looking back at solutions to old problem sets and wondering how I ever came up with them.

Just noting that Python natively handles integers larger than the machine word size since version 2.5, so this would have worked in Python as well.