HN user
I want to understand things clearly and explain them well.
http://distill.pub http://colah.github.io
(Disclaimer: I work on interpretability at Anthropic.)
I wanted to flag that this is an accessible blog post and that there's a link to the paper ( https://transformer-circuits.pub/2025/introspection/index.ht... ) at the top. The paper explores this in more detail and rigor.
Hi! I'm the research lead for Anthropic's interpretability team, and was the decision maker for us publishing our papers web first and not doing traditional publications.
A few thoughts:
(1) As others have commented, I think peer review in ML is pretty widely accepted to be dysfunctional right now. I think most people who have published in ML conferences would agree. It's not unusual for early PhD students and sometimes even undergrads to review, and reviewers are overburdened to the point where they can carefully consider all their papers. Everything I've said so far is just anecdote and opinion though. A more objective test was the NeurIPS 2021 Consistency Experiment ( https://blog.neurips.cc/2021/12/08/the-neurips-2021-consiste... ) which found that if a paper was accepted by the conference, there was only a ~50% chance that a parallel review process would come to the same conclusion.
(2) Modern peer review is a relatively modern invention, arising in post-WW2 science as the scientific community grew dramatically, and there was a need for more systematized ways to make decisions about publication, funding, jobs, etc. Famously, Einstein was offended by one of his papers being sent for review. I don't think it's at all obvious that this transition has been good for science! I see lots of people writing papers for reviewers, rather than with the goal of doing the most impactful science they can.
(3) As background, I spent 5 years of my life running a scientific journal ( https://distill.pub/ ), trying to have excellent review processes and enable non-traditional papers to be peer reviewed. I honestly just burnt out on this. Now I just want to do good research.
(4) We do circulate draft papers to researchers working on similar topics at other industry groups, and in academia. As other comments have noted, this sometimes leads to public comments on our papers. In many cases, these are a much deeper review than you'd see in typical peer review processes, such as independent reproduction of experiments.
See https://transformer-circuits.pub/2022/toy_model/index.html#m...
If you're new to this, I'd mostly just look at all the empirical examples.
The slightly harder thing is to consider the fact that neural networks are made of linear functions with non-linearities between them, and to try to think about when linear directions will be computationally natural as a result.
It's a bit different than what's discussed here, but color-contrast detectors in neural networks can be thought of as forming a Klein bottle: https://distill.pub/2020/circuits/equivariance/#hue-rotation...
(This is, in some sense, for similar reason to Gunnar Carlson et al finding a Klein bottle when looking at high-contrast image patches, except one level more abstract, since it's about features rather than data points.)
I guess I'll plug my hobby horse:
The whole discourse of "stochastic parrots" and "do models understand" and so on is deeply unhealthy because it should be scientific questions about mechanism, and people don't have a vocabulary for discussing the range of mechanisms which might exist inside a neural network. So instead we have lots of arguments where people project meaning onto very fuzzy ideas and the argument doesn't ground out to scientific, empirical claims.
Our recent paper reverse engineers the computation neural networks use to answer in a number of interesting cases (https://transformer-circuits.pub/2025/attribution-graphs/bio... ). We find computation that one might informally describe as "multi-step inference", "planning", and so on. I think it's maybe clarifying for this, because it grounds out to very specific empirical claims about mechanism (which we test by intervention experiments).
Of course, one can disagree with the informal language we use. I'm happy for people to use whatever language they want! I think in an ideal world, we'd move more towards talking about concrete mechanism, and we need to develop ways to talk about these informally.
There was previous discussion of our paper here: https://news.ycombinator.com/item?id=43505748
True! I suppose I was thinking about a 'strong' form of linear representations, which is something like: features are represented by linear combinations of neurons that display the same repulsion-geometries as observed in Toy Models, but that's not what you're saying / that's me jumping a step too far.
Note this happens in "uniform superposition". In reality, we're almost certainly in very non-uniform superposition.
One key term to look for is "feature manifolds" or "multi-diemsnional features". Some discussion here: https://transformer-circuits.pub/2024/july-update/index.html...
(Note that the term "strong linear representation" is becoming a term of art in the literature referring to the idea that all features are linear, rather than just most or some.)
I want to make sure I'm getting my terminology right -- why does superposition necessarily require the linear representation hypothesis? Or, to be more specific, does [individual neurons being used in combination with other neurons to represent more features than neurons] necessarily require [features are linear compositions of neurons]?
When you say "individual neurons being used in combination with other neurons to represent more features than neurons", that's a way one might _informally_ talk about superposition, but doesn't quite capture the technical nuance. So it's hard to know the full scope of what you intend. All kinds of crazy things are possible if you allow non-linear features, and it's not necessarily clear what a feature would mean.
Superposition, in the narrow technical sense of exploiting compressed sensing / high-dimensional spaces, requires linear representations and sparsity.
If you like symmetry, you might enjoy how symmetry falls out of circuit analysis of conv nets here:
Circuits I find less compelling, since the analysis there feels very tied to the transformer architecture in specific, but what do I know.
I don't think circuits is specific to transformers? Our work in the Transformer Circuits thread often is, but the original circuits work was done on convolutional vision models (https://distill.pub/2020/circuits/ )
Re linear representation hypothesis, surely it depends on the architecture? GANs, VAEs, CLIP, etc. seem to explicitly model manifolds
(1) There are actually quite a few examples of seemingly linear representations in GANs, VAEs, etc (see discussion in Toy Models for examples).
(2) Linear representations aren't necessarily in tension with the manifold hypothesis.
(3) GANs/VAEs/etc modeling things as a latent gaussian space is actually way more natural if you allow superposition (which requires linear representations) since central limit theorem allows superposition to produce Gaussian-like distributions.
Since this post is based on my 2014 blog post (https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ ), I thought I might comment.
I tried really hard to use topology as a way to understand neural networks, for example in these follow ups:
- https://colah.github.io/posts/2014-10-Visualizing-MNIST/
- https://colah.github.io/posts/2015-01-Visualizing-Representa...
There are places I've found the topological perspective useful, but after a decade of grappling with trying to understand what goes on inside neural networks, I just haven't gotten that much traction out of it.
I've had a lot more success with:
* The linear representation hypothesis - The idea that "concepts" (features) correspond to directions in neural networks.
* The idea of circuits - networks of such connected concepts.
Some selected related writing:
- https://distill.pub/2020/circuits/zoom-in/
- https://transformer-circuits.pub/2022/mech-interp-essay/inde...
- https://transformer-circuits.pub/2025/attribution-graphs/bio...
A few comments on this thread:
Gwern is correct in his prior quote of how long these articles took. I think 50-200 hours is a pretty good range.
I expect AI assistants could help quite a bit with implementing the interactive diagrams, which was a significant fraction of this time. This is especially true for authors without a background in web development.
However, a huge amount of the editorial time went into other things. This article was a best case scenario for an article not written by the editors themselves. Gabriel is phenomenal and was a delight to work with. The editors didn't write any code for this article that I remember. But we still spent many tens of hours giving feedback on the text and diagrams. You can see some of this in github - e.g. https://github.com/distillpub/post--momentum/issues?q=is%3Ai...
More broadly, we struggled a lot with procedural issues. (We wrote a bit about this here: https://distill.pub/2021/distill-hiatus/ ) In retrospect, I deeply regret trying to run Distill with the expectations of a scientific journal, rather than the freedom of a blog, or wish I'd pushed back more on process. Not only did it occupy enormous amounts of time and energy, but it was just very de-energizing. I wanted to spend my time writing great articles and helping people great articles.
(I was recently reading Thompson & Klein's Abundance, and kept thinking back to my experiences with Distill.)
Yep, that’s right!
If you want to be precise, there are “autoregressive transformers” and “bidirectional transformers”. Bidirectional is a lot more common in vision. In language models, you do see bidirectional models like Bert, but autoregressive is dominant.
Thanks for the great questions! I've been responding to this thread for the last few hours and I'm about to need to run, so I hope you'll forgive me redirecting you to some of the other answers I've given.
On whether the model is looking ahead, please see this comment which discusses the fact that there's both behavioral evidence, and also (more crucially) direct mechanistic evidence -- we can literally make an attribution graph and see an astronomer feature trigger "an"!
https://news.ycombinator.com/item?id=43497010
And also this comment, also on the mechanism underlying the model saying "an":
https://news.ycombinator.com/item?id=43499671
On the question of whether this constitutes planning, please see this other question, which links it to the more sophisticated "poetry planning" example from our paper:
Thanks for the feedback! I'm one of the authors.
I just wanted to make sure you noticed that this is linking to an accessible blog post that's trying to communicate a research result to a non-technical audience?
The actual research result is covered in two papers which you can find here:
- Methods paper: https://transformer-circuits.pub/2025/attribution-graphs/met...
- Paper applying this method to case studies in Claude 3.5 Haiku: https://transformer-circuits.pub/2025/attribution-graphs/bio...
These papers are jointly 150 pages and are quite technically dense, so it's very understandable that most commenters here are focusing on the non-technical blog post. But I just wanted to make sure that you were aware of the papers, given your feedback.
The obvious way to deal with this would be to send forward some of the internal activations as well as the generated words in the autoregressive chain.
Hi! I lead interpretability research at Anthropic.
That's a great intuition, and in fact the transformer architecture actually does exactly what you suggest! Activations from earlier time steps are sent forward to later time steps via attention. (This is another thing that's lost in the "models just predict the next word" framing.)
This actually has interesting practical implications -- for example, in some sense, it's the deep reason costs can sometimes be reduced via "prompt caching".
Hi! I'm one of the authors.
There certainly are many interesting parallels here. I often think about this from the perspective of systems biology, in Uri Alon's tradition. There are a range of graphs in biology with excitation and inhibitory edges -- transcription networks, protein networks, networks of biological neurons -- and one can study recurring motifs that turn up in these networks and try to learn from them.
It wouldn't be surprising if some lessons from that work may also transfer to artificial neural networks, although there are some technical things to consider.
Features correspond to vectors in activation space. So you can just do vector arithmetic!
If you aren't familiar with thinking about features, you might find it helpful to look at our previous work on features in superposition:
- https://transformer-circuits.pub/2022/toy_model/index.html
- https://transformer-circuits.pub/2023/monosemantic-features/...
- https://transformer-circuits.pub/2024/scaling-monosemanticit...
I think the question is: by what mechanism does it adjust up the probability of the token "an"? Of course, the reason it has learned to do this is that it saw this in training data. But it needs to learn circuits which actually perform that adjustment.
In principle, you could imagine trying to memorize a massive number of cases. But that becomes very hard! (And it makes predictions, for example, would it fail to predict "an" if I asked about astronomer in a more indirect way?)
But the good news is we no longer need to speculate about things like this. We can just look at the mechanisms! We didn't publish an attribution graph for this astronomer example, but I've looked at it, and there is an astronomer feature that drives "an".
We did publish a more sophisticated "poetry planning" example in our paper, along with pretty rigorous intervention experiments validating it. The poetry planning is actually much more impressive planning than this! I'd encourage you to read the example (and even interact with the graphs to verify what we say!). https://transformer-circuits.pub/2025/attribution-graphs/bio...
One question you might ask is why does the model learn this "planning" strategy, rather than just trying to memorize lots of cases? I think the answer is that, at some point, a circuit anticipating the next word, or the word at the end of the next line, actually becomes simpler and easier to learn than memorizing tens of thousands of disparate cases.
Just to be clear, the probability for "An" is high, just based on the prefix. You don't need to do beam search.
The planning is certainly performed by circuits which we learned during training.
I'd expect that, just like in the multi-step planning example, there are lots of places where the attribution graph we're observing is stitching together lots of circuits, such that it's better understood as a kind of "recombination" of fragments learned from many examples, rather than that there was something similar in the training data.
This is all very speculative, but:
- At the forward planning step, generating the candidate words seems like it's an intersection of the semantics and rhyming scheme. The model wouldn't need to have seen that intersection before -- the mechanism could easily piece examples independently building the pathway for the semantics, and the pathway for the rhyming scheme
- At the backward chaining step, many of the features for constructing sentence fragments seem like the target is quite general (perhaps animals in one case, or others might even just be nouns).
I used the astronomer example earlier as the most simple, minimal version of something you might think of as a kind of microscopic form of "planning", but I think that at this point in the conversation, it's probably helpful to switch to the poetry example in our paper:
https://transformer-circuits.pub/2025/attribution-graphs/bio...
There are several interesting properties:
- Something you might characterize as "forward search" (generating candidates for the word at the end of the next line, given rhyming scheme and semantics)
- Representing those candidates in an abstract way (the features active are general features for those words, not "motor features" for just saying that word)
- Holding many competing/alternative candidates in parallel.
- Something you might characterize as "backward chaining", where you work backwards from these candidates to "write towards them".
With that said, I think it's easy for these arguments to fall into philosophical arguments about what things like "planning" mean. As long as we agree on what is going on mechanistically, I'm honestly pretty indifferent to what we call it. I spoke to a wide range of colleagues, including at other institutions, and there was pretty widespread agreement that "planning" was the most natural language. But I'm open to other suggestions!
"An astronomer" is two tokens, which is the relevant concern when people worry about this.
Yes, there are two kinds of evidence.
Firstly, there is behavioral evidence. This is, to me, the less compelling kind. But it's important to understand. You are of course correct that, once Cluade has said "An", it will be inclined to say something starting with a vowel. But the mystery is really why, in setups like these, Claude is much more likely to say "An" than "A" in the first place. Regardless of what the underlying mechanism is -- and you could maybe imagine ways in which it could just "pattern match" without planning here -- it is preferred because in situations like this, you need to say "An" so that "astronomer" can follow.
But now we also have mechanistic evidence. If you make an attribution graph, you can literally see an astronomer feature fire, and that cause it to say "An".
We didn't publish this example, but you can see a more sophisticated version of this in the poetry planning section - https://transformer-circuits.pub/2025/attribution-graphs/bio...
Hi! I lead interpretability research at Anthropic. I also used to do a lot of basic ML pedagogy (https://colah.github.io/). I think this post and its children have some important questions about modern deep learning and how it relates to our present research, and wanted to take the opportunity to try and clarify a few things.
When people talk about models "just predicting the next word", this is a popularization of the fact that modern LLMs are "autoregressive" models. This actually has two components: an architectural component (the model generates words one at a time), and a loss component (it maximizes probability).
As the parent says, modern LLMs are finetuned with a different loss function after pretraining. This means that in some strict sense they're no longer autoregressive models – but they do still generate text one word at a time. I think this really is the heart of the "just predicting the next word" critique.
This brings us to a debate which goes back many, many years: what does it mean to predict the next word? Many researchers, including myself, have believed that if you want to predict the next word really well, you need to do a lot more. (And with this paper, we're able to see this mechanistically!)
Here's an example, which we didn't put in the paper: How does Claude answer "What do you call someone who studies the stars?" with "An astronomer"? In order to predict "An" instead of “A”, you need to know that you're going to say something that starts with a vowel next. So you're incentivized to figure out one word ahead, and indeed, Claude realizes it's going to say astronomer and works backwards. This is a kind of very, very small scale planning – but you can see how even just a pure autoregressive model is incentivized to do it.
I'm the research lead of Anthropic's interpretability team. I've seen some comments like this one, which I worry downplay the importance of @leogao et al's paper due to the similarity of ours. I think these comments are really undervaluing Gao et al's work.
It's not just that this is contemporaneous work (a project like this takes many months at the very least), but also that it introduces a number of novel contributions like TopK activations and new evaluations. It seems very possible that some of these innovations will be very important for this line of work going forward.
More generally, I think it's really unfortunate when we don't value contemporaneous work or replications. Prior to this paper, one could have imagined it being the case that sparse autoencoders worked on Claude due some idiosyncracy, but wouldn't work on other frontier models for some reason. This paper can give us increased confidence that they work broadly, and that in itself is something to celebrate. It gives us a more stable foundation to build on.
I'm personally really grateful to all the authors of this paper for their work pushing sparse autoencoders and mechanistic interpretability forward.
I'm glad you've found it easy to follow!
My best guess at the middle regime is that there are _empirical correlations between features_ due to the limited data. That is, even though the features are independent, there's some dataset size where by happenstance some features will start to look correlated, not just in the sense of a single feature, but something a bit more general. So then the model can represent something like a "principal component". But it's all an illusion due to the limited data and so it leads to terrible generalization!
This isn't something I've dug into. The main reason I suspect it is that if you look at the start of the generalizing regime, you'll see that each feature has a few small features slightly embedded in the same direction as it. These seem to be features with slight empirical correlations. So that's suggestive about the transition regime. But this is all speculation -- there's lots we don't yet understand!
Gosh, that's very flattering! Very touched by your interest.
I'm glad you've enjoyed it! If you like the idea of a periodic table of features, you might like the Early Vision article from the original Distill circuits thread: https://distill.pub/2020/circuits/early-vision/
We've had a much harder time isolating features in language models than vision models (especially early vision), so I think we have a clearer picture there. And it seems remarkably structured! My guess is that language models are just making very heavy use of superposition, which makes it much harder to tease apart the features and develop a similar picture. Although we did get a tiny bit of traction here: https://transformer-circuits.pub/2022/solu/index.html#sectio...
Thanks for the kind remark!
I don't think you should feel bad for being slow, or for doing "few" things at all.
Unfortunately, I think it's tricky to do this in a journal format. If you accept submissions, you'll have a constant flow of articles -- which vary greatly in quality -- who's authors very reasonably want timely help and a publication decision. And so it's very hard to go slow and do less, even if that's what would be right for you.
Could you get a Distill editor endowment to pay editors using donations throughout a non-profit fiscal sponsorship partner? ...
I don't think funding is the primary problem. I'm personally fortunate to have a good job, and happily spend a couple thousand a year out of pocket to cover Distill's operating expenses.
I think the key problem is that Distill's structure means that we can't really control how much energy it takes from us, nor chose to focus our energy on the things about Distill that excites us.
It's certainly true that there are strong biological analogies. The analogy between first layer conv features and neuroscience is pretty widely accepted -- a lot of theoretical neuroscience models produce the same features.(It's less clear for later layers whether they're biologically analogous. Several papers have found that the aggregate of neurons in those layers are able to predict biological neurons quite well, but I don't think we have the detailed and agreed upon a characterization of the features that exist on the biological side to make a strong feature-level case.)
The color vs black and white split also has biological analogies.
With that said, I'd hesitate to dismiss the GP comment. Separate from the color vs grayscale split, why do we observe low-frequency preferring to group with color? It seems very plausible to me that if there's a systematic artifact from how the data neural networks are trained on was compressed, that could play a role. Either way, it makes the argument that this is emerging from purely natural data and the network less clean. (One caveat is that these models are trained on very downscaled versions of larger images. Even if high-frequency data was discarded in the original, that wouldn't necessarily mean that high-frequency was discarded in the downsampled version the network sees. It would depend on details of the data processing pipeline.)
To be clear, I'm not a neuroscientist and this is all just my understanding from the ML side!
That's an interesting hypothesis which hadn't been on my radar. (I'm one of the authors.)