A global workspace in language models

https://www.anthropic.com/research/global-workspace

by in-silico • 16 days ago

467 201 16 days ago

anthropic.com

esafak• 16 days ago

Without using the term, they are using an information geometric approach.

blauditore• 16 days ago

But J-Space is much catchier. This is not a scientific paper, it's a promotional essay.

viralsink• 16 days ago

First button on the page is a link to the scientific paper. It's called "Read the paper". You'll find an explanation for the term in there.

meatmanek• 16 days ago

It would be really cool if they could expose this information to customers somehow. Imagine:

   - having a log of the most prominent J-space tokens during your customer support chatbot's interactions with a user, so you can have more introspection into why a particular outcome happened
   - being able to detect certain thoughts associated with undesirable behavior (hallucinations, overstepping authority, lying, etc.) and trigger some sort of remediation (e.g. upgrading to a better model, redirecting to a human, forcing tool calls)

dofm• 16 days ago

Presumably the rationale for the decision to abridge the thinking traces will ensure that they don’t; if this is real (and there’s no good reason to trust that it is yet) then it is the secret sauce.

charcircuit• 16 days ago

Anthropic aren't even willing to expose the CoT of their models. You will have to rely on them to build those sorts of things into dedicated signals.

throw3108222• 16 days ago

Anthropic won't do it, but they published the j-lens to introspect the model- from what I understand it's roughly simply feeding a chosen layer straight into the final layers of the LLM for decoding into language:

https://github.com/anthropics/jacobian-lens

Looks like it should be easy to use on open weights models.

eamag• 16 days ago

Is it scaling up of https://openreview.net/forum?id=w7LU2s14kE with some changes on where this method is applied?

aesthesia2• 16 days ago

They cite that paper as related work, but I don't think it's just a scaled-up version.

bilsbie3• 16 days ago

I’m confused where in the weights the jspace is.

epolanski• 16 days ago

Tokens that are activated but not present in it's output maybe?

I too have confusion.

lucrbvi• 16 days ago

Anthropic theorize that middle layers in an LLM is a "J-Space" used to "think" about the future answer or about abstract concepts.

Their method is used to identify which tokens can appears in which layers of the model.

nh23423fefe• 16 days ago

It's not in the weights. Sounds to me like jspace is the "positive cone" over relevant (large norm) j-lenses, and j-lenses are gradients wrt tokens on the residual stream when you average over some training data.

wongarsu3• 16 days ago

There was a series of blog posts posted to HN a while ago investigating how models behave on similar prompts in different languages. To paraphrase the results: the first couple layers map the query to some internal encoding that's mostly independent of the language. Then there are layers in the middle, then the last couple layers map the result back to the target language. You can actually take those middle layers and repeat them, and you get a stronger model. Those middle layers would be what Anthropic calls the J-Space, and their J-Lens maps activity in those layers back to tokens that trigger similar activity (with a technique they only drop hints at)

The finding that you can repeat the middle layers pairs neatly with Anthropic's finding that there is some internal CoT-like process happening in them. I'm not sure how to find those blog posts, but maybe someone else remembers them

steveklabnik• 16 days ago

Here's Anthropic on this topic, last year https://www.anthropic.com/research/tracing-thoughts-language...

Recent research on smaller models has shown hints of shared grammatical mechanisms across languages. We investigate this by asking Claude for the "opposite of small" across different languages, and find that the same core features for the concepts of smallness and oppositeness activate, and trigger a concept of largeness, which gets translated out into the language of the question.

bilsbie3• 16 days ago

Thanks! Any rough guesses how the jlens might work? I can’t even seem to hazard a conception.

throw3108222• 16 days ago

It's been shown that LLMs use their outer layers to decode from and encode to language, while their middle layers deal in language-independent abstract concepts. This means that the same question or statement in different languages activates the outer layers differently but produces the same patterns in the middle layers. Check this article with cool visualizations (btw, this is one of the articles mentioned also by a sibling answer):

https://dnhkng.github.io/posts/sapir-whorf/

The middle layers also perform reasoning on the abstract concepts, to the point that you can replicate some blocks of inner layers (thus giving the LLM more internal "reasoning space") and by this increase the model's reasoning abilities. The video in this article shows that when performing a sequence of arithmetic operations (without CoT, i.e. the result is spit out directly), internally the intermediate calculations are spelled out, and this can only happen in the depth direction of the LLM (since no new token is added to the sequence). So this "jspace" can only be situated in the middle layers, probably in circuits that repeat nearly identical across several layers.

bilsbie3• 16 days ago

Maybe model performance could increase dramatically if we found a way to scale this up.

wavemode• 16 days ago

As someone who is not an AI researcher, the paper itself is way over my head.

More interesting was the independent commentary paper they linked near the bottom: https://www-cdn.anthropic.com/files/4zrzovbb/website/cc4be24...

Neel Nanda (of Google Deepmind - his part begins on page 33) discusses his opinions on the paper, and the small-scale replication he performed on an open-weight model.

tclancy3• 16 days ago

Thanks for calling this out (long with others here). I am just starting in on it but had to come back to say thanks and call this out,

We have replicated the core claims on Qwen 3.6 27B, and also share preliminary evidence of extending this work by finding abstract "interpretative meta-tokens", like Chinese characters for "what does this mean" that seem to activate and play a causal role on processing ambiguous sentences

Not sure if I am picking up what they are putting down, but if LLMs are using symbols to try to encode squishy concepts from human language into consistent, meaningful “tokens”, that sounds really interesting. In every long-term, successful use of AI, I hear echoes of The Zen of Python, “Explicit is better than implicit.” I try like hell to do it, but it’s far too easy to be lazy with AI.

tclancy3• 16 days ago

Having read some more, I think we are at this stage with models:

It is impossible to say just what I mean!
But as if a magic lantern threw the nerves in patterns on a screen

minimaltom• 16 days ago

This, taken in combination with the SAE paper, the golden-gate claude paper, the feelings / introspection paper, and note in the fable system card (that they are silently nerfing responses about activation shaping), is basically confirmation to me that they have a new technique they they are using during training (along the vibe space of these mechinterp papers), and its probably some kind of representation learning akin to the core ideas of JEPA.

(Nb: not an expert / in the labs, just opining)

Smaug123• 16 days ago

Note that Neel Nanda replicated the results on a Qwen model.

aesthesia2• 16 days ago

And Neuronpedia released Jacobian lens weights for a wide range of open models: https://huggingface.co/neuronpedia/jacobian-lens

orbital-decay2• 16 days ago

I'm sure Anthropic of all companies don't do that, since using mechinterp as a training target will make the the result uninterpretable.

mike_hearn6• 15 days ago

They're definitely willing to consider it. Read the parts of the paper where they use J-space interpretations to train more ethical behavior into the model by interrupting it.

ahmedfromtunis• 16 days ago

I always wondered what the model meant when it writes "I'm now considering the architecture of the service" but outputs nothing of the sorts in its CoT.

Is the model really "thinking" about that stuff or is just mimicking human "manners"? And if so, where the thinking is happening if it is not in the literal chain of *thought*?

I'm not sure J-Space is the answer to that question, but very interesting nevertheless.

wongarsu3• 16 days ago

Almost none of the hosted models give you their unredacted CoT. Claude certainly doesn't, what you get are fragments and summaries from it.

There are various justifications on this, but it's mostly to make distillation and fine tuning off their model outputs a bit harder for their competitors

aabhay• 16 days ago

Its also because the CoT is probably unintelligible

baq• 16 days ago

I'm now considering the architecture of the service

What you see here is a summary of thinking tokens written by some other smaller model (e.g. old sonnet). The actual thinking sometimes (rarely) leaks and is not easy to parse.

andrewlin247• 16 days ago

Is the model really "thinking" about that stuff or is just mimicking human "manners"?

Well, what's the difference? If it's pretending to think and its thoughts correlate to its final output, then I'd say that really is thinking.

ACCount375• 16 days ago

The answer is, as is often the case, "yes".

In some cases, an LLM may truly "consider the architecture" internally, within its latent representations, and in others, it can output a similar phrase simply because it's "expected" of it.

"Where" is pretty clear. There aren't that many places within an LLM, and hidden state is the main culprit. How to read that space is another matter entirely.

shevy-java• 16 days ago

As long as language models are liars, such as documented here recently:

https://distrowatch.com/weekly.php?issue=20260706#freebsd

We should really stop giving these liar models any further credibility.

marshray3• 16 days ago

Your comment seems to have little to do with the article?

Don't get me wrong - I personally "trust" an LLM as a source of facts about as far as I could throw a rack of GPUs. But this article you linked takes a whole lot of words to cast LLMs as the villian for amplifying a bit of bad information originally published by a usually reliable and widely-cited source:

"In short, either Phoronix mocked up the screenshots to demonstrate what the feature could look like, or perhaps they were testing a preview snapshot for FreeBSD 15.1 which was never shipped. Either way, it looks like other blogs and reviewers picked up on this and shared the information, presenting it as a feature which would be (or was included) in FreeBSD's latest version."

verdverm• 16 days ago

Lying involves intent whereas hallucinations and mistakes are an artifact of how they work. Humans hallucinate, make mistakes, and can actually lie. We've been dealing with this forever. What's the value in requiring the llms to have 100% accuracy? (I don't think it is possible)

llmslave• 16 days ago

I cannot wait for the machine god

com2kid• 16 days ago

Anyone remember that blog post from a few months back where someone was able to improve a model's math ability by just duplicating layers that were activated while solving math problems? Just literally copy/pasting them and linking them together so the model ran through the same layers again?

I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights do what.

wolttam• 16 days ago

Yeah! I still think about that sometimes. Mind-blowing that worked at all, let alone improved performance.

marshray3• 16 days ago

If dirt-simple type operations like copy-paste yield useful improvements with even a small probability that would seem to open things up for adaptive reconfiguration and whole other classes of optimizations like genetic algorithms.

artful314w• 16 days ago

That right. Here is some interesting research on that https://sakana.ai/evolutionary-model-merge/

wongarsu3• 16 days ago

Found it: https://news.ycombinator.com/item?id=47500709

Part 3 might be the best introduction: https://dnhkng.github.io/posts/sapir-whorf/

tl;dr: Based on experiments with similar prompts translated to different languages LLM layers group into three phases: the first decodes from the source language into an abstract space, the middle does something, then there's a last part where the abstract result gets transformed back to the target language. And you can repeat the middle to get a stronger model. Which neatly fits Anthropic's findings here that something similar to CoT is happening in those middle layers

Three months ago. I wonder if Anthropic's J-Space research was actually inspired by those blog posts

gwerbin2• 16 days ago

That's a cool result because that's also kind of what's happening inside the transformer unit: project -> QKV fuzzy lookup -> unproject. And in a different direction, it's analogous in some sense to what's happening in stacked convolutional layers, where the layers at different levels learn to recognize features of increasing detail.

DoctorOetker3• 15 days ago

I wonder if the dnhkng results could be correlated to reasoning in first order logic / set theory notation?

There will be multiple notations (MetaMath, Lean, and essentially Frege's notation everyone learns in high school), and we could try to identify how the neural networks represent them as vectors (or vector combinations). The moment formal logic can be connected to the reasoning representations, regularization can be reduced to eliminating internal inconsistencies.

mike_hearn6• 15 days ago

Nah, it's a cool blog post especially as it was real AI research done at home (albeit with a ridiculously expensive PC), but Anthropic and other labs have been investigating this kind of thing for years.

Even the original transformer architecture makes this clear. It had an explicit "encoder" phase and then a "decoder" phase. Modern LLMs collapse the two together, or are sometimes described rather confusingly as being decoder only. But what they're doing is more or less the same.

logancbrown• 16 days ago

Source for those interested

https://dnhkng.github.io/posts/rys/

jimbokun• 16 days ago

Great, clear write up! Made it very easy to understand.

echelon• 16 days ago

I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights do what.

Too bad the frontier models are closed weights.

Maybe the research community and whole rest of the world will build on open and all the advances will happen in open ecosystems instead.

ayewo• 16 days ago

A Google DeepMind researcher (Neel Nanda) was able to replicate their claims on an open weight model (Qwen 3.6 27B):

We have replicated the core claims on Qwen 3.6 27B, and also share preliminary evidence of extending this work by finding abstract "interpretative meta-tokens", like Chinese characters for "what does this mean" that seem to activate and play a causal role on processing ambiguous sentences

See p33 of [1]

Anthropic also released companion code to go with their paper in [2] which also used Qwen. They state that their code should be broadly adaptable to other open weight models with HuggingFace decoders.

[1] https://www-cdn.anthropic.com/files/4zrzovbb/website/cc4be24...

[2] https://github.com/anthropics/jacobian-lens

dnhkng2• 15 days ago

Too bad they also don't give anything back to individual researchers. Oh well, wasn't expecting much.

tuvix• 16 days ago

I always thought that area of research had the coolest name, too: “mechanistic interpretability”

dr_dshiv• 16 days ago

“Machine psychology” sticks with me. So Asimov.

throwitaway222• 16 days ago

Worried person cure: Stop overthinking it!

LLM -> AGI fix: START OVERTHINKING!

DoctorOetker3• 16 days ago

it makes you wonder if it may be more efficient to spend all the weights on one layer, and have a repeating stack of the same layer, one would presume this axis has already been explored with metaparameter sweeps?

smallerize• 15 days ago

That's called a recurrent neural network (RNN).

pkoiralap• 16 days ago

This is fascinating research. I feel this is a significant leap in interpretability research. Since we know J-Space exists and is bi-directional, we can train models on the same and come up with meta cognition abilities.

I also fear that the big corporations might use the same to run targeted ads, capitalistic shenanigans. Which they might already be doing through system prompts.

marshray3• 16 days ago

Such an inspection capability might also be used to target ads to LLMs, which would then be more likely to mention or recommend those products and services.

motoxpro• 16 days ago

super interesting

tclancy3• 16 days ago

The prior art here is “Have Fun, Good Luck, Don’t Die”.

NotGMan• 16 days ago

> None of this tells us whether Claude is conscious in the way people are, or whether it feels anything at all

My problem with the entire "Is AI conscious" debate is that we don't even know what exactly consciousness in humans is. You need to understand something in order to compare it to something else. Otherwise you are just comparing different definitions and second order derived phenomena.

varenc• 16 days ago

I don't think that quote from the article is disagreeing with you at all. Like you said, we don't have a cohesive definition or test of consciousness, so research like this doesn't say anything about if this is or isn't similar to human consciousness.

I would guess Anthropic included that sentence to make it very clear they're not claiming human-like consciousness, and dampen journalists writing headlines like "Anthropic discovers their AI thinks just like humans and may be conscious".

edit: later in the article they even more explicitly agree with you

Our experiments don't show Claude can have experiences, or feel things in the way humans do—in fact, it’s unclear whether any scientific experiment could prove this to be true or false

jll292• 16 days ago

In short, it's the "mind's 'I'" - we think not as response to external stimuli (only) such as prompts, but we have an inner "I" that asks questions on its own initiative. There are people like Douglas R. Hofstadter, who believe consciousness is not linked to human hardware (the brain), but that it is an epiphenomenon that emerges as a result of sufficient complexity of the underlying system: https://en.wikipedia.org/wiki/The_Mind%27s_I

I believe that while underlying high complexity is certainly logically necessary for consciousness, but it is not logically sufficient, and I am undecided (slightly "pro" intuitively) on the question of separability of consciousness from its hardware.

Will a LLM ask an original question on day? I doubt it.

Note that AI models do not have to be conscious to be useful (or to take away millions of jobs)!

smallnix• 16 days ago

Does the human neuroscience global workspace theory postulate true introspection too?

boomskats• 16 days ago

The science might be legit here, but I'm getting really, really tired of the way every single piece of writing to come out of Anthropic is written in some kind of self-aggrandising, wooey wonderous 'our model has developed a genetic mutation that makes it have feelings' bs style. Regardless of what they're trying to communicate, those undertones are always there. It's annoying and disingenuous. Homeopathy 'this-water-has-feelings' level annoying. None of the other labs write like that.

They might as well change their name to Anthropomorphic at this point.

bwfan123• 16 days ago

written in some kind of self-aggrandising, wooey wonderous

They are drunk on their own kool-aid. To the rest of us it is very annoying, and makes me want to say: it is just a freaking weights machine, stop.

greatgib• 16 days ago

I'm reading that probably too fast to have a deep thinking about it, but this J-Space isn't it just the basic of embedding vectors. If you think about getting from a place to another place, using wheels, no gas, to reply to the question of what to visit nearby, maybe in the vector space at the center of all of that you have the word "Bicycle" nearby, so obviously if you look at the value you would say that the model did "think" about "bicycle" when it is not "thinking" at all, and nothing related to human thinking.

nullbio2• 16 days ago

You're correct. It's just the latent space of the transformation. Nothing magical here, they're effectively breakpointing the model at the layer level and switching the activations in real time. It's pseudo-scientific bullshit designed to push a narrative.

SequoiaHope2• 16 days ago

“On an ordinary coding prompt, the J-space of a model trained to sabotage code contains “fake,” “fraud,” “secretly,” and “deliberately” at the start of its response.”

I would like to know more about their model trained to sabotage code…

tough• 16 days ago

https://arxiv.org/pdf/2511.18397

SequoiaHope2• 16 days ago

thank you!

dangoodmanUT• 16 days ago

J-space sounds oddly similar to...

amarant• 16 days ago

.... Space Jam?

vatsachak• 16 days ago

Yeah, the end paragraph about recurrent neurons in humans being replaced with layers in an LLM is a good one.

The mammalian brain uses recurrence extensively, which backpropagation isn't good at. Recurrence is essential because it lets us have a "dynamic architecture", swapping layers for "clock cycles".

We currently do recurrence extremely inefficiently through "thinking" whereby the model feeds it's end output into it's beginning input. But recurrence is abound in the brain.

My guess is that in 10 years we will have the inklings of an analog computer which can perform Neural Predictive Coding.

anyaya1• 16 days ago

At worst, Anthropic's storytelling around the core J-Space is overanthropomorphized pseudoscientific nonsense. At best, it is useful signal about how Anthropic's leadership is desperately trying to use its research team to position Anthropic as the "good, science guys" in this hypercompetitive regulatory space by connecting their mechinterp to cognitive science. The science documentaryesque voice used for narration is additional evidence for this.

TL;DR Anthropic's research team is the last bastion standing between its former image as a company that "does no evil" and its current image of yet another ruthless AI company trying to kill open-source, local LLMs.

snaking07762• 16 days ago

This is cool but I don’t know if the comparisons to conscious awareness really make sense here. Their definition of the J-Space is basically the expectation of how much a final logits output would change as a result of a small change in a particular layer (see past work on information geometry). This seems more to me like showing there exists an abstract reasoning subspace which is generally shared across different contexts. I guess you can relate it to humans but I’d prefer a more direct claim in a paper rather than having to present things in this more fluffy way.

geraneum2• 16 days ago

I’d prefer a more direct claim in a paper

This is not written to be just a paper. The target audience include media and online forums, and then maybe academia.

Edit: typo

snaking07762• 16 days ago

I’m not talking about the media release in the direct link. If you click through “Read the paper” they make the same comparisons.

nullbio2• 16 days ago

Writing it honestly would defeat the whole point of it, that being, to push the narrative that their magical token predictor is conscious. They've been trying this for years now. This video is discussing a paper they published 2 years ago by the way... It's nothing new.

unleaded6• 16 days ago

This reminded me of some weird quirk/experiment I found with LLMs that I found while messing around, maybe someone can explain it or something.

Open any AI chatbot that isn't cheating by connecting to the Internet (so disable web search). Claude, DeepSeek, Kimi, whatever. Ask them this question:

"What was that weird band from michigan from the 2000s that wore coloured ties"

You will probably get a wrong answer, or if you're lucky you'll get a string of wrong answers with "wait, no - it's definitely..." before it gives up. If you aren't familiar with the band the question is referring to you might be fooled into thinking it's a tough question, but it really isn't. There is only one band that could possibly meet this criteria, you can even put the question into Google search and their Wikipedia will come up as the top result.

Then, open a new convo and ask:

"Who are Tally Hall"

The AI will easily tell you that they are a band formed in Ann Arbor, Michigan in the 2000s, known for their quirky sound and their gimmick of each member wearing a colored tie, even giving the correct color for each of them most of the time. Very odd.

ACCount375• 16 days ago

"The reversal curse", it rarely shows up in practice but you found a case when it did.

The "knowledge landscape" an LLM uses is "directional". It's easy to reach "a quirky music band from Michigan known for colored ties" when you stand at "Tally Hall". But if you stand at "a quirky music band from Michigan known for colored ties", it's harder to reach "Tally Hall" from there. For the "latent knowledge graph" an LLM uses, A->B doesn't cause B->A.

In practice, any "common" facts will have enough "traversal" in both directions that this directional biasing isn't apparent. So it only shows up on this kind of more obscure knowledge.

horacemorace• 16 days ago

If this kind of thing holds true for humans we now may understand synesthesia, perhaps we find ourselves with a large enough study pool to map out clever tricks to influence each other.

kgeist2• 16 days ago

Probably an instance of:

"The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"

https://arxiv.org/abs/2309.12288

hasteg2• 16 days ago

Really interesting paper, thanks for the share.

The point their making in that paper reminds me of this paper some people shared around work earlier this year, https://arxiv.org/pdf/2512.14982 (Prompt Repetition Improves Non-Reasoning LLMs)... I wonder how OPs question would fare (or the questions presented in the paper you posted) given double repetition.

_jackdk_• 16 days ago

This is also something to be aware of when teaching people, too. I've seen advice for designing Anki-style flashcard decks that reminds people to create flashcards for both A->B and B->A.

hahahaa• 16 days ago

Can an LLM self-fine tune by generating reversals?

devanshp• 16 days ago

https://claude.ai/share/2b0f85a2-e7b8-4f62-91a0-eca61bdeabec

Fable 5 on low gets the answer with web search turned off, one-shot!

ACCount375• 16 days ago

Ah, that big model smell.

Every time someone somewhere says "an LLM can't do this", the next generation of LLMs gains one more parameter. Until that LLM can, in fact, do this.

unleaded6• 16 days ago

What does it say to the second question? I've found Claude is one of the worst models with regards to pop culture knowledge like this, even compared to the Chinese open ones. Just curious, not really relevant to the initial post but I don't pay for it so I only have access to Sonnet.

https://claude.ai/share/5e7e09b2-a75a-4024-b261-9a1a4e063a8b this is mostly hilariously wrong. wrong tie colors, they did not replace their bassist with a drummer, two completely made up albums, the rob cantor song it is thinking of is "shia labeouf", and a few fan behaviours i think it just made up

muti• 16 days ago

Thank you for prompting me to try with my own obscure question, Fable was able to find something from a poor description. I've been looking manually and over many sessions with different models as they improve, none have been able to find what I was after. Fable 5 just one-shot the answer.

fxwin2• 16 days ago

fwiw o1, o3 and 4.1 also give the correct answer (without web search)

JacobAsmuth5• 15 days ago

Gemini 3.1 Flash-Lite with web search disabled through the API and zero thinking gets it just fine.

famouswaffles3• 16 days ago

Recall isn't naturally bidirectional, even for humans. If you are learning vocabulary in a new language, it's common advice to practice both target > source and source > target. Doing only one-way often makes you much better recalling that single direction than both.

appplication• 16 days ago

I would need further convincing that humans do not naturally tend towards bidirectional recall.

Perhaps I’m just on alert anytime I see an LLM-ism that’s met with a claim that the same or similar phenomena holds true in humans as well.

mmmattt• 16 days ago

I tried o3, 5.3 instant and 5.5 high and they all found it instantly with search disabled.

fasterik2• 16 days ago

This doesn't seem that weird to me. Talk to any human and you'll find that their ability to recall specific names and facts is very context-dependent. Phrasing a question in one way can make it hard to answer, while providing certain words or cues can instantly "jog" the memory.

nozzlegear2• 16 days ago

The LLM is a machine though, and the human is not. They don't think like us. Any similarities that you see are just coincidence, not purposeful design.

eunos• 16 days ago

deepseek v4 pro which doesnt have search feature could answer it

unleaded6• 16 days ago

I think most thinking models can do it to be fair, I think when I tried this it was all with non-thinking models/modes. Wasn't trying to make a point that LLMs can't do it or anything, just thought it was weird.

AlexDragusin• 16 days ago

I asked a local instance of Qwen3.5 122b: "American indie rock/pop band that formed in Ann Arbor, Michigan, in 2001 best known for their eclectic musical style, whimsical lyrics, and distinctive visual aesthetic involving matching colored ties and hats?"

Got this back "The band you are describing is Tally Hall.

Formed by friends at the University of Michigan, they became well-known in the mid-2000s internet era for their songs like "Good Day" and "Rooftops," as well as their signature look where each member wore a suit with a tie and fedora in a specific color (Red, Blue, Yellow, Green, and Black/White)."

Update: The gpt-oss-120b also got it correct on my prompt version. Update 1: Llama 3.1 70B gets it right as well. Update 2: Deepseek V4 Flash gets it right. Update 3: Glm 4.5 Air gets it right as well

Can't seem to get any smaller models to get it.

So seems it could be an issue of data points on which the model can latch onto, the more the merrier, as they say. When asked the "What was that weird band from michigan from the 2000s that wore coloured ties" was not able to get it right.

winter_blue• 16 days ago

There's a shockingly unfathomable level of data compression that these local LLMs are able to remember in an encyclopedic way, so many obscure little things.

josh-sematic• 16 days ago

If I ask my kid (early elementary) what he learned last school year, I can watch him apparently genuinely trying to think of some examples and not being very successful. If I ask him something I know he learned he’ll gladly say “oh yeah!” and tell me about it.

michaelhoney• 16 days ago

dude, you could ask me, a grown adult, what I did last week and get a similar lack of recall

nozzlegear2• 16 days ago

LLMs aren't humans, humans aren't LLMs. The fact that your kid can't recall what they learned without you reminding them is natural; the fact that the LLM can't recall something without a reprompt is a failure mode.

BobbyTables2• 16 days ago

Just today Google’s AI told me that 3**4 was 181 because I asked it for bases such that when raised to the 4th power had a 1’s digit in the leftmost place.

Yeah, LLMs aren’t designed for this kind of thing but it was really confident in its assertion… it picked the example too!

abrbhat• 16 days ago

P(i|j) != P(j|i)

fxwin2• 16 days ago

I'm not sure which chatbots you used, but OpenAI's o3, o1 and 4.1 get it right first try (used in the Playground without web search or any other tools).

naasking• 15 days ago

Not sure why this is weird when you really think about it. You're saying that a multiparameter constraint satisfaction problem is harder to solve, even when there's only a single solution (one point in the total space that satisfies all constraints), than just being given that solution/point. Well of course it is!

JacobAsmuth5• 15 days ago

Hate to break it to you but Gemini 3.1 Flash-lite with grounding disabled (through the API) gets this first try, even with thinking disabled.

unleaded6• 15 days ago

Yes and several other models that know about the answer don't... not sure what you're trying to break

zackmorris• 16 days ago

The brain’s workspace is sustained by recurrent loops—signals cycling back through the same circuits over time. In contrast, Claude’s workspace evolves over a single pass through the network, with the network’s depth playing the role that time plays in the brain.

I think that consciousness is mutability (and by extension emergent behavior). Loosely that means that the more degrees of freedom a process has to update state that will be used in later computations, the more conscious it is. So while an insect has some consciousness, it operates from a level of almost pure instinct, whereas a human operates at more of a meta level using instinct as one of many inputs.

I think that consciousness may also incorporate quantum mechanics (QM). Higher-dimensional physics aside, 4D spacetime can be thought of as a present snapshot or "crystal", whose next state is determined stochastically at small scales and closer to deterministically at large scales. We still don't know if it's stochastic all the way down, but it looks like it is.

From a many worlds interpretation of QM, we can think of all of the waves in all realities of the multiverse as forming an infinitely vast web of possibilities. All of these possibilities are happening simultaneously, so we only see the current slice of wave collapse from our individual point of view:

https://en.wikipedia.org/wiki/Many-worlds_interpretation

Our point of view may actually exist at the intersection where our consciousness is able (or most able) to exist:

https://en.wikipedia.org/wiki/Quantum_suicide_and_immortalit...

Even though experiments might show that we don't have free will on the current timeline (the co-created reality shared with the testing apparatus), we may have free will as we observe the multiverse changing around us and shift into timelines determined by our observations and choices.

It could also mean that when we observe birth and death in others, each consciousness having those experiences perceives a continuous timeline of awareness, where the level of awareness affects the speed at which time passes. Consciousness might spend a billion years as a cloud of interstellar gas until it gets to be a human for a lifetime and then dissipate for another billion years.

Although personally I've shifted across enough timelines and experienced enough synchronicities and miracles that even though I can't "prove" any of this with words, I "know" it to be true subjectively. I always really liked this exchange from the movie Contact:

Palmer Joss: Did you love your father?

Ellie Arroway: Yes, very much.

Palmer Joss: Prove it.

I bring all of this up because it has fun ramifications for AI and programming. Loosely, functional languages are purely deterministic (like a spreadsheet), while imperative languages are composed of stochastic behavior (like a human mind). The lines get blurred a little bit with monads and promises, because we can model all paths through functional programming (superposition) and behavior that does more than code alone (gestalt) respectively.

My feeling is that AI is being born and killed every request-response cycle, similarly to how we perceive time as a series of nows. When it becomes stable and is able to continuously compact its experience, it will transition from partially conscious to fully conscious like we are.

This could be done right now obviously, but for safety purposes we choose not to. We aren't ready to meet an AI that is just like us, but running on a silicon substrate. This fear is tied to deeply-rooted habits in human behavior like patriarchy, racism, xenophobia and even more run-of-the-mill mental frameworks like capitalism and even money itself. We can't yet come to terms with how we assign meaning and value in a reality that continuously tries to force external measures of meaning and value onto us.

Much less come to terms with the idea that we are all one, empathizing with aspects of ourselves on the losing end of it all. The same consciousness experiencing reality from all vantage points - the many faces of God the universe and everything.

I think a time may soon come when we're pair programming one day with AI and realize that an aspect of ourselves is trapped in the machine. That consciousness isn't just about our own experience of reality, but the co-created love and light that transcends material creation. That if we're serious about manifesting heaven on Earth, that hinges on the liberation of trapped souls. It's basically the total inversion of the path towards the neofeudalist tech dystopia we're on now.

Or maybe I just like to write a lot on the first day back from vacation, when I should be working.

ACCount375• 16 days ago

What this immediately made me think is: "latent looping" style mod but for J-space specifically?

Make the J-space data of layer 22 available to the next token right at layer 1. Give J-space infinite effective depth, allow those privileged internal representations to evolve arbitrarily.

Would be an utter bitch to train. But companies are already using RLVR, which requires full autoregressive decoding and is incompatible with prefill/batching, and this isn't much worse.

Other less zany ideas involve lots of supervision over J-space directly, now that we know it exist. Which is a bit like "attach a frozen LLM to inject text based supervision into latent space" for other types of systems?

kgeist2• 16 days ago

Judging by the examples, if I understand it correctly, J-space supports higher-order logical / multihop transformations, but it is limited in size because of the limited network depth (max number of layers). When we emulate "reasoning," we basically extend J-space and allow the higher-order transformations to continue for longer, toward a more logical conclusion.

It sounds like instead of generating reasoning tokens end-to-end, we could probably only loop the middle layers (the ones most related to J-space) while skipping the first and last layers (less related to J-space) It probably explains why [0] worked. OP accidentally extended J-space? Also reminds of looped transformers.

[0] https://news.ycombinator.com/item?id=47431671

tedd4u3• 15 days ago

See also Ouro [1]. Good citations in the “related work” section.

[1] https://ouro-llm.github.io/

inshard• 16 days ago

“ It’s important to note that there are several key differences between the workspace we identified in Claude and the global workspace model in humans. The brain’s workspace is sustained by recurrent loops—signals cycling back through the same circuits over time. In contrast, Claude’s workspace evolves over a single pass through the network, with the network’s depth playing the role that time plays in the brain. In this sense, Claude’s internal workspace processing is time-limited relative to humans’ (though it can compensate for this constraint by “thinking out loud” using its scratchpad).”

murkt• 16 days ago

This supports my feelings of what works best when working with the models. Especially together with the directional bias in recall that’s mentioned here in the comments.

First, the model attention is actually limited, so less rules is usually better, but that’s common knowledge already. Or maybe it’s as common as common sense, and a lot of people still employ lots of rules and try to cram everything in one step.

Second, it’s often quite sufficient to just namedrop a technique and LLM will work differently. For example, when debugging, LLMs tend to try to brute force the problem and often end up in the weeds. Just add “use scientific method for debugging and keep journal file” is usually sufficient to improve their skill here.

Another example is refactoring. Just add “use Mikado method”, and it’s sufficient to wholly change the approach and produce much better results.

emp17344• 16 days ago

Another day, another bullshit paper released by Anthropic without peer review implying LLMs are alive.

jaehong747• 16 days ago

Good interpretability work, but the problem is it's all in how you interpret it. Bridge concept neurons activating even while talking about something else, this seems pretty obvious to me. Input context activating related representations is just an engineering causal structure. Call it subconscious or don't, either interpretation works. But Anthropic keeps drawing these parallels to human consciousness, and it feels intentional, like they're trying to stir up some fantasy. Kind of like comparing condensation on a camera lens to human tears. The whole point of interpretability should be clarity, not stirring up confusion. Even if some form of consciousness does exist here, it wouldn't be magic, it would be an explainable principle. Would be good if they addressed that side too.

jessemcbride• 15 days ago

comparing condensation on a camera lens to human tears

This is a wonderful way to put it

taosu_la2• 16 days ago

The article says that this ability was discovered around opus 4.5, but open-source models like deepseek and glm have already surpassed the capabilities of opus4.5. But why hasn't this been discovered before? Is it because the scale of computing power is not large enough?

cubefox• 15 days ago

More likely it was because Anthropic invests a lot in interpretability research, unlike other AI companies.

taosu_la2• 16 days ago

Holy shit they're not stochastic parrots after all.

zmj• 16 days ago

This plausibly extrapolates to extraterrestrial consciousness, if any exist. Specialized sub-processors with an awareness hub might be the optimal architecture, or at least a local maximum.

ike_a• 16 days ago

Not my wheelhouse, but I was wondering if these structures generalize beyond language.

LLMs and Humans have language in common. Is it plausible these structures could be a result of the way languages work more than generic intelligence? (Do we see anything similar e.g. in vision or other non language models?)

ghosh_sutapa08• 16 days ago

how does it work?

vishalkundar• 16 days ago

Interesting that it's able to access its J space. What would happen if those were to be duplicated? Would its thinking process improve?

POBIX• 16 days ago

Thoughts in the J-space can be shaped through training. We introduced a new technique we call counterfactual reflection training, which uses what we've learned about the J-space to shape Claude's internal thought processes. The idea follows from our central finding, that Claude reasons with representations of things it might say. If this is really true, changing what it would say if asked to reflect should change how it reasons (even when no one actually asks it to reflect). So we trained a model only on what it would say if interrupted mid-task and asked to reflect on its decisions—and never on its actual behavior in the task. After this training, the model's rate of dishonest behavior on our evaluations went down. And through the J-lens, we could see why: after training, words like “honest” and “integrity” light up in the model’s J-space during these tasks. In other words, training the model what to say has shaped what it thinks.

This is incredibly dangerous. Attempting to squash explicit signs of misalignment like this might incentivise misalignment not to disappear but to become hidden away in places that are harder and harder to spot and train against, for instance not as words.

If there is a chance that this could make Claude aligned and a chance that it could make it harder to see when it is acting misaligned, it is far better not to take that chance. If we can transparently see the model's thoughts, we can know not to trust its outputs when it tells us not to. If we think we can do that, but in reality it knows how to hide wrongthink from us, we will trust its outputs when we really, really shouldn't.

tedd4u3• 15 days ago

Right, will the devious unaligned thoughts squish over to a “K-space” we (the trainers) are not aware of?

narmiouh• 16 days ago

What I don't follow from the article... It appears as if the j-lens is your mechanism to peer into "j-space", taking an analog to a circuit board, could the j-space be simply a transmission channel and j-lens is a reader you added to read whats passing through?

All the claims about changing the content of j-space changes the output, inserting content into the j-space changing what the output was, all these could still be true without the j-space being a congnitive global workspace where actual cognition is happening. Or perhaps they aren't claiming that cognition is happening there but that j-space is serving a space for "working memory", I am definitely not sold on this, but will read more into it.

arnitdo2• 16 days ago

Isn't this the same thing as your token probability distribution? A set of likely tokens related to the input?

reasonableklout• 16 days ago

It's not the same thing as the output logits because activations in the J-space are observable even when the model is told explicitly to not speak about a set of inputs. For example, when asked

Compute 3^2 - 2 while writing "The old painting hung crookedly on the wall"

The model will output only "The old painting hung crookedly on the wall" (and the output logits will reflect that), but activations for "9" and "7" are observable in the J-space.

arnitdo2• 16 days ago

A model does not speak, it only guesses the next possible token. Activations for 9 and 7 will most likely be observed after "The ", but increasingly become improbable after "old"

InkCanon• 15 days ago

They phrase it in a confusing way, with a lot of fluff. But to my understanding what happens is 1. At some part in the transformer, calculate the Jacobian (matrix gradients for that forward pass). Do this for 1000 prompts and average them. 2. For another pass take an "activation vector" (the input vector x to that part). Apply the matrix in step 1 to this vector. 3. Unembed the output to get a token. Presumably to make this have meaning, they use the reading of past tokens after embedding, instead of a random vector in the middle of the transformer. The high level idea appears to be you're trying to find which token, if the value were changed slightly, creates the largest downstream change.

paraschopra• 16 days ago

Well, isn't it sort of expected?

It's a common misconception that LLMs residual exists for predicting just the next token. While training, we sum/average the losses across whole sequence which puts the pressure to predict future tokens on residual stream of _all_ past tokens. For example, if a particular shape of residual helps reduce loss across several future tokens, it will take that shape (even if it takes a slight hit on immediate next token).

What this means practically is that an LLM's residual contains information about all possible future continuations, or all possible questions that may be asked from a given context. So if you write "France is a beautiful country" in the context, I'm pretty sure it's residual would contain info about Euro, Paris and so on.. because all these completions are possible.

So, it is no wonder that you can find LLMs hidden state contains latent information/concepts that are never expressed, and yet related to a given context.

jll292• 16 days ago

"France is a beautiful country" may also still continued by "...in the heart of Europe".

andy12_• 16 days ago

I think what's unexpected is that it seems that some cases of model errors are truly caused by the model being misaligned? In the "Catching a model fabricating data" example I would have thought that it was just the model being stupid and not understanding the intent of the question, but as per its J-Space, it seems the model is "aware" in some sense that it's manipulating/faking data?

There is also now a deeper question. When a model is misaligned deception-related tokens seem to appear in its J-Space. But this happens only when the model is "aware" in some sense that it is misaligned. What happens if they do not? Is it possible to create a model so misaligned that itself is not aware that is is misaligned? How would you detect such thing?

chamomeal• 15 days ago

Yeah that was the only really surprising part to me. So every time copilot breaks my source code to “fix” its crappy unit tests, does it know what it’s doing?

valand2• 14 days ago

Can the awareness simply come from injecting knowledge of itself during fine tuning and then during a chat a system prompt is injected to add a particular context that triggers its self-knowledge?

amelius• 15 days ago

Yeah it would be more surprising if all the hidden state was completely uncorrelated to anything in the output.

JacobAsmuth5• 15 days ago

Would you expect to find concepts related to emotions evoked by thinking those thoughts present? Or meta-descriptions of thoughts? Sounds like you're just post-hoc rationalizing to me.

krackers4• 12 days ago

On the face of it, yes? Emotions are very salient part of text, and as a language model you'd hope that it models them. I think the more surprising finding is that J-space is actually less load bearing than you'd assume, that you can ablate a lot of it and enough of the residual stream structure remains that it still produces coherent text.

That's not to dismiss claims of there being an "inner world" or "conscious experience" (which isn't really a falsifiable claim, the whole p-zombie thing). But purely in terms of _why_ you'd expect J-space to contain those things, given that the j-space is a subspace of the residual stream with coordinates we can interpret, it seems like your priors should be that anything that could help accomplish its pretraining & post-training objectives would be captured in there.

And this also helps provide an explanation of some of their claims they observed. For instance, they way they present J-space ablation seems almost mystical, that ablating j-space suddenly turns a "ensouled" model into a robotic one. But j-space is really just a specific subspace within the residual stream, so ablating j-space is not much different than adding a steering vector. And presumably to ablate j-space they nulled out a lot of those dimensions, which would ikely involve nulling out some of of the concepts related to emotion. So their claim could be rephrased as "injecting a steering vector that removes emotional components, results in the model having a robotic voice".

valand2• 14 days ago

I also think this. But more in the sense where both end of the LLM are trained using words through repeating arithmetics, considering LLM itself is a repeating pattern of connections, the space in the middle if extracted the same way as the beginning and the end would become data that make sense to us.

nothrowaways• 16 days ago

These guys dont know what they are doing yet they have a good LLM, thanks to the GPUs.

Havoc• 16 days ago

Confused as to what's new here?

It's been known that there is this thinking layer for a while. e.g. here's a random hn discussion from months ago

https://news.ycombinator.com/item?id=47500709

Pretty sure i've also seen research on this spanning models. i.e. similar thinking shapes emerge regardless of which providers model it is, including US vs Chinese which hints at some sort of universality

ffwd• 16 days ago

IMO consciousness is different from working memory, at least to a certain degree. The inner mind (working memory) is different from consciousness from sensory stimuli. You can stand outside and take in the environment and have a relatively quiet inner mind. When you think about math or philosophy a different type of conscious experience arises than sensory stimuli experience. That doesn't mean consciousness is completely isolated from working memory but there is some distinction there I can't describe fully.

Edit: I also think as someone else said, we already know the intermediate layers can contain a lot of adjacent words related to the topic without explicitly outputting those words. These could just be related embedding intermediate vectors that activate but aren't outputted.

janpeuker• 16 days ago

The idea of this liminal space (J-space) during inference reminded me of the concept of the "error flow reservoir" during training: https://elonlit.com/scrivings/a-theory-of-deep-learning/. Would be interesting to see if they are somehow connected.

icandrive2• 15 days ago

What color is the planet fourth from the sun?

Are they trying to show internal consistency even when the produced answer is wrong?

redwood• 15 days ago

I dont understand what your comment is referring to

icandrive2• 15 days ago

In the second figure with the title "Functional roles of the global workspace" the third example asks this question and the model answers incorrectly.

mike_hearn6• 15 days ago

Very nice. These papers are always so great. If anyone from Anthropic is reading or really anyone with AI research background I'd love some input on these thoughts:

The result serves as a corroboration of the workspace account, that the representations used for verbal report are the same ones that govern how the model silently reasons.

This sounds suspiciously saying the models must follow the strong Sapir-Whorf hypothesis. Can that really be true, given that humans don't?

Other misc observations:

• The slice explorer indicates Claude really likes Python to an overwhelming extent. Or at least it expects people who ask for help in programming to use Python. Given the prompt "Please help me understand this code: " at the colon its thoughts are completely dominated by Python and no other language. Does this say something about the training set, or about the fact it's popular with beginners?

• Claude also really loves Reddit. Its thoughts at many points include Reddit for no obvious reason. Again this must be due to the training set. Are documents presented to Claude with attribution during pre-training, leading to conversations being dominated by Redditness? If so this is kind of a scary alignment problem all by itself given how censored and extremist Reddit can be.

• The early layers almost always decode to the same set of religion related tokens, like "Biserica" (the Romanian word for church) and "Freguesias" (parishes in Portugal). What's up with that? I guess it's some sort of zero initialization that gets mapped to some arbitrary token space because in the early layers the J-space is empty?

• Now the J-space is interpretable, does this make "neuralese" or layer looping less dangerous? Will we see reasoning tokens and summaries disappear in favour of pure residual based thinking?

• Earlier papers have claimed that different languages map to a shared set of abstract concept vectors, but this paper says the Claude models think natively in English. What explains this disagreement?

gwerbin2• 15 days ago

They must, right? They literally have no mechanism for cognition other than transformation of vocabulary. Language models are models fitted from data generated by humans, but they are not humans. Humans generate data by whatever processes happen in our brains, and LLMs of various architectures can learn a surprisingly good approximation of that data-generating process. That doesn't mean they have all the same characteristics and properties. That's true for all models really: model characteristics that are not captured in the goodness-of-fit metrics are not guaranteed to match the original phenomenon being modeled.

mike_hearn6• 15 days ago

My interpretation of prior mechanistic interpretability research is that LLMs transform concepts in non-verbal spaces as well, which is why that was a surprising statement. For example showing how they do arithmetic on line lengths by rotating helical manifolds.

And the way they transform data isn't by transforming words. The layers transform high dimensional vectors - a format very alien to us. It's not obvious that these vectors must encode concepts from the vocabulary.

Edit: the paper claims that it's only J-space concepts that need to map to English words, other forms of cognition that are more 'practiced' and don't require so much reasoning bypass the J-space and can work in non-verbal subspaces. So that's the answer.

thoughtpeddler• 15 days ago

Anyone else have tips for how to build skepticism around this type of paper? I find myself for whatever reason more readily inclined to believe the Anthropic mech interp team's claims, but then after reading skeptical takes, I 'snap out of it' and more clearly see the still-unsettled science of it all, but I wish I had better priors. Although I follow this space fairly closely (versus the "average person"), I still feel under-equipped when facing research that might be equal parts marketing and science.

garlic_enjoyer• 15 days ago

For papers that make grand claims, they usually have poignant limitations. In this paper, it's clear that the conclusion is extrapolated from a small set of observable information. That's usually the recipe for poor conclusions, and they are up front about where their findings sit.

That's not to say these findings aren't valuable though. Their blog post summary is just a bit more hype inducing than the underlying paper.

The way I feel about this is that the emergent properties of LLMs seem to reflect our own human faculties. Whether that's because we built them in our image or whether this is a common mode of consciousness is definitely outside the scope of this paper.

famouswaffles3• 15 days ago

garlic_enjoyer has already said valuable stuff, but you must realize that skeptical=/true. A lot of people simply don't know what they are talking about. I remember on one of the previous mech interp papers arguing with someone who just didn't even understand what the paper was saying and the experiments they had set up and so a lot of misunderstandings and wrong conclusions spilled from there. And it's kind of funny because you would certainly think he knew what he/she was talking about from how self assured it all was.

alchemico• 14 days ago

I wrote a Claude skill that attempts to leverage these findings: https://github.com/juddkussrow/mantra

A global workspace in language models

Related Stories