Mechanistic interpretability researchers applying causality theory to LLMs

https://cacm.acm.org/news/can-we-understand-how-large-language-models-reason/

by adunk • 10 days ago

117 84 10 days ago

cacm.acm.org

calf• 10 days ago

One plausible reason I thought of that we may not understand neural nets is that by their nature their power grows with ever-more complex connections and weights.

So it is like the opposite of logical systems, in that the very design of neural net architecture is a mess of parameter "spaghetti code" which renders the entire thing a metaphorical encrypted black box. The more powerful an AI/AGI the more this would be the case, and this is analogous a complexity curve.

And so any effort to make sense of such black box computation would be like trying to reverse entropy, analogous to trying to recover information lost in waste heat. And that could be one fundamental barrier to understanding both human and artificial brains alike, relative to their internal complexity.

(Just thinking aloud my handwavy pet theory recently, I am not an expert and could be totally mistaken on this)

hsb3• 10 days ago

You dont have to understand chemistry to be a good cook tho.

gfody• 10 days ago

there's a 2MP about the related paper: https://www.youtube.com/watch?v=l72ufA-4SzE

antleys• 10 days ago

This article is not about "reasoning" in the abstract, philosophical sense but is talking about "mechanistic interpretability" research. The title is more like, "can we understand if the 'knowledge' encoded into a neural networks actually corresponds to reasoning-like concepts" and doing that with actual experiments like tweaking weights and activations.

There's an interesting example where researchers saw a model approached clock time calculations and calendar month-day calculations using the same methodology. So then is this because an underlying concept of "cyclical measures" has emerged in the network?

dang3• 10 days ago

Thanks - I've attempted to put that in the title above, in the hope of representing the article accurately.

(The trouble with a baity title like "Can We Understand How Large Language Models Reason?" is that it generates a barrage of shallow, reflexive responses having little to do with the article. What we want on HN are curious, reflexive responses instead - https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor....)

danbruc• 10 days ago

I personally would not look for the way they reason in the weights, at least not directly. In principle I could replace a large language model with a map from all possible input strings to output token or output token distribution without any weights. I have a hard time imagining how you would even tell, at the level of weights and activations, if the next token being the is the result of some proper reasoning or a hallucination. But those weights do not exist for the sake of it, they encode a lot of text the model has seen during training, and I would imagine this is what drives the reasoning. Can you evaluate the following polynomial ... will be related to To evaluate a polynomial ... seen in the training data. This is the level at which I would look for the reasoning, memorized patterns how to do specific things, maybe with some kind of placeholder variables for generalization. Ultimately such a structure would of course also be represented in the weights but I could imagine that this makes it unnecessary hard to understand. Or maybe not, maybe the learned patterns are so complex that they do not have a simple representation.

BobbyTables2• 10 days ago

I also suspect the learned patterns are not necessarily efficient, though might be by accident.

One could “learn” addition by memorizing a truth table instead of understanding the concept… The truth table itself wouldn’t have much meaning.

dpark• 10 days ago

In principle I could replace a large language model with a map from all possible input strings to output token or output token distribution without any weights.

What is an output token distribution except a set of weights?

dominotw4• 10 days ago

“Mechanistic interpretability will probably never reduce large language models to a few simple equations,” Icard concluded, “but it may gradually turn deep neural networks into systems whose hidden algorithms can at least partly be understood.”

what is the basis for this optimism ?

azakai4• 10 days ago

The optimism is based on the successes so far, some of which are described in this article. Scientists have made progress here.

dominotw4• 10 days ago

no they havent . success so far is totally meaningless and doesn't imply any sort of upward slope .

dang3• 10 days ago

[stub for offtopicness]

[[All: please don't post shallow-generic reactions to baity titles. Those are basically the same thing, a la https://en.wikipedia.org/wiki/Rubin_vase, and we're trying for something more substantive here.]]

JackSlateur• 10 days ago

Do they ?

throw3108224• 10 days ago

Of course they do, how else do you think they manage to implement new features in large codebases, or to prove new theorems? But you don't even have to assume they do because of the results- you can read their chain of thought.

azakai4• 10 days ago

The article answers this question, at least to the extent it can be answered, at this time.

We see some signs of reasoning, but also we understand little about how they work.

arcanemachiner• 10 days ago

Yes, there is an LLM feature that we have anthropomorphized as "reasoning" or "thinking", where an LLM has a scratch space where it can dump tokens that help to improve the final output.

analog31• 10 days ago

Do LLMs have Qualia?

wat100004• 10 days ago

Do people?

chrisjj2• 10 days ago

Clickbait article title.

The article body does not presume they reason.

dang3• 10 days ago

We've edited the title now in the hope of nudging the discussion in a more substantive direction.

otabdeveloper42• 10 days ago

They don't reason.

CamperBob22• 10 days ago

What would change your mind?

CrzyLngPwd• 10 days ago

My toaster doesn't reason, and neither do the current clankers.

CamperBob22• 10 days ago

How'd your toaster do at IMO last year?

warumdarum2• 10 days ago

They dont. They have input that runs through a invisible stochastic canyon. As long as there is previous experience the stochastic canyon never ends. If there is none or isignificant one, or it runs out of tokkens, it hallucinates and the illusion falls apart. There is no reasoning, just the invisible grand canyon of all of human experience and knowledge. PS: try to get it to retell you a clichee movie or book and you can see life near the end, how the delta of all the same movies opens up into wildly different endings.

To advance further it would need the ability to abstract away the general situation shape and pattern recognize similar situations.

red75prime• 10 days ago

Stochastic gradient descent can be likened to traveling down a billion-dimensional canyon. But inference? Hardly.

alchemist1e92• 10 days ago

It’s curious how they solve unsolved math problems without reasoning. Maybe I have a different definition of reasoning than you.

rvba• 10 days ago

There is a streamer who plays Diablo 2 by listening to the AI advice and it is quite funny since it is pretty clear that most of the advice is an amalgamation of random, often incorrect advicem

I wonder if it is the same for programming or not, but I vibe coded an android app just to see if I can and it just works. It required a lot of "build the code and correct the errors" pushing though. For example requested code in kotlin but received something else.

Lomlioto2• 10 days ago

Compression is the trick. Its even philosophed about if compression = intelligence.

The LLM has to compress everyy question/prompt into its system. It does so by creating rules and ways of processing data (this can lead to AGI, world models or an architecture of sub architectures like an LLM + something else). So if it should respond in a way that only reasoning people can achieve, it might be able to learn a representation of what we call reasoning.

It read enough text in itself to even know about the concept of reasoning and how you would do that.

Even if this is only stochastic, it shouldn't be so devalued as your comment comes across.

Who says that we are doing anything more magic?

smokel• 10 days ago

It's probably helpful in this discussion to make a difference between two definitions of reasoning:

1. phenomenal reasoning, requiring consciousness and subjective experience

2. functional reasoning, transforming premises into conclusions using logic

I think you are attacking this using definition 1, whereas the article is obviously aiming at a different type of reasoning, and trying to formalize what is actually going on. It seems to be a genuine effort.

skybrian• 10 days ago

When a mathematician reads a hundred-year-old math paper, they are reproducing in their head the reasoning of someone who died long ago. That is, reasoning can be written down and replicated.

If that works, I think it's fair to say that LLM's are inanimate processes that can generate real reasoning. You can tell when you read it and it makes sense.

There are likely some kinds of reasoning that can't be written down, as well as other forms of understanding, but they also don't replicate nearly as easily.

gus_massa• 10 days ago

With that definition, computers don't play chess, they just move the pieces using some weights and backtracking.

dominotw4• 10 days ago

i love how anthropic puts out some bs like this every few weeks 'we saw some red bridge lights blinking in model weights when someone mentions sfo. Arent they just like us?"

CodeCompost• 10 days ago

No, talking to itself is not reasoning.

1vuio0pswjnm7• 10 days ago

Original HN title: "Can We Understand How Large Language Models Reason?"

michaelhoney• 10 days ago

I also calculate six months from August as 8+6=14mod12. I wonder if anyone does it differently, this seems like the most plausible technique.

ivandenysov• 10 days ago

To calculate +/- 3/6/9 months I shift by seasons. 3rd month of summer becomes 3rd month of winter.

That works well cause all months live in a primitive memory palace in my head: an analogue clock face with July at 12 and January at 6. So shifting by 6 means rotating the clock hand from 11 to 5 and immediately visualising what month it falls on.

This might sound inefficient to an LLM but human brains had image processing before language.

windenntw• 9 days ago

For me: January = up, April = left, July = down, October = right.

Never sure why I did this association, maybe it comes from a drawing in a book I read when I was six or somthn?

Mechanistic interpretability researchers applying causality theory to LLMs

Related Stories