LLMs struggle to explain themselves

https://www.jonathanychan.com/blog/llms-struggle-to-explain-themselves/

by jonathanyc • 2 years ago

50 80 2 years ago

jonathanychan.com

jonathanyc6• 2 years ago

Hi HN! I'm about to go to bed, but I promise to go through comments when I wake up later today. Happy Friday!

The source code for the demo is on GitHub: https://github.com/jyc/stackbee

latexr• 2 years ago

One day in March, I was walking my dog, saw a house numbered 3147, and thought it was a funny pattern. It’s the Fibonacci sequence with a different seed (31 instead of 11).

The Fibonacci sequence with 3,1 would be 3,1,4,5. I think you mean the house number was 1347. That would work and be easier to notice.

jonathanyc6• 2 years ago

Haha thanks for catching that—fixed! That’s pretty ironic. Looks like I’m the LLM and you’re the human :P

Now I can’t remember whether the house was 3147 or 1347. The pattern might have been to add the first digit to the last digit (unrot + in the stack language). That’s what I get for writing at 3am!

beardyw2• 2 years ago

It’s interesting to me that in spite of the fact that the LLM does so poorly with number sequences in general, it does pretty well with variations of the Fibonacci sequence.

Not surprising, since the Fibonacci sequence will be in the text swallowed by the LLM.

jonathanyc6• 2 years ago

Yea! Do you think the LLM learns a Fibonacci-specific circuit? Do you think it is possible for an LLM to learn a general pattern-recognition circuit?

beardyw2• 2 years ago

Well, a basic LLM would not really even distinguish a Fibonacci series from a hole in the ground. But models are being built with extra stuff, though I doubt recognising number series is high on the list.

reasonableklout• 2 years ago

Thanks for sharing! The custom stack-based language that you created for randomly generating interesting integer sequences was the most interesting part of this post for me. Wish the post focused on that rather than LLMs!

EternalFury• 2 years ago

Either you believe in emergent behavior or it’s only very good at recognizing patterns. Which is it? I made a test called LLMLOLinator (stupid name) and I can tell they cannot stray from the probability distributions they learned during training. So, I am not confident in so-called emergent behaviors.

fsndz6• 2 years ago

There are no emergent behaviors; LLMs are essentially memorizing statistical patterns in the data and using lexical cues to generate responses. They cannot explain themselves reliably because they don't know what they know, nor do they know what they don't know. In fact, LLMs don't truly 'know' anything at all. These are not thinking machines—they are simply the result of statistical pattern matching on steroids. That's why there will be no AGI, at least not from LLMs: https://www.lycee.ai/blog/why-no-agi-openai

d_sem3• 2 years ago

And yet I can provide modern LLMs unique never before asked scenarios that require the ability to reason about real world phenomenon and they can respond in ways more thoughtful than the average person.

Much of human education is feeding books of information about things we will never experience in our day-to-day lives, and convincing ourselves it reflects reality. When in fact most of us have not personally experienced any evidences that what we learned is true.

That vast majority of what a person "knows" is a biological statistical pattern matching on steroids.

x3haloed• 2 years ago

I’d like you to explain what you don’t know, and why.

fjdjshsh• 2 years ago

Your post isn't providing any argument, you're just providing statement after statement of a position that is pretty well known (no emergent behavior, just pattern matching) and pretty popular here on hackernews. Maybe the link provides all the good stuff, but I'd suggest you at least provide some flavor of what your arguments are.

mycall• 2 years ago

Have you tried trying an agentic approach, i.e. https://github.com/microsoft/autogen

hun3• 2 years ago

Even humans are bad at explaining their decisions sometimes, especially if they did not reach there by reason.

In fact, if asked for the reason post mortem, people (and LLM) are likely to make them up on the spot. I wonder if the same dynamics is at play here.

threeseed• 2 years ago

LLMs will provide an answer for every question even when told it doesn't know what it is talking about.

Overwhelming majority of people won't do this.

So I don't think the comparison really makes any sense.

adastra226• 2 years ago

Have you used them recently? They have gotten much better about this.

scratcheee• 2 years ago

I’ve met humans who do that too

jonathanyc6• 2 years ago

Yea that’s a good point! On the one hand, it’s cool to look at the JavaScript the LLM generates to test its hypotheses in the logs (you can click “expand” to see them). On the other hand you’re right that especially given that it’s (presumably) an autoregressive LLM, if it writes the choice before the pattern there’s no way for the pattern to influence the choice.

You can muck with the prompts by clicking “Settings”! I think there’s a lot of room for improvement with all aspects of the experiment.

One prompt I tried made the LLM take tons of turns, but it ended up with a weird fixation with assuming everything was some sort of approximately geometric sequence and would always conclude none of the options matched. Before I gave it the eval_js tool it still did a pretty good job making choices, but the explanations were worse.

lelandfe• 2 years ago

Unaccountably correct is not a frequent quality I see in humans

skywhopper• 2 years ago

This is because LLMs do not reason. They pattern-fit. The fact that that solves a lot of things humans often use reason to solve most likely speaks to the training data or unrecognized patterns in standardized tests, not to LLMs reasoning capability, which does not exist.

To excuse their assumption of reasoning capabilities, the author in the FAQ snarkily points to “research” indicating evidence of reasoning—all of which was written by OpenAI and Microsoft employees who would not be allowed to publish anything to the contrary.

It’s a shame people continue to buy into the hype cycle on new tech. Here’s a hint: if the creators of VC-backed tech make extraordinary claims about it, you should assume it’s heavily exaggerated if not an outright lie.

quantadev14• 2 years ago

LLMs can't explain their past behavior because any time you ask them to, all they're really doing is reading the previous context from a previous inference and looking for any reason they can think of that they might have said that, so to speak.

That is, they have absolutely no genuine recollection of what they were thinking at the time they said something in the past. Even with "Tree of Thought" approaches all you're doing is recording past conversations, states, and contexts, and your new inference asking for the "justification" of that, will be a similarly totally fake justification, because as I said they have no memory but only context.

In my own app I can switch to a different LLM right in the middle of a conversation and the new LLM will just continue to always think it said everything in the prior context even though that's not the case.

Terretta2• 2 years ago

the new LLM will just continue to always think

Even when you know, as you do, it's tough to avoid such characterizations.

01HNNWZ0MV43FF• 2 years ago

It might be fair to say they think but they can't directly introspect the thinking process. They can only confabulate reasoning post-hoc, or use science to try to understand themselves from the outside. Same as humans.

quantadev14• 2 years ago

I didn't use the word "think" in the human sense of the word, or even in the consciousness/qualia sense of the word. I of course only meant inference (Matrix multiplications)

What I like to say however is that LLMs are doing genuine "reasoning"; and what the emergent "intelligent" behavior of LLMs has proven to mankind is that "reasoning" and "consciousness/qualia" are separable things (not identical), and I think as recently as 2021 we didn't really know/expect that to be the case.

resonious• 2 years ago

Split brain experiments show there's a good amount of evidence that humans also largely operate in this way.

quantadev14• 2 years ago

You're right. Humans have short term memory, so we can genuinely recall our reasoning, but at the same time, once an action (or even a thought) has been had, we begin weaving a new narrative about why we did that thing or had that thought, that often has no resemblance to actual truth.

greyface-• 2 years ago

Split brain experiments show that humans missing a functional corpus callosum largely operate in this way. I think it's an open question whether all humans operate in this way, or if communication between the hemispheres is necessary for 'real' explanations of behavior. The data we have at this point would support either conclusion.

samtho• 2 years ago

This is very reductive and gross misrepresentation of split brain experiments.

Humans have various level of memory (working, short term, long term), the ability to experience meta cognition, and the autonomy to make a decision - all of which (and much, much more) informs our behavior and are also things that an LLM fundamentally does not have.

These “studies” are painfully naive at best and are at worse an intentional misinformation campaign used to bolster the value of LLM past what they are useful for.

owenpalmer• 2 years ago

That is, they have absolutely no genuine recollection of what they were thinking at the time they said something in the past.

I really like this point. Imagine if model observability got to the point where a model could self-observe after running a response. Not only would it read the context, it could observe it's own thoughts from the past.

mattmaroon• 2 years ago

To be honest, there is a lot of reason to believe that this is more or less how humans work also. We might be better at figuring out why we did something, and we might be more deliberate in some cases. But for the most part I think and then logically justify their actions.

igornadj• 2 years ago

Isn't this covered in the first FAQ in the OP?

mandevil2• 2 years ago

Almost a decade ago I ran into a couple of problems where our logs started to change qualitatively on a SaaS tool I was supporting, but not in a way that printed more error level messages in logs. When someone complained about incorrect behavior in our code we could see clearly in the logs that the wrong code paths were being engaged, and when that started- people had carefully logged everything that was happening! They just hadn't logged it at the "Error" level so we never got alerts about it. And it would have been impracticable to have that level of alerting, because the message wasn't an error, it was just that due to other code changes we were now going down the wrong path for this span.

So for a hackweek I built a tool to tokenize all of our log messages, and then grabbed all of our logs and built a gigantic n-dimensional vector for every five minute chunk of two days of those logs, then calculated the pythagorean difference for each of those five minute chunks, and looked at the biggest differences, most outlier five minute chunks. And they were all from 8-8:30AM CET on the two days (our company and most of our customers were US based, I just was looking at what timezones matched up to the interesting time). I said "okay, this looks interesting, let me see what is happening in the logs then" but it was impossible to figure out what the statistics were seeing. Because the math thinks in ways that human brains don't- it views the entire dataset simultaneously, and human brains just can't keep five minutes of busy log files in their working memory, but humans build narratives and the math can't understand that. So I ended up getting frustrated and giving up on the project. Because explaining in terms that I could understand and start debugging was the whole point of the project!

jonathanyc6• 2 years ago

Wow, I'm actually really interested in how you did that. Did the differences between chunks correspond to changes in word/token frequencies? Or did you embed the logs into some other space first? This is a pretty nifty idea; the only analyses I've ran on production logs were just simple SQL queries.

mandevil2• 2 years ago

I wrote a python script that parsed all of our back-end code, looking for every log message, and used that as the basis for a giant n-dimensional vector representing every possible log message. Then I looked at every five minute chunk of our logs, and incremented the appropriate row in the vector for every message printed, then I used a clustering algorithm (don't remember which one, sorry, but it something from PyTorch I'm pretty sure, not at this company any more so I don't have the code any more) to identify the five vectors that were farthest from any other vector. I first computed the n-dimensional pythag difference between each pair of five minute chunks and selected the vectors with the highest combined difference but the true clustering algorithm was a cleaner choice.

I did throw away a couple of messages, basically all the traffic from our uptime and health checks I threw away because they seemed like they would distort the data (if our health-checker went down that was the health-checkers fault, and we ought to get an actual alarm, having this alert would just be a duplicate alarm). Dunno, that might have been a mistake. It was a hackweek project, I'm not saying this was perfect- in fact, as I said, it never provided anything useful because it couldn't be explained!

In our actual log files on the two days I looked at, all five of the outlier five minute chunks were from 8-8:30 AM Central European Time (our logs were in UTC, I just looked at different time zones and that seemed the most likely source of interesting behavior). And then when I looked at those times in the logs- and also when I eyeballed the ~1000 dimensional vectors for them- I couldn't tell what the clustering algorithm was seeing, because it was 'thinking' so differently from how I do. It wasn't like one of the rows in the vector was suddenly 150 and then went to 0 outside that half-an-hour, it was hard to see any patterns, So I couldn't set this up to alerts or anything, because it would have produced a whole lot of wild goose chases without a lot of further refinement.

Talking with someone more expert in ML than I, he recommended trying to fine-tune a LLM to predict the next word (or maybe even the next log message depending on windows and verbosity of log messages) and potentially alerting when the differences between predicted and actual got too large. Maybe that would work, I dunno. But that was what I would have tried next hackweek, if I hadn't moved on from that company.

distortionfield• 2 years ago

The house number you saw is actually part of the Lucas sequence. it’s related to and is a good approximation at large numbers of the Fibonacci but at small numbers becomes distorted.

https://en.m.wikipedia.org/wiki/Lucas_number

jonathanyc6• 2 years ago

Ha, this is neat!! I’d never heard of the Lucas sequence before, thanks for the new knowledge!

anon291• 2 years ago

In some ways it's obvious why. An LLM produces a probability distribution and then a random word is sampled.

Imagine if you said to a secretary that you're 60% yes and 40% no, and she arbitrarily decided to write NO in your report and then a day later the board asked you why you made that decision.

You'd be confused too.

LLMs struggle to explain themselves

Related Stories