Reasoning skills of large language models are often overestimated

https://news.mit.edu/2024/reasoning-skills-large-language-models-often-overestimated-0711

by meysamazad • 2 years ago

37 36 2 years ago

news.mit.edu

rco8786• 2 years ago

Because there is no "reasoning" right? LLMs create output that appear to have been reasoned about, but this is not at all reality

nathan_compton• 2 years ago

I'd put it this way: LLMs may reason in the sense that sampling from the distribution of human generated text sometimes reproduces chains of thought that are valid, but this kind of changed sampling is the only compelling model of how they might reason (in my opinion). There might be some kind of "reasoning" in the weights and activations but by construction this is a single pass, feed forward, process calculating outputs from inputs, so if there is reasoning there, it can only be a approximately a single step.

onemoresoop5• 2 years ago

That's right I did this on chat gpt and went off the rails quickly:

  a man with a wolf and a cabbage wants to cross a river/ There is a boat but the man can only bring one more item beside himself at a time. What are the steps the man needs to do in order to cross the river?

The response was:

To solve the puzzle where a man needs to cross a river with a wolf and a cabbage, while ensuring none of them are left together unsupervised (since the wolf will eat the sheep, and the sheep will eat the cabbage), follow these steps:

  1.Man takes the sheep across the river (leaving the wolf 
 and the cabbage on the original side).
  2.Man returns alone.
  3.Man takes the cabbage across the river.
  4.Man leaves the cabbage on the other side and takes the sheep back
  5.Man takes the wolf across the river.
  6.Man returns alone.
  7.Man takes the sheep across the river.
  8.This way, at no point are the wolf and the sheep left alone together, nor are the sheep and the cabbage left alone together.

at which I prompted: I didn't mention a sheep

The response was corrected but wrongly:

  Oops, my mistake! Let’s solve the puzzle correctly with a man, a wolf, and a cabbage:

  1.Man takes the wolf across the river (leaving the cabbage on the original side).
  2.Man returns alone.
  3.Man takes the cabbage across the river.
  4.Man leaves the cabbage on the other side and takes the wolf back.
  5.Man takes the man across the river.
  6.Man returns alone.
  7.Man takes the wolf across the river.
  8.Now, both the man and the wolf are across the river, with no danger to the cabbage.

At step 3 the problem was solved but it kept on going

cmrdporcupine2• 2 years ago

I agree, but apologists will say that that's all the human brain is doing, too.

Myself, I think that's a terrible false analogy, and also have a higher bar for anything to be called artificial "intelligence."

These programs shouldn't be using personal pronouns or statements that imply cognition or esp reasoning. It's deceptive.

AnimalMuppet• 2 years ago

I think nathan_compton is right in describing LLMs as "a single pass, feed forward, process calculating outputs from inputs". That is not all the human brain is doing. Human reasoning is not single pass - far from it.

So I concur that the apologists are wrong about what the human brain is doing, and therefore wrong about the similarity to LLMs.

wongarsu• 2 years ago

In principle you would expect reasoning to eventually emerge. During training LLMs predict large volumes of text, and that text contains a lot of arguments and chains of thought. You can achieve some success predicting these with pattern matching, but the best way to complete arguments at scale is a reasoning process.

But I think we are only scratching the surface of achieving this. Based on my admittedly casual observation of current model capabilities you need a model that's fairly big (closer to 70B than 7B) and trained far beyond what's implied by the Chinchilla metric.

caesil• 2 years ago

Why not exactly? The model weights cannot encode a sort of math engine? The hidden state cannot encode carryover values? Why do we assume these things can't happen at some level?

jasfi• 2 years ago

I agree. The reasoning is there, and becoming more capable every year (across the various models). It's easy to look for limitations, but what was once glaring problems are now much more subtle.

shermantanktop• 2 years ago

Chord fingering is mentioned. I’ve asked ChatGPT questions about drop-2 voicings and it explained drop-2 and then gave examples of drop-3. I told it that it was in correct and it then changed some wording but repeated the incorrect examples.

bckr2• 2 years ago

ChatGPT is terrible with music-related prompts. It might be able to get some basic theory, the type you can read about in Wikipedia. But ask it to do anything beyond that and it will fail pretty hard.

apsec112• 2 years ago

FWIW the version of Claude they use is quite old, v1.3. I'd be curious to see how the new Claude 3.5 performs on these

ArcaneMoose• 2 years ago

This is why I find ARC-AGI so fascinating - it really seems like a great benchmark to test for this sort of behavior. Hope that it doesn't get solved through a brute-force memorization approach. I do wonder how reasoning is different from other sorts of 'thinking' if it's all neurons firing in the brain.

alexsereno• 2 years ago

I’d love to see this placed into the context of the (actual) average Joe. If you consider that the average person dramatically overestimates their reasoning skills and can’t correctly do high school level math, I think it would help understand if we have reached AGI. It would also just provide some color to the state of things.

armitron• 2 years ago

The "GPT-4 Can't Reason" paper [1] [2] is excellent and proves without a shadow of a doubt that there is no reasoning taking place. It also raises strong doubts about the efficacy and reliability of multi-agent approaches that rely on LLM-based planning (such as Cradle [3], which is currently on the front page).

When it was previously discussed here, it received a torrent of low-quality comments that can only be described as confirmation bias: commenters tried the examples verbatim (instead of introducing random perturbations to get around model updates) in order to disprove the thesis.

It's sad that there are "engineers" out there so blinded by their own (wishful thinking, vested interests) that can not accept the obvious.

[1] https://arxiv.org/abs/2308.03762

[2] https://medium.com/@konstantine_45825/gpt-4-cant-reason-adde...

[3] https://baai-agents.github.io/Cradle/

krageon• 2 years ago

I think most people are a little bit tired of hearing that things that are manifestly possible, are instead impossible. If we'd take the time to recognise that the truth is somewhere in the middle (lots of tasks can be done, lots of them cannot), the conversation could at the very least be interesting.

As it stands every publication on this subject is filled with a lot of people that feel they need a polarised stance or they'll immediately be shot: Either they say LLMs are fundamentally broken and can't do shit (this is ridiculous) or they say that LLMs are self-aware and we should give them every single task ever (also ridiculous). It is really tedious to read the same dumb shit every single time.

mewpmewp23• 2 years ago

If any LLM can solve a novel exercise that requires some form of reasoning, wouldn't that be evidence that LLMs are capable of at least some level of reasoning?

Not necessarily AGI or average human level reasoning, but just any form.

cs702• 2 years ago

Paper: https://arxiv.org/abs/2407.02678

Previous discussion: https://news.ycombinator.com/item?id=40899309

Reasoning skills of large language models are often overestimated

Related Stories