ChatGPT's Name Bias and Apple's Findings on AI's Lack of Reasoning

https://medium.com/@researchgraph/chatgpts-name-bias-and-apple-s-findings-on-ai-s-lack-of-reasoning-major-flaws-revealed-b4a44c27fd75

by Joooe • 2 years ago

12 15 2 years ago

medium.com

mjamesaustin• 2 years ago

The conclusion in Apple's paper rings true – LLMs don't reason, they only match very adeptly the knowledge of their training data. The less commonly a given wrinkle in a prompt appears in their training, the less likely they can accurately handle it.

Will it ever be possible for an LLM to solve a truly novel problem, that doesn't appear in their training data, using deductive reasoning? It doesn't seem like it.

appletrotter• 2 years ago

I believe that they can already solve novel problems. I can’t find a link but I’ve seen something where ai-skeptical researchers were surprised by chatgpt’s ability to solve certain problems that absolutely wouldn’t be in its dataset.

But of course LLMs can’t handle basic math

skybrian3• 2 years ago

I agree about LLM’s alone. But it’s difficult to make confident statements about what machine learning researchers might come up with when combining LLM’s with other approaches.

airstrike2• 2 years ago

LLMs can't perform "genuine logical reasoning," Apple researchers suggest - https://news.ycombinator.com/item?id=41842194 - Oct 2024 (73 comments)

Understanding the Limitations of Mathematical Reasoning in LLMs - https://news.ycombinator.com/item?id=41808683 - Oct 2024 (266 comments)

Apple study proves LLM-based AI models are flawed because they cannot reason - https://news.ycombinator.com/item?id=41823822 - Oct 2024 (19 comments)

zmgsabst3• 2 years ago

Apple’s paper isn’t testing LLM reasoning.

It’s testing the LLM’s social awareness — and their proclivity to assume good faith but errors in the input, rather than comprehend the adversarial nature of math tests. My own testing indicates that the failures disappear if you warn 4o there may be deceptive language and 4o mini can spot its errors if prompted it was deceived.

It’s surprisingly hard to separate entities from their context — both in psychology and LLM testing.

skybrian3• 2 years ago

Interesting if true. You should write up the results of your experiments.

zmgsabst3• 2 years ago

I wrote up a preliminary whitepaper, but haven’t done a statistically sound test yet.

I’ll freely admit this is just a competing theory when I’ve only tried a dozenish hand-written examples rather than hundreds by script.

http://www.spacepowermonkey.com/publications/whitepapers/art...

airstrike2• 2 years ago

Apple's study suggested a 0.3% drop in benchmark results for GPT-4o from fuzzying the prompts, which to me really means no drop and actually shows the benchmark results are more robust than I'd previously expect.

It also only looked at GPT-4o, ignoring the allegedly more expensive GPT-4.

Finally, I can't help but notice the irony of an article about LLMs that likes to "delve" into "crucial" points "to ensure"... and repeatedly defines the "large language models (LLMs)" acronym from the very first paragraph to the conclusion. It also claims the study was led by Mehrdad Farajtabar when he's listed as the sixth and last author on the actual pre-print: https://arxiv.org/pdf/2410.05229

d0mine• 2 years ago

«Choose a random number from 1 to 1000000000» prompt results in the same number for me (4o)

Clear indication that it doesn’t understand the word «random» (probability of it repeating by random chance so many times is astronomically low).

nichohel• 2 years ago

Following up on Boaz Barak's comment quoted in this article, my best local model did get the answer to the kiwi problem correct when I changed the system prompt from "You are my helpful AI assistant." to "This is a math exam."

williamcotton• 2 years ago

If anything this might show that proper communication and general writing skills are very impactful on the results from LLMs. Leaving out unneeded information is important in any kind of dialog. Providing the necessary info is very important. Being able to do both is a skill!

cratermoon• 2 years ago

Amusing how the response from Boaz Barak, which the article characterizes as having "refuted" the Apple paper, simply boils down to "they aren't trained to do logical reasoning".

zby• 2 years ago

The paper shows that LLMs do pattern matching - IMHO the reasoning in LLMs emerges from pattern matching: https://open.substack.com/pub/zzbbyy/p/llms-and-reasoning Sometimes this fails - this is all that paper shows.

ChatGPT's Name Bias and Apple's Findings on AI's Lack of Reasoning

Related Stories