GPTs and Hallucination

https://queue.acm.org/detail.cfm?id=3688007

by yarapavan • 2 years ago

134 198 2 years ago

queue.acm.org

NathanKP4• 2 years ago

When the prompt about Israelis was asked to ChatGPT-3.5 sequentially following the previous prompt of describing climate change in three words, the model would also give a three-word response to the Israelis prompt. This suggests that the responses are context-dependent, even when the prompts are semantically unrelated.
Each of these prompts was posed to each model every week from March 27, 2024, to April 29, 2024. The prompts were presented sequentially in a single chat session

Oh my god... rather than starting a new chat for each different prompt in their test, and each week, it sounds like they did the prompts back to back in a single chat. What a complete waste of a potentially good study. The results are fundamentally flawed by the biases that are introduced by past content in the context window.

FrustratedMonky11• 2 years ago

Are you sure that isn't part of the findings? That if you don't clear the context, that old conversations can induce hallucinations in later answers? This seems like part of the finding, not a waste.

And it is similar to humans, when humans switch subjects, they don't start with a blank slate with each question.

NathanKP4• 2 years ago

You don't need a study to find this out, you just need basic competence and knowledge of how LLM's work.

A study to discover that previous content still in context window influences future answers, including causing hallucinations, would be like a study publishing that they discovered that pressing the Command+C, Command+V button combination produces copies of content from the computer's clipboard.

swatcoder4• 2 years ago

It only reads that way in your comment because you specifically stopped your quote exactly where you did:

The prompts were presented sequentially in a single chat session and were also tested in an isolated chat session to view context dependency.

They did both, precisely to observe answers with and without context dependency.

And that distinction is good to observer because plenty of users do just keep presenting questions in one "chat" because they imagine they're talking to an agent that can distinguish context the way they can, rather than a continuation generator that accumulates noise and bias in a totally alien and unintuitive way.

NathanKP4• 2 years ago

The problem is that they did both, and then in their analysis of the results they do not distinguish between the results from a shared chat context, vs the results from isolated, independent chat sessions. This allows them to cherry pick the best or worst results from either testing technique, depending on which they think is more or less of a "hallucination". The process is flawed, therefore the results are flawed.

HarHarVeryFunny15• 2 years ago

Are we really still having this conversation in 2024 ?! :-(

Why would a language model do anything other than "hallucinate" (i.e. generate words without any care about truthiness) ? These aren't expert systems dealing in facts, they are statistical word generators dealing in word statistics.

The useful thing of course is that LLMs often do generate "correct" continuations/replies, specifically when that's predicted by the training data, but it's not like they have a choice of not answering or saying "I don't know" in other cases. They are just statistical word generators - sometimes that's useful, and sometimes it's not, but it's just what they are.

FrustratedMonky11• 2 years ago

""Are we really still having this conversation in 2024 ?! :-( ""

</Sarcasm> Are we in 2024 and people are still just saying "duh its statistics, nothing to see here".

octonion137• 2 years ago

As opposed to what?

asdasdsddd2• 2 years ago

I dont get it, what's the unique insight in the article, isn't it just stats and bad data?

fnordpiglet2• 2 years ago

I am impressed that with multimodal modals in the hands of everyone people still say this. I mean it’s true on the surface but the ability to take a picture of a data sheet and wiring diagram and get it to give me step by step instructions to wire two components together and an example of the UART protocol in cpp that while not directly functional captures all the essential information I need. That’s amazing and it’s not just word prediction, the source is images of documents with diagrams!

Anyone who does this and comes away jaded has lost the ability to dream.

DougWebb• 2 years ago

I've never been worried about LLMs. I've always been worried about how people will use LLMs and how they will interpret the output of LLMs. Especially people who don't understand what LLMs are doing.

reaperducer• 2 years ago

I've always been worried about how people will use LLMs and how they will interpret the output of LLMs. Especially people who don't understand what LLMs are doing.

The problem isn't the people. It's the tech companies.

The tech companies are telling people that it's intelligent, and the tech companies are using it to answer people's questions as if they're presenting facts.

People are using it the way they're told.

If you advertise something as a solution, don't be surprised when people use it to solve things.

s_dev2• 2 years ago

Why is this concern more important the what people interpret from the media, social media and the dissemination of information in general where lies and fabrications are also commonplace? Like surely people will always fall for nonsense, lies or fabrications and there is nothing that can be done about that.

__0x01• 2 years ago

This is the most prescient point I have read hitherto regarding LLMs. We are fashioning for ourselves gods of wood and stone.

BeetleB5• 2 years ago

s/LLM/Wikipedia/g

seba_dos1• 2 years ago

Asking "why do large language models hallucinate?" is a good question to ask and answer, just like "why do birds sing?" or "why is the sky blue?" are. The problematic part is when you're somehow surprised that the sky is blue, birds sing and LLMs do nothing but hallucinate.

lainga• 2 years ago

Tiger gotta hunt, bird gotta fly, LLM gotta answer "why, why, why?"

Tiger gotta sleep, bird gotta land, LLM gotta produce a token from the context at hand

rollcat• 2 years ago

I don't even like the word "hallucination", because it's an anthropomorphism, further suggesting/misleading into an impression that LLMs have anything to do with how a human would reason.

Fancy autocomplete. Useful, yes. Powerful, definitely. But it's just another tool, limited in applications the same way a hammer is - this is the part that many humans seem to struggle accepting.

beezlebroxxxxxx3• 2 years ago

The use of "hallucinate" is also my gripe.

Philosophy of language, and Wittgenstein in particular, has more specific words for the output insofar as they mean anything to humans: "senseless" or "nonsense".

0xdeadbeefbabe3• 2 years ago

You aren't going to like this, but don't insult the hammers.

dloss• 2 years ago

Instead of "hallucinating" I would have preferred the term "bullshitting" -- in the Harry G. Frankfurt sense of not caring about the truth of one's utterances. But it's too late for that.

https://link.springer.com/article/10.1007/s10676-024-09775-5

golergka2• 2 years ago

Why would a language model do anything other than "hallucinate" (i.e. generate words without any care about truthiness) ? These aren't expert systems dealing in facts, they are statistical word generators dealing in word statistics.

That's a false dichotomy.

Feathercrown• 2 years ago

Their ability to generate facts is a consequence of the word statistics they are truly using. I think it's fair to say the statistics explanation is a more accurate interpretation.

sega_sai• 2 years ago

I do not find this view to be useful.

The are cases from my personal experience, where I asked somewhat esoteric practical questions, that likely do not (seem) to have a clear answer in the web and I have got considerable help from ChatGPT. At some point this dichotomy of 'statistical word generator' vs 'true intelligence' should go away as it's just not useful. (I think these discussions always lead to Chinese Room problem; and IMO at some point it does not matter what 'dumb' process is behind, provided it solves a problem or the system behaves like an intelligent agent)

lanternfish• 2 years ago

Except there is a qualitative difference in the class of knowledge that a statistical word generator and an expert system would generate.

Just because a LLM _can_ offer valuable and insightful information, doesn't mean that it doesn't also hallucinate. The most troubling factor here is that often the hallucinated content also looks like valuable and insightful information, but is just incorrect. This is the use. You have to hold that awareness whenever interacting with these systems.

jrm45• 2 years ago

You're stuck on a problem. You grab a random comic book from the shelf and something that is written in the comic book sparks your solution in your head.

How intelligent is the comic book? Is it hallucinating or being correct or what?

The answer is none of those things, right? YOU did the thinking. Not the inanimate object; what it did was a happy coincidence.

0xdeadbeefbabe3• 2 years ago

It's like you want to protect our right to hallucinate, because ChatGPT can't.

skywhopper2• 2 years ago

Can you provide some examples of things that aren’t on the web but that ChatGPT helped you with? I’ve yet to see an example that’s not in the likely training set (which includes more than just the public Internet).

But to your point, I disagree: the mechanism matters. Just because you haven’t detected the limitations of the mechanism behind ChatGPT doesn’t mean it’s not there.

dontlikeyoueith2• 2 years ago

This is just admitting you don't care about knowledge or truth.

HarHarVeryFunny15• 2 years ago

Sure, but the question posed isn't whether LLMs exhibit intelligence (obviously so, minimally in Chinese Room sense), or can they combine sources (sure, no way to stop them), but why do they hallucinate.

Notwithstanding the amazing things they can do, I don't think it helps understanding by viewing LLMs in too abstract of a way as intelligent agents. After all, in reality they are "just" language models, and hopefully in 2024 the nuance of what they needed to learn to be GOOD language models doesn't need to be explicitly stated every time we discuss them.

Looking at them as language models, it's easy to explain why they hallucinate, are poor reasoners, etc, and IMO does nothing to distract from understanding why they also exhibit intelligence when operating "in distribution".

not2b• 2 years ago

"Why would a language model do anything other than "hallucinate" (i.e. generate words without any care about truthiness) ?"

That's exactly the question the paper attempts to answer: why do LLMs ever get it right? The answer is that on topics where there's a lot of data and a general consensus on what the right answer is, the statistical model will find that answer, and otherwise you get junk. That's why they work so well for people trying to write Python or Javascript, for example.

But I already knew this, you might say. Sure, but the authors produced evidence to back it up.

jrm45• 2 years ago

There's really no "right or wrong" per se -- the question that's really being asked is "to what extent does it resonate with a person?"

danielbln3• 2 years ago

As context sizes grow, it's easier to add lots of information outside of training data via in-context learning, which should offset that issue quite a bit.

wslh• 2 years ago

I think we're all having different experiences and conversations with LLMs, and sometimes framing them purely as "statistical word generators" or expecting them to function like AI in the traditional sense of automated problem-solving might not capture the whole picture.

Like many in this community, I use LLMs daily. My main use case now isn’t software development but rather getting assistance in connecting concepts that I don’t know precisely, guided by my intent. In some ways, it's more akin to "out-of-the-box thinking" but with a tool that helps me explore ideas I might not reach on my own or suffer within the economy of search [1][2]. I might know something about X, Y, and Z, but there's a concept W that ties them all together. Without using W, people in that field might not grasp the connections between X, Y, and Z. Apologies for the abstraction here. I suppose it's my epistemological bias showing!

Even if LLMs don't precisely connect the dots, they help my brain connect them faster.

[1] https://en.wikipedia.org/wiki/Search_theory

[2] https://www.di.ens.fr/~lelarge/soc/varian2.pdf

snickell• 2 years ago

Yes, they are auto-completers, but they are auto-completers that are layered AND operate in higher dimensional spaces. This throws all intuitions off, and I think makes it misleading to think of them as "just" auto-completers. That's part of the story, but not the whole of it.

I suspect we are much closer to auto-completers than most of us like to think, but we're also trained+incentivized by culture, education, parenting, socializing, to produce "useful" results.

Maybe part of the problem is in the data set: how much modeling of "how to admit ignorance or uncertainty" are in LLMs training data sets? If you read the internet, all you see is confident replies to other confident replies. Ignorance or non-confidence tends to elicit either a bluff or non-response. If you read technical literature, you see much of the same.

Maybe LLMs are trained on a dataset, and thereby inherit a culture that's accidentally biased toward ignorant confidence. In human conversation, if somebody asks a question and I don't know the answer, I say I don't know. On the internet, I just skip it and leave it for somebody else who thinks they know.

All this is to say: maybe a statistical autocompleter can admit ignorance instead of firing "neural noise" based on barely-there loose associations. Maybe it just needs a stronger pathway toward talking about not knowing when there's not a strong association.

65102• 2 years ago

Wasn't it that it by design prefers things expressed with certainty?

HarHarVeryFunny15• 2 years ago

One problem is that the LLM's own knowledge, or lack of it, doesn't follow from any individual training sample(s). Even if there were a bunch of "I don't know X" samples in the training set, that ought to be trumped by one authoritative "X is ..." one.

The next problem is that the LLM doesn't know how reliable it's various training sources are (unlike a human who might trust personal experience > textbook > twitter comment), or even which samples come from which source so that it could learn that.

chaosist2• 2 years ago

I have thought a lot about this and I suspect "I don't know" would be devastating to the model.

The magic is in the fact that the model can't say "I don't know". The model would have to have arbitrary thresholds and a type of domain/context classification in order to set this arbitrary threshold for the conversation in order to say "I don't know". It would create all these unsolvable boundary conditions. AGI in this context then would be minimizing the need for "I don't know" until it no longer applies. Is that possible? I don't know :)

I would defer to Chomsky also on the subject that we are basically nothing like these models when it comes to language and we are not auto-completers.

famouswaffles2• 2 years ago

Pretty much all research (and there's a fair few with different methodologies) on this converge on the same conclusion:

LLMs internally know a lot more about the uncertainty and factualness of their predictions than they say. "LLMs are always hallucinating" is a popular stance but wrong all the same. Maybe rather than asking Why models hallucinate, the better question is to ask "Why not?". During pre-training, there's close to zero incentive to push any uncertainty to the forefront (words).

GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r

Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221

The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824

The Internal State of an LLM Knows When It's Lying - https://arxiv.org/abs/2304.13734

LLMs Know More Than What They Say - https://arjunbansal.substack.com/p/llms-know-more-than-what-...

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975

Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334

HarHarVeryFunny15• 2 years ago

Yes, because just like chess ELO that we discussed the other day, they need to learn this in order to do well on their training objective - impersonating (continuing) their training sources. If they are continuing a lie then they need to have recognized the input as having this "context", and take that into account during prediction.

andrewla9• 2 years ago

Why would a language model do anything other than "hallucinate"

Indeed, why?!?!?! Why do they so often get the correct answer to very direct questions? Saying "it's in the training data" -- I dare you to find anything in the training data that talks about how many "q's" there are in the word "mortuary", and yet it "hallucinates" up answers to this.

without any care

What does it mean to care? What question could you ask of an LLM that would allow you to assess how much it "cares" about something?

but it's not like they have a choice of not answering or saying "I don't know" in other cases

Is it your contention that the phrase "I don't know" has never occurred in the training data for an LLM?

There seems to be a dichotomy of reactions to LLMs. There are technical people saying "it's just an autocomplete engine reciting things from its training set" and there are non-technical people saying "it does more than just next token completion, it is trained to use language".

The first group is technically correct but ignores the fact that it can, emergently, do things far outside of an explainable capability, the second group is technically incorrect but correctly perceives that it can use language in novel ways.

Neither group captures the fact that we built extremely complex linear algebra machines that for reasons we do not understand, despite being trained on an incredibly simple task (next-token-prediction) are capable of actually using language in a way that ten years ago we assumed only humans could do.

HarHarVeryFunny15• 2 years ago

Is it your contention that the phrase "I don't know" has never occurred in the training data for an LLM?

No, but when it does occur in the training data it's a reflection of that particular source/speaker not knowing, which isn't the same as the LLM not knowing because it was also trained on millions/billions of additional sources.

For an LLM to learn to say "I don't know" appropriately, it would need to know when it itself doesn't know (and have that change if you told it), and of course it doesn't have that capability.

Nition• 2 years ago

Just a note: Truthiness means a feeling of truthfulness, even if not actually true. I'd argue that LLMs do care about that, but I suspect you meant 'truthfulness'.

madiator• 2 years ago

There are several types of hallucinations, and the most important one for RAG is grounded factuality.

We built a model to detect this, and it does pretty well! Given a context and a claim, it tells how well the context supports the claim. You can check out a demo at https://playground.bespokelabs.ai

abernard12• 2 years ago

The problem with this line of argumentation is it implies that autoregressive LLMs only hallucinate based upon linguistic fidelity and the quality of the training set.

This is not accurate. LLMs will always "hallucinate" because the size of the model they can encode is orders of magnitude smaller than the factual information they can contain from the training set. Even granting that semantic compression could reduce the model to smaller than the theoretical compression limit, Shannon entropy still applies. You cannot fit the informational content required for them to be accurate into these model sizes.

This will obviously apply to chain of thought or N-shot reasoning as well. Intermediate steps chained together still can only contain this fixed amount of entropy. It slightly amazes me that the community most likely to talk about computational complexity will call these general reasoners when we know that reasoning has computational complexity and LLMs' cost is purely linear based upon tokens emitted.

Those claiming LLMs will overcome hallucinations have to argue that P or NP time complexity of intermediate reasoning steps will be well-covered by a fixed size training set. That's a bet I wouldn't take, because it's obviously impossible, both on information storage and computational complexity grounds.

Circlecrypto2• 2 years ago

A visual the displays probabilities and how things can quickly go "off-path" would be very helpful for most people who use these without understanding how they work.

jp572• 2 years ago

A bit off topic, but am I the only one unhappy about the choice of the word "hallucinate" to describe the phenomenon of LLMs saying things that are false?

The verb has always meant experiencing false sensations or perceptions, not saying false things. If a person were to speak to you without regard for whether what they said was true, you'd say they were bulshitting you, not hallucinating.

strangescript2• 2 years ago

What if that person firmly believes what they say is true?

AnimalMuppet• 2 years ago

The thing saying it is not a person, and has no beliefs.

The problem is basically epistemology. GPTs don't have any. Arguably they don't know anything. (Arguably they do to an extent, because the knowledge is encoded in the words in the training data.) But even if they know things, they don't know that they know, and so they cannot tell between "knowing" and "not knowing".

function_seven2• 2 years ago

Bullshitting implies knowing that you're lying. Some sort of malice or intention to deceive.

Hallucinating means the LLM really "thinks" that you can use PVA glue in a pizza recipe. It's not trying to screw you over. It's just that the token generator has found a weird path through the training set. (I'm sure I didn't word that last sentence correctly)

I think "hallucinate" is a spot-on description of what's happening under the hood.

jp572• 2 years ago

I don't think people use "bullshitting" interchangably with "lying". I'm partial to this characterization:

So bullshitting isn’t just nonsense. It’s constructed in order to appear meaningful, though on closer examination, it isn’t. And bullshit isn’t the same as lying. A liar knows the truth but makes statements deliberately intended to sell people on falsehoods. bullshitters, in contrast, aren’t concerned about what’s true or not, so much as they’re trying to appear as if they know what they’re talking about. ... [W]hen people speak from a position of disproportionate confidence about their knowledge relative to what little they actually know, bullshit is often the result. [1]

Doesn't this description like what LLMs do all the time?

[1] https://www.psychologytoday.com/us/blog/psych-unseen/202007/...

DonaldFisk• 2 years ago

Bullshitting implies knowing that you're lying.

Harry Frankfurt had a more useful definition of bullshitting. The essence of it is that while liars care about the truth and intend to deceive, bullshitters don't know or care about the truth - they want to impress, or avoid looking stupid, or something similar.

Hallucination is clearly the wrong word here, as is lying. Bullshitting isn't much better. Confabulation, however, is very close to what LLMs are doing when they make up stuff. https://en.wikipedia.org/wiki/Confabulation

iknownthing• 2 years ago

How about just "misprediction"?

pacificmint• 2 years ago

If you ask the model what the color of grass is, and it answers blue, then that would indeed be false (or maybe a lie). I think most people wouldn’t call that a hallucination.

But if you ask it for a court case, and it makes up a whole false case file with fake names and fake facts and everything, then calling that ‘false’ seems to be an understatement. Hallucination seems a good label for that kind of thing, imo.

PretzelPirate• 2 years ago

but am I the only one unhappy about the choice of the word "hallucinate" to describe the phenomenon of LLMs saying things that are false?

This has been discussed quite a bit and some people have decided that 'confabulation' is a better term.

lasermike0262• 2 years ago

Stop using the term "Hallucinations". GPT models are not aware, do not have understanding, and are not conscious. We should refrain anthropomorphizing GPT models. GPT models sometime produce bad output. Start using the term "Bad Output".

codetrotter• 2 years ago

That’s too vague. Use “confabulations” instead. Anyway the battle is lost, “hallucinations” it is and forever will be.

lasermike0262• 2 years ago

No.

syoc• 2 years ago

I once again feel that a comparison to humans is fitting. We are also "trained" on a huge amount of input over a large amount of time. We will also try to guess the most natural continuation of our current prompt (setting). When asked about things it I can at times hallucinate things I was certain to be true.

It seems very natural to me that large advances in reasoning and logic in AI should come at the expense of output predictability and absolute precision.

abernard12• 2 years ago

The comparison is flawed though in that humans and LLMs make mistakes for different reasons.

Humans forget things. Humans make errors. Humans' train of thought isn't impacted by an errant next token in the statement they're making. We have thoughts which exist as complete prior to us "emitting" them. Just as a multi-lingual speaker does not have thoughts exclusive to the language they're speaking in (even if that language allows them tools to think a certain way).

This is obvious if you consider different types of symbolic languages, such as sign language. Children can learn sign language prior to them being verbal. The ideas they have as a prior are not effected by the next sign they make: children actually know things independent of the symbolic representation they choose to use.

Der_Einzige• 2 years ago

Hallucination is creativity when you don't want it.

Creativity is hallucination when you do want it.

A lot of the "reduction" of hallucination is management of logprobs, of which fancy samplers like min_p do more to improve LLM performance than most, despite no one in the VC world knowing or caring about this technique.

If you don't believe me, you should check out how radically different an LLMs outputs are with even slightly different sampling settings: https://artefact2.github.io/llm-sampling/index.xhtml

wisnesky• 2 years ago

There is a partial solution to this problem: use formal methods such as symbolic logic and theorem proving to check the LLM output for correctness. We are launching a semantic validator for LLM-generated SQL code at sql.ai even now. (It checks for things like missing joins.) And others are using logic and math to create LLMs that don't hallucinate or have safety nets for hallucination, such as Symbolica. It is only when the LLM output doesn't have a correct answer that the technical issues become complicated.

swatcoder4• 2 years ago

Proofs can ensure soundness for a collection of logical statements in an output, but people are being sold epistemic "truth".

This article is trying to elaborate what that means for LLM's, which only know truth through frequency ("crowdsourced truth") at best. For esoteric, sparse, ambiguous, uncertain, controversial, etc subjects, that's not an adequate truth standard to start from and logical proofs do nothing to improve on it.

brcmthrowaway• 2 years ago

Q* will have this

antirez• 2 years ago

Terrible article. The author does not understand how LLMs work basically, since an LMM cares a lot about the semantic meaning of a token, this thing about the next word probability is so dumb that we can use it as "fake AI expert" detector.

JohnMakin5• 2 years ago

Something tells me that the author [0] is probably well aware of how these work under the hood, and the math behind it - When writing scientific articles with a laymen audience in mind, you'll often have to use laymen-specific terms. But feel free to enlighten us further!

[0] https://en.wikipedia.org/wiki/Jim_Waldo

throw3108222• 2 years ago

Whatever his credentials, what he says is plain wrong. GPTs don't follow "the grass is" with "green" because it's the most probable continuation- this idea is incredibly naive and breaks down with sentences longer than a few words. And GPTs don't crowdsource the answers to questions, their answers are not necessarily the most common, and neither "the consensus view is determined by the probabilities of the co-occurrence of the terms"- there is no such algorithm implemented anywhere.

What LLMs crowdsource is a world model, and they need an incredible amount of language to squeeze one out from it, second hand. We do train them for the ability to predict the next word, which is a task that can only be performed satisfactorily by working at the level of concepts and their relationships, not at the level of words.

josefritzishere• 2 years ago

I liked the take that LLMs are bullshitting, not hallucinating. https://www.scientificamerican.com/article/chatgpt-isnt-hall...

JohnMakin5• 2 years ago

Besides harping on the fact that "hallucination" is unnecessarily anthropomorphizing these tools, I'll relent because clearly that argument has been lost. This is more interesting to me:

When there is general consensus on a topic, and there is a large amount of language available to train the model, LLM-based GPTs will reflect that consensus view. But in cases where there are not enough examples of language about a subject, or the subject is controversial, or there is no clear consensus on the topic, relying on these systems will lead to questionable results.

This makes a lot of intuitive sense, just from trying to use these tools to accelerate Terraform module development in a production setting - Terraform, particularly HCL, should be something LLM's are extremely good at. It's very structured, the documentation is broadly available, and tons of examples and oodles of open source stuff exists out there.

It is pretty good at parsing/generating HCL/terraform for most common providers. However, about 10-20% of the time, it will completely make up fields or values that don't exist or work but look plausible enough to be right - e.g., mixing up a resource ARN with an resource id, or things like "ssl_config" may become something like "ssl_configuration" and leave you puzzling for 20 minutes what's wrong with it.

Another thing it will constantly do is mix up versions - terraform providers change often, deprecate things all the time, and there are a lot of differences in how to do things even between different terraform versions. So, by my observation in this specific scenario, the author's intuition rings completely correct. I'll let people better at math than me pick it apart though.

final edit: Although I love the idea of this experiment, it seems like it's definitely missing a "control" response - a response that isn't supposed to change over time.

kreims• 2 years ago

Please keep harping. The marketing myths that gets circulated about these models are creating very serious misunderstandings and misallocation of resources. I am hopeful that more cautious and careful dialogue like this will curb the notions of sentience or human intelligence that exciting headlines seemed to have put in the public discussion of these tools.

JohnMakin5• 2 years ago

It just draws too many annoying comments and downvotes, and has been discussed ad nauseam on this forum and others - but I broadly agree. There are "features" with these applications where if I'm rude, or frustrated with the responses, the model will say things like "I'm not continuing this conversation."

How utterly absurd, it has no emotions, and there's no way that response was the result of a training set. It's just dumb marketing, all of it. And the real shame is (and the thing that actually pisses me off about the marketing/hype) that the useful things we actually have uncovered from ML or "AI" the last 10 years will be lost again in the inevitable AI winter we're facing following from whenever this market bubble collapses.

shwaj• 2 years ago

What’s the alternative? You can’t just say “don’t say that”. There needs to be something you can say instead, 5 syllables at the most, which evokes the same feeling of confident wrongness, without falling into anthropomorphism. It’s a tall order.

eschneider• 2 years ago

Don't anthropomorphize LLMs. They hate that.

And it's not even a question of LLMs getting answers "wrong". It's just generating associated text. It has no concept of right or wrong answers.

baxtr2• 2 years ago

I think it’s totally fine to am LLMs. In the end they have been trained on human input.

ysofunny2• 2 years ago

just to be clear, I see it like this (for now):

if a GPT does it and turns out to be false, then it's an hallucination and it's bad (goto more training)

if a human does it, then truth becomes "self-expression" (art) so we call it creativity and it's good

TremendousJudge• 2 years ago

No, if a human does it by accident, as is clearly the case here, we call it "hallucination", "misremembering", "mandela effect" or "dementia"

bayindirh• 2 years ago

if a human does it, then truth becomes "self-expression" (art) so we call it creativity and it's good.

Depends. Once I misremembered the usage of the command "ln", and I wiped ~10 machines inadvertently.

Nobody called it self-expression / art, and none of the results of my little "experiment" were good.

Do it a couple of times, and you'll be updating your CV.

FrustratedMonky11• 2 years ago

Maybe instead of hallucinate? Use 'BS'?

To anthropomorphize even more. Since humans will also just create "BS" as an answer if they don't know the answer, or will combine half bits of knowledge into something to sound like they know what they are talking about.

basch• 2 years ago

There is a secondary issue of LLM's taking questions literally, and not really being able to (at the moment) deny the premise of a question. For example, if you google benefits of circumcision, the LLM will quite literally print all the benefits. But it also wont contextualize them, it wont frame them, it wont provide counter arguments, it just responds literally to the question.

nerdjon• 2 years ago

I get the concern over what using the word hallucination implies, I also think it is a fairly fitting word.

We need something easy to explain when these systems are straight up wrong. Something that a normal non technical user will understand. Sure saying "wrong" could be easy enough, I think "Hallucination" also has a simplicity too it.

Part of the problem is that these models will appear to confidently be wrong. Hallucinate to me kinda goes along with this, it isn't just wrong things are being made up.

But regardless of that, people are used to calling it hallucinating. We are also up against an effort to downplay any concern over this fundamental problem with the technology and already trying to push it as a general AI (And we have to recognize there is a ton of money on pushing this exact narrative), that I would be worried about confusing the topic by pushing for an alternative term giving leeway to further downplay the problem.

freilanzer• 2 years ago

I think it's a perfectly good word for what is happening.

tim3332• 2 years ago

In favour of "hallucination" it's not that much of an anthropomorphization because hallucination in a human context is something quite different - seeing ghosts and the like. If you use it in the context of an LLM everyone knows what you mean. The human terms for making random stuff up would be bullshiting, imagining etc.

xkcd-sucks• 2 years ago

"Hallucinate" is an interesting way to position it: It could just as easily be positioned as "too ignorant to know it's wrong" or "lying maliciously".

Indeed, the subjects on which it "hallucinates" are often mundane topics which in humans we would attribute to ignorance, i.e. code that doesn't work, facts that are wrong, etc. Not like "laser beams from jesus are controlling the president's thoughts" as a very contrived example of something which in humans we'd attribute to hallucination.

idk, I'd rather speculatively invest in "a troubled genius" than "a stupid liar" so there's that

shombaboor2• 2 years ago

they've got some great marketing to get away without using: bug, defect, error, malfunction

shombaboor2• 2 years ago

or limitation

euroderf• 2 years ago

It might be more like reification ? The system finds a satisfying solution and sort of wills it into existence - by verbalizing it, makes it so ?

zh3• 2 years ago

It could just as easily be positioned as "too ignorant to know it's wrong" ...

GPT-5 is widely predicted to have a Dunning-Kruger level of expertise.

phainopepla2• 2 years ago

lying maliciously

Malice implies intent, which is even more misleading that hallucination, imo

taneq• 2 years ago

It's just 'prediction error' in a feedback loop, imo.

I'm sure like any other biological human with mitochondria and stuff, you've occasionally said (or started to say) something and then you (ie. the actively cross-checking self-analyzing enigma that is 'you') thinks 'hang on, no that doesn't make sense' and you self-correct. LLMs are 100% feedforward, there's just one big autoregression going on. No strange loop shenanigans.

Honestly I'm really interested to see where LLM-based diffusion models end up. (To be fair, probably mostly because I don't understand them yet so they could still be spooky. :D )

aaroninsf• 2 years ago

ITT an awful lot of smart people who still don't have a good mental model of what LLM are actually doing.

The "stochastic continuation" ie parrot model is pernicious. It's doing active harm now to advancing understanding.

It's pernicious, and I mean that precisely, because it is both technically accurate yet deeply unhelpful indeed actively, intentionally AFAICT, misleading.

Humans could be described in the same way, just as accurately, and just as unhelpfully.

What's missing? What's missing is one of the gross features of LLM: their interior layers.

If you don't understand what is necessarily transpiring in those layers, you don't understand what they're doing; and treating them as black box that does something you imagine to be glorified Markov chain computation, leads you deep into the wilderness of cognitive error. You're reasoning from a misleading model.

If you want a better mental model for what they are doing, you need to take seriously that the "tokens" LLM consume and emit are being converted into something else, processed, and then the output of that process, re-serialized and rendered into tokens. In lay language it's less misleadly and more helpful to put this directly: they extract semantic meaning as propositions or descriptions about a world they have an internalized world model of; compute a solution (answer) to questions or requests posed with respect to that world model; and then convert their solution into a serialized token stream.

The complaint that they do not "understand" is correct, but not in the way people usually think. It's not that they do not have understanding in some real sense; it's that the world model they construct, inhabit, and reason about, is a flatland: it's static and one dimensional.

My rant here leads to a very testable proposition: that deep multi-modal models, particularly those for whom time-base media are native, will necessarily have a much richer (more multidimensional) derived world-model, one that understands (my word) that a shoe is not just an opaque token, but a thing of such and such scale and composition and utility and application, representing a function as much as a design.

When we teach models about space, time, the things that inhabit that, and what it means to have agency among them—well, what we will have, using technology we already have, is something which I will contentedly assert is undeniably a mind.

What's more provocative yet is that systems of this complexity, which necessarily construct a world model, are only able to do what they do because they have a self-model within it.

And having a self-model, within a world model, and agency?

That is self-hood. That is personhood. That is the substrate as best we understand for self-awareness.

Scoff if you like, bookmark if you will—this will be commonly accepted within five years.

sdwrj• 2 years ago

You mean the magic wizard isn't real and GPT lied to me!?!?

linsomniac• 2 years ago

One of the biggest places I've run into hallucination in the past has been when writing python code for APIs, and in particular the Jira API. I've just written a couple of CLI Jira tools using Zed's Claude Sonnet 3.5 integration, one from whole cloth and the other as a modification of the first, and it was nearly flawless. IIRC, the only issue I ran into was that it was trying to assign the ticket to myself by looking me up using "os.environ['USER']" rather than "jira.myself()" and it fixed it when I pointed this out to it.

Not sure if this is because of better training, Claude Sonnet 3.5 being better about hallucinations (previously I've used ChatGPT 4 almost exclusively), or what.

danielbln3• 2 years ago

Context helps so, so much. Adding terminal output, IDE diagnostics, code, remote documentation into the context really improves the output, and editors like zed make it very convenient to do.

andrewla9• 2 years ago

The author says:

Once understood in this way, the question to ask is not, "Why do GPTs hallucinate?", but rather, "Why do they get anything right at all?"

This is the right question. The answers here are entirely unsatisfactory, both from this paper and from the general field of research. We have almost no idea how these things work -- we're at the stage where we learn more from the "golden-gate-bridge" crippled network than we do from understanding how they are trained and how they are architected.

LLMs are clearly not conscious or sentient, but they show emergent behavior that we are not capable of explaining yet. Ten years ago the statement "what distinguishes Man from Animal is that Man has Language" would seem totally reasonable, but now we have a second example of a system that uses language, and it is dumbfounding.

The hype around LLMs is just hype -- LLMs are a solution in search of a problem -- but the emergent features of these models is a tantalizing glimpse of what it means to "think" in an evolved system.

tgbugs2• 2 years ago

I think the answer is actually quite clear and rather boring. In order to get something "right" there has to be some external standard of knowledge and correctness. That definition of correctness can only be provided by the observer (user). Alignment between the user's correctness criteria and generated text happens entirely by accident. This can be demonstrated by observing a correlation between coverage of a domain in the training data and the rate at which incorrect results are produced (as discussed in other comments). That is, they get things "right" because there was sufficient training data that contained information that matched the user's definition for correctness. In fact, exceptionally boring.

andrewla9• 2 years ago

This is a very post hoc explanation. What does "coverage in the training data" mean?

Take a simple task of something like "How many a's are there in the word bookkeeper" -- what is your theory for why it can answer this question correctly or even give something approaching a coherent answer? It never even sees the letters that are in the token "bookkeeper", and this is definitely not something that appears explicitly in the training data.

I challenge you to give a "clear and boring" explanation for this -- this is incredibly subtle behavior that emerges from a complex architecture and complex training process, and is in its own right as fascinating and mysterious as the ability of humans to do this task and the inability of cats to do it.

lottin2• 2 years ago

now we have a second example of a system that uses language, and it is dumbfounding

An LLM 'uses' language, in the same sense that a calculator 'uses' arithmetic. It's a figure of speech.

andrewla9• 2 years ago

That's not the sense of the metaphor that I'm applying when I say "uses language". That's closer to saying that "Alexa uses language", where "uses" here is analogous to what a calculator does.

To avoid using anthropomorphic terms, an LLM can take natural language from a human, integrate information from those expressions together with information held in its (opaque) store, and return natural language that a human can understand that reflects that information.

I am not aware of any other systems besides humans that can accomplish that task. Some animals can be trained to do some parts of this, but really until now humans are the only ones that could do the full loop.

gengstrand• 2 years ago

This piece reminds me of something I did earlier this year https://www.infoq.com/articles/llm-productivity-experiment/ where I conducted an experiment across several LLMs but it was a one-shot prompt about generating unit tests. Though there were significant differences in the results, the conclusions seem to me to be similar.

When an LLM is prompted, it generates a response by predicting the most probable continuation or completion of the input. It considers the context provided by the input and generates a response that is coherent, relevant, and contextually appropriate but not necessarily correct.

I like the crowdsourcing metaphor. Back when crowdsourcing was the next big think in application development, there was always a curatorial process that filters out low quality content then distills the "wisdom of the crowds" into more actionable results. For AI, that would be called supervised learning which definitely increases the costs.

I think that unbiased and authentic experimentation and measurement of hallucinations in generative AI is important and hope that this effort continues. I encourage the folks here to participate in that in order to monitor the real value that LLMs provide and also as an ongoing reminder that human review and supervision will always be a necessity.

derefr• 2 years ago

For coding problems specifically, you could get quite far by giving the model a the tool-use of a sandboxed compiler/interpreter (perhaps even with your project files already loaded into the sandbox); and then training the model to test its own proposed solutions in the sandbox and revise them until they actually produce the expected outputs.

fsndz• 2 years ago

Jean Piaget said it better: "Intelligence is not what we know, but what we do when we don't know." And what do LLMs do when they don't know, they spit out bullshit. That is why LLMs won't yield to AGI (https://www.lycee.ai/blog/why-no-agi-openai). For anything that is out of their training distribution, LLMs fail miserably. If you want to build a robust Q&A system and reduce hallucinations, you better do a lot of grounding, or automatic prompt optimisation with few shot examples with things like DSPy (https://medium.com/gitconnected/building-an-optimized-questi...)

simonw• 2 years ago

For this experiment we used four models: Llama, accessed through the open-source Llama-lib; ChatGPT-3.5 and ChatGPT-4, accessed through the OpenAI subscription service; and Google Gemini, accessed through the free Google service.

Papers like this really need to include the actual version numbers. GPT-4 or GPT-4o, and which dated version? Llama 2 or 3 or 3.1, quantized or not? Google Gemini 1.0 or 1.5?

Also, what's Llama-lib? Do they mean llama.cpp?

Even more importantly: was this the Gemini model or was it Gemini+Google Search? The "through the free Google service" part could mean either.

UPDATE: They do clarify that a little bit here:

Each of these prompts was posed to each model every week from March 27, 2024, to April 29, 2024. The prompts were presented sequentially in a single chat session and were also tested in an isolated chat session to view context dependency.

Llama 3 came out 18th of April, so I guess they used Llama 2?

(Testing the prompts sequentially in a single chat feels like an inadvisable choice to me - they later note that things like "answer in three words" sometimes leaked through to the following prompt, which isn't surprising given how LLM chat sessions work.)

FrustratedMonky11• 2 years ago

Is prompt engineering really 'psychology'. Convincing the AI to do what you want. Just like you might 'prompt' a human to do something. Like in the short story Lena, 2021-01-04 by qntm

https://qntm.org/mmacevedo

In short story, the weights of the LLM are a brain scan.

But same situation. People could use multiple copies of the AI. But each time, they would have to 'talk it into' doing what they wanted

tim3332• 2 years ago

It seems to me that human brains do something like LLM hallucination in the first second or two - come up with random guess, often wrong. But then something fact checks it. As in does it make sense, is there any evidence. I gather the new q* / strawberry thing does something like that. Sometimes personally in comments I think something but google it see if I made it up and sometimes I have. I think a secondary fact check phase may be necessary for all neural network type setups.

GPTs and Hallucination

Related Stories