Hallucination is inevitable: An innate limitation of large language models

https://arxiv.org/abs/2401.11817

by louthy • 2 years ago

308 474 2 years ago

arxiv.org

chrisjj2• 2 years ago

hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function.

That's simply inaccuracy or fabrication.

Labelling it hallucination simply panders to the idea these programs are intelligent.

somewhereoutth4• 2 years ago

Yes, imagine a pocket calculator that gave a completely wrong answer 10%? of the time, and calling it 'capricious' instead of simply broken.

BlueTemplar2• 2 years ago

A real shame that Douglas Adams didn't live to see all of this...

chrisjj2• 2 years ago

Indeed. The best solution would be to market it as intelligent ;)

tgv4• 2 years ago

That statement is also a bit easy on the "computable ground truth." There is no such thing for the tasks we use an LLM for (unless you make up some definition that mangles the definition of each word).

paulsutter• 2 years ago

The models are just generating probable text. What’s amazing of how often the text is correct. It’s no surprise at all when it’s wrong

Their bold confidence to be flat out wrong may be their most human trait

baq3• 2 years ago

This again.

They’re trained to generate probable text. The mechanisms created in the parameter blob during training to do that are basically a mystery and have to be pulled out of the model with digital brain surgery. E.g. LLMs are reasonable at chess and turns out somewhere in the blob there’s a chessboard representation, and you can make the model believe the board is in a different state by tweaking those parameters.

So yeah they generate probable text, sure. Where they get the probabilities is a very good research problem.

timr2• 2 years ago

E.g. LLMs are reasonable at chess and turns out somewhere in the blob there’s a chessboard representation, and you can make the model believe the board is in a different state by tweaking those parameters.

Broadly agreed, but there's no "representation"...the model has no memory, let alone a "concept" of a chessboard. It's just trained on a bunch of textual replays of chess games, and this works well enough for a game with finite probability space.

Likewise, I have asked generic LLMs to generate "novel" drugs that solve particular problems, because their training sets have included lots of examples of drug molecules in string form (aka SMILES strings). This works far less well (because chemical space is much larger than a chess game), and the "novel" drugs usually end up looking like some mishmash of existing chemicals for the same problem. This can be a useful tool, but nobody is confusing it for a human brain doing problem solving.

People are easily misled by the purported intelligence of these things because they assume that common examples of human intelligence are less probable than they really are. Our languages and games and stories are pretty regular, all things considered. Expand the probability space to something truly vast (even images work for this), and you can easily see the stochastic parrot emerge.

Mistletoe• 2 years ago

I hope to remember your last line for the rest of my life when I think about AI.

irq• 2 years ago

paulsutter said: > Note that this is the single most human attribute of LLMs

It might be if LLM hallucinations looked like or occurred at the same frequency as human hallucinations do, but they don’t.

regularfry• 2 years ago

You could make an argument that what we currently see are effectively internal monologues. It is extremely hard to evaluate how much subconscious or conscious filtering happens between a human's internal state and the eventual outbound communications, but I wouldn't be at all surprised if the upstream hallucination rate in humans was much higher than you'd think.

By analogy to Kahneman and Tversky's System 1 and System 2, the whole field of Prospect Theory is about how often System 1 is wrong. This feels connected.

BlueTemplar2• 2 years ago

Don't they ?

Yesterday I read "Building a deep learning rig" as "Building a deep learning pig" at first for some reason I can't explain...

alpaca128• 2 years ago

Why do you expect hallucination frequency to be the same when the LLM doesn't even remotely compare to a human brain yet? And what do they have to "look like"? This always reminds me of that time Bing's chat AI doubled down on a wrong fact about the Avatar 2 movie, which people used as evidence that the technology is dumb when it really is exactly the behaviour you can observe in many people every day. And there's a reason adults do it less frequently than children.

Compare the hallucination behaviour of a 7B model with a 70B model and then GPT4 and you'll quickly see the frequency of hallucinations right now doesn't mean much.

bamboozled3• 2 years ago

Geoffrey Hinton has entered the chat…

ithkuil2• 2 years ago

The confidence has been selected for.

You can train a model to always carefully claim that what they say may or may not be true, but that may not necessarily make them easier to work with.

IIUC what we're missing right now is the ability for the model to get a good estimate of how confident they _should_ be.

This can be done "externally" or "internally". Doing it externally would mean: make the model not directly stream its train of thought directly to the end user but instead use it first to produce a query info some system that can help it weigh its confidence factor. Then a se and run on that output and its previous output can reformat the answer to be more balanced.

Doing it internally would require this process to be part of the incremental output token production. I don't know about the field to know whether that's even doable and whether we have some progress on that front

sschueller• 2 years ago

You have to very carefully ask your question for it to not make things up. For example don't ask "how do I do this in in x?". Ask "can I do this with x?"

These "AI" s are like "yes men". They will say anything to please you even if it's untrue or impossible.

I have met people like that and they are very difficult to work with. You can't trust that they will deliver the project they promised and you always have to double check everything. You also can't trust them that what they promised is even possible.

ithkuil2• 2 years ago

The analogy is apt indeed. It's all about training and selection. As long as the incentives are that you either behave in that way "or else" it's unsurprising that we end up with a system that uses its intelligence to meet the goals you've set up.

Fortunately that doesn't tell much about the limitations of the underlying intelligence but mostly about the limitations of the incentive structure you put around it

impulsivepuppet• 2 years ago

Before ChatGPT, human language translation had a similar problem but people weren't as vocal about it.

What I find frustrating that it's increasingly challenging to have DeepL translate thou -> du, as this was my go-to "hack" to overcome the incompatibility of the English language due to its missing features.

To somewhat remedy the "yes man" problem, one needs to become a pedantic mathematician about posing your questions and I don't believe that LLM technology alone is capable of overcoming it entirely. As silly as it sounds, I must concede to the existence of "prompt engineering" as I can forsee the development of abstractions aimed to decompose questions for you.

tudorw• 2 years ago

I've had some success with 'Ask follow up questions where additional clarity is required'. My best experiences start with a much more freeform conversation about what we are going to attempt to do, doing a Q&A first to make sure that both I and the AI are thinking about the same domain and sharing associated nomenclature seems to help.

empath-nirvana3• 2 years ago

I have often had it tell me that things I want to do with various technologies aren't possible or that there are better ways to do it.

bonzaidrinkingb6• 2 years ago

Current AIs are RLHFd to avoid being a "yes man"/sycophant.

The point about employing better prompting is well taken. Don't ask "Who was the first female president?", ask "Was there ever a female president?". Much like on StackOverflow you want to ask the right question and not assume things (since you don't know enough to make assumptions).

Imagine if every time on early Google you found a spam result and then blame the search engine for that (and not your choice of keywords, or ignoring that you always want to return something, even if remotely related). Like a user banging a slab of concrete with a chisel and complaining that this does not produce a beautiful statue.

golol4• 2 years ago

This is just another diagonalization against some finite thing, like the no free lunch theorem. An "LLM" in their definition is essentially any finite thing which predicts the next token. The same result applies to your brain too, for example.

Like all such diagonalization results, it is not really relevant for real world considerations. The reason is that it does not matter if your model fails on none, finitely many or infinitely many inputs. In reality the space of possible inputs is equipped with a probability measure, and the size of the hallucinating inputs set w.r.t. that measure is relevant. Diagonalization arguments usually, make no claim to the size of that set, and it is most likely negligible in the real world.

FeepingCreature• 2 years ago

It seems a stretch to call it "inevitable". "Inevitable given the current architecture without modifications" at most.

Also, I'm missing a section on how (if) human brains manage to avoid hallucinations in this.

Also, it doesn't have to never hallucinate, it just has to hallucinate less than we do.

bamboozled3• 2 years ago

According to Buddhist philosophy, our whole identity is a hallucination :) I kind of concur.

jetanoia• 2 years ago

Username checks out :)

SubNoize• 2 years ago

Or catch itself that it's hallucinating? I feel like humans would do that a fair bit.

How often do we sit somewhere thinking about random scenarios that won't ever happen and are filled with wild thoughts and sometimes completely out of the world situations.. then we shake our heads and throw away the impossible from that thought train and only use what was based in reality

dkjaudyeqooe2• 2 years ago

Because we have parts of our brain that supervise other parts of our brain and evaluate its output.

For example: if you smoke pot and get paranoid, it's because pot dials back the work of the part of your brain that prunes thought paths that are not applicable. Normally, paranoid thoughts do not make sense, so they are discarded. That's also why you're more 'creative' when you smoke pot, less thought paths are pruned and more stuff that doesn't quite make sense gets through. Or thoughts that overly focus on some details get through, which are normally not required.

Our brains are inherently "higher level", current AI is hopelessly simplistic by comparison.

avidphantasm2• 2 years ago

Perhaps solving hallucinations at the LLM level alone is impossible, hence the inevitability. I reckon that lots of human “hallucination” is simply caught by higher-level control loops operating over the output of the generative mechanism. Basically, our conscious mind says, “nah, that doesn’t look right” enough that most of the time most of us don’t “hallucinate”.

avidphantasm2• 2 years ago

So this implies that instead of spending resources on training bigger and bigger LLMs, AI practitioners need to shift focus to developing “ontological” and “epistemological” control loops to run on top of the LLM. I suspect they already have rudimentary such control loops. In a sense, the “easier” part of AI may be a largely “solved” problem, leaving the development of “consciousness” to be solved, which is obviously the hard part.

resolutebat2• 2 years ago

Humans do hallucinate, there's lots of literature on how memories are distorted, we see and hear things we want to see and hear, etc.

The particular pathology of LLMs is that they're literally incapable of distinguishing facts from hallucinations even in the most mundane circumstances: if a human is asked to summarize the quarterly results of company X, unlike an LLM they're highly unlikely to recite a convincing but completely fabricated set of numbers.

exe342• 2 years ago

And yet if you ask a random person at a rally about their favourite cause of the day, they usually spew sound bites that are factually inaccurate, and give all impressions of being as earnest and confident as the LLM making up quarterly results.

t_mann4• 2 years ago

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations).

It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute some functions, but the hallucination problem isn't about LLMs needing to know everything. The problem that we care about is the 'I don't know'-answering problem, which may still be computable.

az09mugen2• 2 years ago

I think there is no easy way to make an LLM answer "I don't know". For that, it should learn among all the stuff ingested when people effectively don't know. But most people on internet write down irrelevant stuff even when they don't know instead of simply writing "I don't know".

That's a very good point.

timini• 2 years ago

I think its fairly simple, it needs a certain level of proof e.g references to authoritative sources, if not say "i don't know".

rini17• 2 years ago

Maybe it needs some memory retrieval step that can measure the confidence - whether there's anything related to the prompt. No idea how to train a LLM to do that.

dmd• 2 years ago

Consider the extremely common Amazon product question section, where you see Q: Will this product fit my Frobnitz 123? A: I don't know, I ended up buying something else. Q: Does it come with batteries? A: IDK I RETURN IT

golol4• 2 years ago

I can assure you it has no relevance for people working with LLMs, as the result includes your brain, for example.

svantana• 2 years ago

According to their definition, answering "I don't know" is also a hallucination. Even worse, the truth function is deliberately designed to trip up the models, it has no connection to any real-world truth. So for example, if the input is "what is 2 + 5?" and the LLM answers "7", - their truth function will say that's a hallucination, the correct answer is "banana".

cornholio• 2 years ago

Transformers have no capacity for self reflection, for reasoning about their reasoning process, they don't "know" that they don't know. My interpretation of the paper is that it claims this weakness if fundamental, you can train the network to act as if it knows its knowledge limits, but there will always be an impossible to cover gap for any real world implementation.

GaggiX• 2 years ago

Do you have a source? That's also what I thought but I wouldn't be surprised if the model learned to identify its own perplexity during the reinforcement learning phase.

ddalex5• 2 years ago

Actually it seems to me that they do... I asked via custom prompts the various GPTs to give me scores for accuracy, precision and confidence for its answer (in range 0-1), and then I instructed them to stop generating when they feel the scores will be under .9, which seems to pretty much stop the hallucination. I added this as a suffix to my queries.

mike_hearn3• 2 years ago

They do have that capacity. The generated token probabilities reflect some degree of certainty and additionally that information is available earlier in the network too, such that it can explain its own level of confidence.

mysterydip• 2 years ago

Would a possible solution be a hybrid approach? I remember back years ago seeing AI research around predicate logic, with hundreds of thousands of classification entries and "x can't be y" type stuff.

Maybe the potential output of an LLM could be run through that kind of logic and fed back to itself for sanity before being decided as final output?

moyix2• 2 years ago

Seems to be contradicted by this paper, no?

https://arxiv.org/abs/2207.05221

cpuguy832• 2 years ago

Are humans not the same way? There's a saying "I don't know what I don't know".

cultureswitch6• 2 years ago

you can train the network to act as if it knows its knowledge limits

Humans need more training to do the same.

And this does not answer the question of whether there's self-reflection going on. Practical LLMs available today are perfectly capable of telling you about their own reasoning process.

And much like a human, an LLM is incapable of fully explaining it's reasoning process from first principles to a human. Why is that? Probably because humans are too limited to grok large scale complex processes like the human brain or an LLM.

Finally, there is no difference between acting as if you know your training limits and knowing your training limits, if your acting is good enough. This goes for both humans and LLMs.

drdrek• 2 years ago

Not saying anything about LLM But in CS in general many issues "cannot be solved" or "Cannot be solved in reasonable time (NP)" but approximations upper bound by some value are solvable in reasonable time (P).

And in the real world if the truck route of amazon is 20% off the mathematically optimal solution the traveling salesman is "Solved" in a good enough way.

startupsfail• 2 years ago

The claim of the paper is that computation is irreducible (assuming P!=NP), LLMs have limited computational capacity and will hallucinate on the irreducible problems.

I don’t know, the claim seems dubious to me. We usually are able to have algorithms that return a failure status, when the problem proved to be too large. Avoiding the “hallucination”. Don’t see why LLMs can’t have that embedded.

nottorp3• 2 years ago

we can get LLMs to answer 'I don't know' more often

Have any nets been trained specifically to be able to go to an 'i don't know' state, I wonder?

It may be the humans' fault.

gessha3• 2 years ago

Yes, you can find some of the work on this topic under the terms open world recognition or open world X where X is a topic in computer vision or NLP.

https://arxiv.org/abs/2011.12906

intended9• 2 years ago

If a model can say ‘I don’t know’, then the hallucination problem would also be solved.

When we say “know” it usually means being factual. For an LLM to ‘know’ it doesn’t know, it would have had to move away from pure correlations on words, and meta processing about its own results.

I can see this happen with two LLMs working together (and there are Evals that use just this), however each LLM still has no self awareness of its limits.

This was a terribly convoluted argument to make.

empath-nirvana3• 2 years ago

The models that exist now say "I don't know" all the time. It's so weird that people keep insisting that it can't do things that it does.

Ask it what dark matter is, and it won't invent an answer, it will present existing theories and say that it's unknown.

Ask it about a person you know that isn't in it's data set and it'll tell you it has no information about the person.

Despite the fact that people insist that hallucinations are common and that it will invent answers if it doesn't know something frequently, the truth is that chatgpt doesn't hallucinate that much and will frequently say it doesn't know things.

One of the few cases where I've noticed it inventing things are that it often makes up apis for programming libraries and CLI tools that don't exist, and that's trivially fixable by referring it to documentation.

wseqyrku2• 2 years ago

Feeding the output to another inference would probably reduce hallucination? but I have this impression that the models talking to each other don't need to know English, a byte stream would do. Just like the hidden layers of an ANN.

andsoitis10• 2 years ago

the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations).

To answer "I don't know" requires one to know when you know. To know when you know in turn requires understanding.

MuffinFlavored• 2 years ago

how did LLMs get this far without any concept of understanding? how much further can they go until they become “close enough”?

t_mann4• 2 years ago

Maybe it requires understanding, maybe there are other ways to get to 'I don't know'. There was a paper posted on HN a few weeks ago that tested LLMs on medical exams, and one interesting thing that they found was that on questions where the LLM was wrong (confidently, as usual), the answer was highly volatile with respect to some prompt or temperature or other parameters. So this might show a way for getting to 'I don't know' by just comparing the answers over a few slightly fuzzied prompt variations, and just ask it to create an 'I don't know' answer (maybe with a summary of the various responses) if they differ too much. This is more of a crutch, I'll admit, arguably the LLM (or neither of the experts, or however you set it up concretely) hasn't learnt to say 'I don't know', but it might be a good enough solution in practice. And maybe you can then use that setup to generate training examples to teach 'I don't know' to an actual model (so basically fine-tuning a model to learn its own knowledge boundary).

somethingsaid• 2 years ago

I also wonder if having a hallucination-free LLM is even required for it to be useful. Humans can and will hallucinate (by this I mean make false statements in full confidence, not drugs or mental states) and they’re entrusted with all sorts of responsibilities. Humans are also susceptible to illusions and misdirection just like LLMs. So in all likelihood there is simply some state of ‘good enough’ that is satisfactory for most tasks. Perusing the elimination of hallucinations to the nth degree may be a fools errand.

skydhash• 2 years ago

Tools are not people and people should not be considered as tools. Imagine your hammer only hitting the nail 60% of the time! But workers should be allowed to stop working to negotiate work conditions.

sandworm1013• 2 years ago

They cannot say "I dont know" because they dont actually know anything. The answers are not comming from a thinking mind but a complex pattern-fitting supercomputer hovering over a massive table of precomputed patterns. It computes your input then looks to those patterns and spits out the best match. There is no thinking brain with a conceptual understanding of its own limitations. Getting an "i dont know" from current AI is like asking navigation software how far it is to the Simpsons house in Springfield: the machine spits out answers but cannot fathom the cultural reference that makes the answer impossible. Instead, it finds someone named simpson in the nearest realworld Springfield.

williamcotton3• 2 years ago

What if you worked on the problem and tried to come up with some kind of solution?

paulnpace• 2 years ago

My observation is that comments similar to GP come from the constant anthropomorphizing of things by marketers and without realizing that this subtle influence on language can alter one's view on what "I" means. The first time I really noticed this was when someone using Siri produced a response from Siri that included "I". Ever since I am acutely aware of this every time I hear or read it.

caditinpiscinam3• 2 years ago

In real world conversations, people are constantly saying "I don't know"; but that doesn't really happen online. If you're on reddit or stack overflow or hacker news and you see a question you don't know the answer to, you normally just don't say anything. If LLMs are being trained on conversations pulled from the internet then they're missing out on a ton of uncertain responses.

Maybe LLMs don't truly "understand" questions, but they're good at looking like they understand questions. If they were trained with more uncertain content, perhaps they'd be better at expressing uncertainty as well.

DiogenesKynikos7• 2 years ago

The answers are not comming from a thinking mind but a complex pattern-fitting supercomputer hovering over a massive table of precomputed patterns.

Are you sure you're not also describing the human brain? At some point, after we have sufficiently demystified the workings of the human brain, it will probably also sound something like, "Well, the brain is just a large machine that does X, Y and Z [insert banal-sounding technical jargon from the future] - it doesn't really understand anything."

My point here is that understanding ultimately comes down to having an effective internal model of the world, which is capable of taking novel inputs and generating reasonable descriptions of them or reactions to them. It turns out that LLMs are one way of achieving that. They don't function exactly like human brains, but they certainly do exhibit intelligence and understanding. I can ask an LLM a question that it has never seen before, and it will give me a reasonable answer that synthesizes and builds on various facts that it knows. Often the answer is more intelligent than what one would get from most humans. That's understanding.

fl73056• 2 years ago

a complex pattern-fitting supercomputer hovering over a massive table of precomputed patterns

That was perhaps true of earlier and smaller LLMs, like GPT-1 and GPT-2.

But as they grew larger and were trained with more and more data, they changed from pure pattern matching to implementing algorithms to compress more information into their structure than pure pattern matching can achieve.

These algorithms are incomplete and buggy, but they are nonetheless executing algorithms, and not just pattern matching.

This phenomenom can be seen in toy-sized neural networks. For instance, addition of two input values modulo a constant. As a small network is trained, at some point the internal structure can change from pattern matching to implementing addition using Fourier transforms. This is clearly visible in its structure. The network now performs the task perfectly for all inputs, regardless of having seen them in training.

You can ask ChatGPT 4 to execute an algorithm for you. I just tried this one:

  I would like to play a game, where you are the host. We start off with a score that is 1234143143. At the start  of each turn, you tell me the current score and ask me if I want to play a or b. If I choose a, the score is halved, and 30 is added. If I choose b, the score is doubled, and 40 is subtracted. Only use integers and round down.

It will happily execute this algorithm. For large numbers, it is slightly off on the arithmetic. When I asked it to double check, it did so using Python code. After that, it kept using Python code to perform the math. It was also able to reason intelligently about different outcomes if always picking a (or b) given different starting points.

Now, if you have enough memory and training data, of course you can build a gigantic lookup table that has this exact text sequence in it to replicate "executing the algorithm" I described.

Is that your claim? How much memory are we talking about? My feeling is that it'd be far more than the number of atoms in the universe.

  Me: How far it is to the Simpsons house in Springfield?

  ChatGPT: The Simpsons' house in Springfield is a fictional location from the animated TV series "The Simpsons." Since Springfield is a fictional town and its location is not consistently specified in the series, it's not possible to determine a real-world distance to the Simpsons' house.


  Me: Do I have a brother? 
  
  Please answer with either of:
  
  a) The answer
  
  b) There is an answer, but I do not know
  
  c) There is no answer
  
  
  ChatGPT: b) There is an answer, but I do not know

furyofantares• 2 years ago

But the can say "I don't know." They can be trained to do so ("as of my knowledge cutoff in September 2020 I don't know who Bob Whatgenflabl is") and they can be given context that makes it more likely they do so (I've had good success with this for RAG applications, and extremely little, but not zero, for general prompts.)

cultureswitch6• 2 years ago

The answers are not comming from a thinking mind but a complex pattern-fitting supercomputer hovering over a massive table of precomputed patterns. It computes your input then looks to those patterns and spits out the best match

Can you tell that's not how you yourself function?

jncfhnb6• 2 years ago

They cannot say "I dont know" because they dont actually know anything.

print(“I don’t know”)

You don’t need proper cognition to identify that the answer is not stored in source data. Your conception of the model is incomplete as is easily demonstrable by testing such cases now. Chat gpt does just fine on your simpsons test.

You, however, have made up an answer of how something works that you don’t actually know despite your cognition

naasking7• 2 years ago

The answers are not comming from a thinking mind but a complex pattern-fitting supercomputer hovering over a massive table of precomputed patterns.

Sorry, how do you know that "thinking minds" are not also just "complex pattern-fitting supercomputers hovering over a massive table of precomputed patterns"?

rf15• 2 years ago

I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often

I feel like writing an entire paper about the practical approach to the problems posed in this paper, but you'll probably have to first formally define the language used in the training data before you can try to map it (through training and sampling algos, which this paper conveniently skipped) to the target form. This sounds really fun at first, but then we're once again talking about the strict formalisation of natural language (which you could still do - the training data is limited and fixed!)

mr_toad4• 2 years ago

Saying “I don’t know” implies you understand what “I” means.

karol• 2 years ago

Organisms that evolved to perceive true reality instead of the "user interface" have smaller chances of survival. Donald Hoffman

tibbydudeza• 2 years ago

The brain fakes it to approximate reality - more so for reasons of the limitations of the wetware it runs on than anything else.

For others here is a TED talk.

https://www.youtube.com/watch?v=oYp5XuGYqqY

Borealid4• 2 years ago

hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function.

With this definition, you can trivially prove the titular sentence - "hallucination is inevitable" - is untrue.

Let your LLM have a fixed input context length of one byte. Continue training the LLM until such a time as it replies to the input "A" with "yes" and all other inputs with "no".

Define your computable ground truth function such that the correct output for the input "A" is "yes" and the correct output for all other inputs is "no".

This LLM provable never hallucinates - we have exhaustively verified that its output matches the ground truth function for all possible inputs.

There is nothing stopping inductively increasing the size of the input context and the number of entries in the ground truth table arbitrarily, and at no step do hallucinations become "inevitable".

resolutebat2• 2 years ago

Continue training the LLM until such a time as it replies to the input "A" with "yes" and all other inputs with "no".

This is basically the same as saying "train your LLM until they never hallucinate", which reduces your claim to a tautology: an LLM trained not to hallucinate does not hallucinate. The trick is making that happen.

ProxCoques• 2 years ago

As I always tell my students: the solution to unreliable code is not to put the bugs in there in the first place.

Borealid4• 2 years ago

It's a tautology that for a given truth table mapping inputs to "correct" outputs there exists a function that produces that mapping.

Saying that you can't train an LLM to NOT hallucinate is saying that it's impossible for any LLM to always produce output matching any particular truth table.

There may exist truth tables where it's not possible to produce an LLM to match them (for some finite size of LLM perhaps), but my claim isn't a tautology - it's just an assertion that there exist some truth tables which an LLM can be trained to match.

It may be tricky to make the LLM in the first place, but it's certainly not as tricky to verify it. You can test it with every input you consider to be present in the truth table, record its results, and throw it away if it doesn't match. The only possible results here are either non-convergeance or a "perfect" LLM.

You can get rid of the non-convergeance if you bound the size of the LLM and literally iteratively verify every single possible model. The authors of the paper didn't do that. For trivially sized models and inputs, that's completely possible. For a 7B parameter model, nobody is doing that ever. But you can prove the title statement wrong with a ten-parameter model and a one-byte input.

less_less• 2 years ago

I also disagree with the paper, but not for the same reason.

With this definition, you can trivially prove the titular sentence - "hallucination is inevitable" - is untrue.

Unsurprisingly, that one sentence fragment doesn't capture the entirety of their assumptions. Instead they prove something intuitively obvious, along the lines of: LLMs with arbitrary-length inputs and certain resource restrictions (e.g. they can take up to poly-time to compute, and this poly-time behavior must be provable, so that during training they don't take even longer by mistake) cannot compute certain functions that don't have those restrictions (e.g. can take more than poly-time, or must take poly-time but a proof of this is not needed). For some cases this proof assumes P != NP. Then they argue that some useful real-world questions are likely to be in the class that the LLM cannot compute, basically because you can ask math problems to LLMs and math problems are sometimes really hard.

This formal model is asymptotic (assumes arbitrary-length inputs etc), but in my experience this kind of theorem is usually true for realistic problems even at modest query lengths.

But this isn't the same as proving that hallucination is inevitable, because (according to any reasonable definition) an LLM (or like, a person, or whatever) should be allowed to say "I don't know", and this should not be considered a hallucination. Then an LLM (or whatever) can avoid hallucinating, and the question becomes how much useful work it can do without hallucinating.

Borealid4• 2 years ago

It's not a bad paper honestly, I just don't like it when people take a line from it and assume something untrue.

The pigeonhole principle proves that if you only have N slots to work with, and you need to fit N+1 items into them, you're going to get at least one slot with at least two items. That makes sense, and it logically follows that constrained functions can't perfectly mirror less-constrained ones: at some point a "wrong" and a "right" input have to produce the same output.

mepiethree• 2 years ago

There is nothing stopping inductively increasing the size of the input context and the number of entries in the ground truth table arbitrarily

This isn’t induction. You’ve only done the base case, not the induction hypothesis or induction step. Maybe you’ve done those steps in your head but that’s not really a trivial proof as you claim.

Borealid4• 2 years ago

Induction is "if this is possible for value X, then it is also possible for value X+1".

Where X isn't used as part of the step this is always true. Nothing I did depends on the size of either the input nor the truth table, so long as both are finite-size and so long as the truth table can be expressed as a function of the input.

An LLM is an arbitrary convolution of the input text; for any mapping, some function you can call an "LLM" produces that function.

Sparkyte• 2 years ago

Definitely a given, it isn't like AI has an actual brain capable of resolving and forming new connections. The LLM and human brains is that LLMs are interactive compendiums and our brains organize and sort information that ensures survival as an organism. There is no survival of whether or not LLMs are accurate and a machine wouldn't understand what is good or bad without weighted context. Its good for analyze, process, storing, retrieving and decomposing information. It isn't good at understanding, validating and forming connections between the things it says and what you want of it. It lacks comprehension, it doesn't lack composure.

golol4• 2 years ago

The result in the theorem applies to your brain. Your brain can be modelled as an LLM in the sense of the paper up to arbitrarily small error.

The result is a diagonalization argument that is not very relevant for the real world.

wredue• 2 years ago

your brain can be modelled as an LLM in the sense of the paper

The vast majority of people actually writing LLMs don’t claim this, and in fact, actually claim the very opposite: that LLMs do not accurately model a human brain in any capacity.

The fact is that science has no clue what happens in the nucleus of a neuron, so claiming that computer scientists must is… well. You fill in the word.

breck• 2 years ago

capable of resolving and forming new connections
There is no survival of whether or not LLMs are accurate

I agree that today's LLMs are still missing important components like these needed for breakout intelligence, but I would not be surprised if researchers discover how to add them (and other important things) within 0-5 years.

carlossouza• 2 years ago

Someone smart once said:

If it is good, we call it "creativity."

If it is bad, we call it "hallucination."

This isn't a bug (or limitation, as the authors say). It's a feature.

tgv4• 2 years ago

Asking it to write code for you is basically asking it to hallucinate.

gardenhedge2• 2 years ago

I don't think so. I think it's asking it to repeat code it has been trained on

LightBug1• 2 years ago

I imagine the gold is in knowing whether the LLM understands when it's doing either?

Isn't this the difference between a human and an LLM?

A human knows it's making an educated guess and (should) say so. Or it knows when it's being creative, and can say so.

If it doesn't know which is which, then it really does bring it home that LLM's are not that much more than (very sophisticated) mechanical input-output machines.

vladms• 2 years ago

You mean "some humans know and could say so". And this reflection process is not captured in the data we fed to LLM-s (like let's say a lesson in which teacher asks "do you know X?", and students first answer "I don't know", etc.)

Also, LLM-s could report more statistical measures for each answer and external tools could interpret them.

devjab• 2 years ago

Isn’t it always hallucinating though? We just don’t care when it gets it “right”. As I understand it, it’s still just probability based on what is likely to be a good set of words to answer the prompt tasking it. It doesn’t actually know anything, it’s just extremely good at making stuff up.

Which is still very useful for a lot of things. Just maybe not things to which value is assigned based on how efficient and correct the answer is. Like you can have GPT make a marketing campaign for you, or you can have it design all the icons you need for your application UI, but you can’t reliably make it wrote high performance back-end code without having humans judge the results. Similarly you can’t use it to teach anyone anything, not really, because unless you’re already an expert on the subject being taught, you aren’t likely to spot when it gets things wrong. I guess you can argue that a lot of teaching is flawed like that, and you wouldn’t be wrong. Like, I was taught that the pyramids was build by slave labour, even after the archeological evidence had shown this to be likely false. But our text books were a decade old because our school didn’t really renew them very often… in such a case GPT might have been a more correct teacher, but the trick is that you won’t really know. Which is made even more complicated by the fact that it might teach different things to different students. Like, I just asked ChatGPT 3.5 who build the pyramids in 3 different prompts, in one it told me it was ordinary people. In the others it told me it was mostly skilled labour under guidance of “architects” and “engineers”. Still better than teaching us it was done by slave labour like my old book, but the book was still consistent in what was considered to be the truth at the time.

audunw• 2 years ago

True, but I think we can fall into the trap of expecting too much of LLMs. Their knowledge can seem perfect. They can answer almost anything, so it’s easy to get the illusion that they can answer anything truthfully.

In terms of what we can expect of future improvements, I think it’s overly optimistic to expect any kind of super intelligence beyond what we see today (that is, having access to all the worlds publicly available information, or rapidly generating texts/images/videos that fall into existing creative patterns).

I suspect that more creative intelligence requires an extremely fine balance to not “go crazy”.. that is, producing output we’d consider creative rather than hallucinations.

I think getting this balance right will get exponentially harder as we create feedback loops within the AI that let its intelligence evolve.

And it’s entirely possible that humans have already optimised this creative intelligence feedback loop as much as the universe allows. Having a huge amount of knowledge can obviously benefit from more neurons/storage. But we simply don’t know if that’s true for creative intelligence yet

badgersnake• 2 years ago

True, but I think we can fall into the trap of expecting too much of LLMs.

We’re already well past that point. Why? Because saying incredible things about AI attracts VC money.

Seb-C• 2 years ago

That is correct, it's always hallucinating and making things up.

Just because those hallucinations sometimes randomly happens to be right, people concluded that being wrong is the exception, while being right is somehow the rule.

It's like when people read [insert millenias old text here], finds a part that happens to illustrate something in their life today and conclude that it is a prophecy that predicted the future.

The meaning/truth in those is nothing more than a cognitive bias from the mind of the reader, not an inherent quality of the text.

somewhereoutth4• 2 years ago

For heavy LLM users, there is probably a dopamine hit when it does something right, much as gamblers get a hit when the fruit machine pays out. Perhaps LLM use is no more productive than gambling, and perhaps can be abused in a similar way.

zer00eyz• 2 years ago

This isn't a bug

If it isn't a bug, it dam well isn't a hallucination, or creativity.

This is a deeply integrated design defect. One that highlights what we're doing (statistically modeling lots of human language)...

Throwing more data against this path isnt going to magically make it wake up and be an AGI. And this problem is NOT going to go away.

The ML community need to back off the hype train. The first step is them not anthropomorphizing their projects.

precompute• 2 years ago

Ah, the AI hype is now entering the "let's be real" phase. Haven't seen a frenzied post on alignment in a while now.

Culonavirus• 2 years ago

The hype is insane. Listen, I think LLMs still have a lot of room to grow and they're already very useful, but like some excellent researchers say, they're not the holy grail. If we want AGI, LLMs are not it. A lot of people seem to think this is an engineering issue and that LLMs can get us there, but they can't, because it is not an engineering issue.

hnfong• 2 years ago

Do you have evidence to back your claims up besides "the hype is overblown"? Because hype only indicates that the precise hyped up claims are wrong, it doesn't imply the opposite extreme (i.e. LLMs can never achieve AGI) must be true.

mistermann5• 2 years ago

What kind(s) of an issue do you think it is fundamentally?

PopePompus• 2 years ago

I don't think you can say with confidence that the LLM approach will not lead to AGI, unless you understand in detail how human intelligence operates, and can show that no modification to current LLM architectures can achieve the same or superior results. I think the fact that adding "attention" to LLMs made a huge difference means that we are probably still in the low hanging fruit stage of LLM architecture development, and a few more design improvements on a par with "attention" might lead to something that could legitimately be called AGI. Many people, myself included, believe that LLMs are now exhibiting emergent behavior properties. If that's true, then saying that LLMs are not intelligent because they just predict the next token of output is like saying collections of neurons cannot be intelligent because they just stimulate other neurons chemically.

Der_Einzige• 2 years ago

I'm going to take the exact opposite take and claim that "some excellent researchers" support it.

"AGI" is practically already here, you just don't want to admit it: https://www.noemamag.com/artificial-general-intelligence-is-...

bbor• 2 years ago

I’m sorry… does this paper just point out that LLMs by definition are not as good at holding data as a direct database? Cause A) duh and b) who cares, they’re intuitive language transformers, not knowledge models.

Maybe I’m missing something obvious? This seems like someone torturing math to imply outlandish conclusions that fit their (in this case anti-“AI”) agenda.

anonylizard• 2 years ago

It at least disproves LLMs from being 'god models'. They will never be able to solve every problem perfectly.

ninetyninenine26• 2 years ago

Humans aren't God models either. The goal is to get this thing to the level of a human. God like levels are not possible imo.

TylerE• 2 years ago

I miss the days when HN posts about hallucinating were about microdosing.

hyperpape• 2 years ago

The result seems to rely on stipulating the LLM must answer true or false to all its questions, and can't say "I don't know." So it's an interesting result, but it's not obvious that it tells us much about our actual problem, which is 100% about how to get a system that accurately understand the level of confidence it should have.

cjdell• 2 years ago

I thought the industry was already experimenting with the idea that you have another LLM observing the output of the primary LLM which is trained more towards safety than creativity.

On top of that it would be good if the safety LLM could give a confidence score in the answer given by the main LLM. Then you can try multiple attempts with different parameters and only show the highest confidence answer to the user.

franze• 2 years ago

Hallucinations and Ideas are the same thing.

DebtDeflation• 2 years ago

There used to be an entire sub-field of NLP called Open Domain Question Answering (ODQA). It extensively studied the problem of selecting the best answer from the set of plausible answers and devised a number of potential strategies. Like everything else in AI/ML it fell victim to the "bitter lesson", in this case that scaling up "predict the next token" beats an ensemble of specialized linguistic-based methods.

falling_myshkin• 2 years ago

For those who don't know: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

I agree with you for the NLP domain, but I wonder if there will also be a bitter lesson learned about the perceived generality of language for universal applications.

sgammon• 2 years ago

Splinters inevitable: An innate limitation of working with wood /s

HKH23• 2 years ago

People expect splinter-free epistemology instead of just putting on some gloves.

Scene_Cast2• 2 years ago

I don't think anyone has mentioned Bayesian Neural Nets (I forget the exact term). Sure, the paradigm adds an order of magnitude overhead (at least - and that's why I've never seen it used in the industry), but you can bolt it on to existing architectures.

The basic idea is that besides the probabilities, the network also spits out confidence (IIRC based on how out-of-distribution the input is). There's been a ton of work on getting confidence values out of existing neural nets without as much overhead, but I've never seen those approaches replicate in the industry.

wnkrshm• 2 years ago

I would imagine that to propagate any confidence value through the system you'd need to have priors for the confidence of correctness for all data in your training set. (and those priors change over time)

somewhereoutth4• 2 years ago

For production systems the considerations are:

- how often does it error?

- how bad are the errors?

- how tolerable are the errors?

- how detectable are the errors?

- how recoverable are the errors?

For example, a pocket calculator that occasionally was out at the 3rd decimal place might do much more damage than one that quite regularly returned NaN.

LLMs error both regularly and badly, so detectability and recoverability are going to be crucial for useful deployment in systems outside of those that have a high tolerance for errors (e.g. algorithmic feeds).

intended9• 2 years ago

Yes. If you want to make something that works, your life is figuring out evaluation and LLM ops.

At that point, you know its not thinking, its doing token prediction.

macrolime• 2 years ago

They define an LLM as "a probabilistic model of a string that conditions the output at time t based on all the tokens that come before it in the string".

I think that definition is wide enough to include human intelligence, so their finding should be equally valid for humans.

jddj• 2 years ago

Silk silk silk silk silk silk silk.

What do cows drink?

moooo992• 2 years ago

I think that definition is wide enough to include human intelligence, so their finding should be equally valid for humans.

Which is definitely true. Human memory and the ability to correctly recall things we though we remembered is affected by a whole bunch of things and at times very unreliable.

However, human intelligence, unlike LLMs, is not limited to recalling information we once learned. We are also able to do logical reasoning, which seems to improve in LLMs, but is far from being perfect.

Another problem is how different we treat the reliability of information depending on the source, especially based on personal bias. I think that is a huge factor, because in my experience, LLMs tend to quickly fall over and change their opinion based on user input.

ben_w5• 2 years ago

We can do logical reasoning, but we're very bad at it and often take shortcuts either via pattern matching, memory, or "common sense".

Baseball and bat together cost $1.10, the bat is $1 more than the ball, how much does the ball cost?

A French plane filled with Spanish passengers crashes over Italy, where are the survivors buried?

An armed man enters a store, tells the cashier to hand over the money, and when he departs the cashier calls the police. Was this a robbery?

magicalhippo• 2 years ago

We are also able to do logical reasoning

This is effectively like coming up with an algorithm and then executing it. So how good/bad are these LLMs if you asked them to generate say a LUA script to compute the answer, ala counting occurrences problem mentioned in a different comment, and then pass that off to a LUA interpreter to get the answer?

Cacti• 2 years ago

When we can define and measure intelligence, perhaps these discussions will be meaningful.

zamalek• 2 years ago

Which might explain the evolutionary reason for dreaming: it prunes hallucination. Might it make sense to interleave training and dreaming?

Almondsetat• 2 years ago

Of course it's inevitable.

Things can be facts or deductions of facts (or both). If I ask an LLM the date of birth of Napoleon and it doesn't have it in its dataset there are only 2 options: either it has other facts from which Napoleon's birthday can be deduced or it doesn't. If it does then by improving the LLM we will be able to make more and more deductions, it if doesn't then it can only hallucinate. Since there will always be a set of facts that the LLM is not aware of and which cannot be deduced we will always have hallucinations

jtc331• 2 years ago

Is this a way of saying that large language models don’t have the concept of “I don’t know”?

andybak• 2 years ago

Your "Of course" is a red flag.

Unless you have a very precise definition of "LLM" then there's no "of course". It's possible to imagine a hypothetical software system capable of returning "I don't know" or "I'm not sure".

I haven't RTFA but I presume they are arguing within specific constraints. The important point is - could an augmented LLM or an "LLM plus something else" solve this.

I don't have an answer to that but I don't think it's an "of course" type answer.

samatman3• 2 years ago

I don't see why that's inevitable at all. I immediately recognize that I don't happen to know Napoleon's birthday, and act on that basis.

keepamovin3• 2 years ago

This is why you need to pair language learning with real world experience. These robots need to be given a world to explore -- even a virtual one -- and have consequences within, and to survive it. Otherwise it's all unrooted sign and symbol systems untethered to experience.

phh2• 2 years ago

I think I agree with you (I even upvoted), but this might be an anthropomorphism.

Back like 3-5 years ago, we already thought that about LLMs: They couldn't answer questions about what would fall when stuff are attached together in some non-obvious way, and the argument back then was that you had to /experience/ it to realize it. But LLMs have long fixed those kind of issues.

The way LLMs "resolve" questions is very different from us. At this point, I think that if we want to prove that LLMs need to be rooted in the real world to achieve intelligence, we need to find some real-world phenomenon that is so obvious that noone ever wrote about it... but then we'd have written about it?

barnacs• 2 years ago

Think of it this way:

Intelligent beings in the real world have a very complex built-in biological error function rooted in real world experiences: sensory inputs, feelings, physical and temporal limitations and so on. You feel pain, joy, fear, have a limited lifetime, etc.

"AI" on the other hand only have an external error function, usually roughly designed to minimize the difference of the output from that of an actually intelligent real world being.

keepamovin3• 2 years ago

It's not anthropomorphism, it's just life.

The exploring of an environment, learning and surviving is not unique to humans, but all life on this planet.

Sure, some of us may not see them (our alive brothers and sisters on this planet) as "intelligent", but undeniably their learned, and hereditary behaviors (learned through evolution, I guess), are very intelligent, especially for survival, and are tethered to the real world. They are all part of an intelligence which we all share.

They are (these behaviors), in fact, from a certain point of view -- reflections of the real world, or models of it. In a way that is even closer than language, or at least, orthogonal. We need that orthogonal source of real-world data (which is even more enriched with reality-info than language is), to bootstrap these AIs to a higher level of utility. :) hahaha :)

Regarding your point on degrees of fidelity with reality, the way I explain that is that they (OpenAI et al, or the AI models) have extracted world-information from semantic mining of enormous data. That's good, but only up to a point, as we see.

I think we need real world experience to get the rest of the way. Or, put it differently, to make it a helluva lot easier to get that juicy world info haha! :)

I don't think we to "prove" it. It's not math. It's a blackbox. We just have to try it and see. :)

pfdietz• 2 years ago

A hallucinator seems like an excellent thing to have as a component to an intelligent system, but it must be coupled with evaluators.

Such an architecture seems plausible for the human brain as well.

viktour19• 2 years ago

If hallucination is inevitable? What should developers do?

Design user experiences that align users with this behaviour!

Relatedly, I built a game to demonstrate how one might calibrate users to the responses of LLMs:

https://news.ycombinator.com/item?id=39255583

demondemidi• 2 years ago

I bet that, basically, an LLM is just a part of a future AI. The same way that a distributor is part of an internal combustion engine. Or the way Minsk’s described The Society of Mind. Eager to see if an LLM can be bolted on to some new kind of giant model that does something like mitigate the LLMs weaknesses. Maybe it’ll be a collection of radically different models working together in 20 years, and not just a single model. Like, uh, our own brains. It reminds me of how lex and yacc are super important as grammar and tokenizer, but are only just the front end to much larger projects.

WirelessGigabit• 2 years ago

I feel the same way about information provided by LLMs as with lots of pre-LLM articles and overall YouTube 'information' videos.

Sources matter. You're either rehashing information from a collection of sources or you have your own data to prove your statements.

coffeefirst• 2 years ago

Yeah, this is why I really like Kagi's approach: it's clearly marked for what is, and cites its sources so you can verify the quality of the answer (or at least get a ballpark idea of where it's coming from) yourself.

lebuffon• 2 years ago

Complete AI layman here but it seems to me that the human mind's architecture has an overarching "executive" module that is involved in managing the numerous "expert" systems that do other stuff for us. (split brain experiments demonstrate multiple "minds" in the brain)

Based on what we know about different systems in the brain it might be a stretch to expect hallucination free AGI, using only a single LLM.

Swizec• 2 years ago

Based on what we know about different systems in the brain it might be a stretch to expect hallucination free AGI, using only a single LLM.

Based on what we know about brains, it might be a stretch to expect hallucination free AGI. I’ve yet to meet a general intelligence that doesn’t hallucinate.

Lots of fun examples from chickens who turn off when they see a straight line to cats who get spooked by cucumbers and the endless fun cognitive biases humans fall for.

pylua• 2 years ago

Sometimes hallucination is sort of a feature instead of a bug. For instance, if I ask it how to do something and it hallucinates , usually it is perfectly logical for it to work the way it suggests, even though it is wrong.

In other words, it can be a good feature request or idea.

Fripplebubby3• 2 years ago

The core argument in this paper it seems to me from scanning it is that because P != NP therefore LLMs will hallucinate answers to NP-complete problems.

I think this is a clever point and an interesting philosophical question (about math, computer science, and language), but I think people are mostly trying to apply this using our commonsense notions of "LLM hallucination" rather than the formal notion they use in this paper, and I don't see an obvious connection, since commonsense hallucinations (eg inventing chapters of a novel when asked to produce summaries, inventing specific details when asked) don't seem to be NP-complete problems but rather are hallucinatory for some other interesting reason. (I apologize if I have not captured the paper correctly and would welcome correction on that, I read it quickly)

The statement that the formal world (the world of math and logic and formal grammars) is a subset of the "real" world (or perhaps, the world of natural language) is really interesting to me as well. Most humans can't solve formal logic problems and parse formal grammars but don't suffer from a (strong) hallucination effect, and can work in natural language in great proficiency. Is hallucination inevitable in humans since we also can't solve certain NP-complete problems? We have finite lifespans, after all, so even with the capabilities we might never complete a certain problem.

foobarian2• 2 years ago

Humans have some amount of ability to recognize they hit a wall and adjust accordingly. On the other hand this (completeness theorems, Kolmogorov complexity, complexity theory) was only arrived at what, in the 20th century?

digitalsushi• 2 years ago

'Adjust accordingly' includes giving up and delivering something similar to what I asked, but not what I asked; is this the point at which the circle is complete and AI has fully replaced my dev team?

p1esk6• 2 years ago

The only way to reduce hallucinations in both humans and LLMs is to increase their general intelligence and their knowledge of the world.

robrenaud• 2 years ago

A smart bullshit artist who loves the sound of their own voice is going to generate more hallucinations than a less smart, more humble, more self aware person.

Making LLMs more knowledgeable is great (more data, bigger models, yay!), but there are other avenues of plausible attack as well. Enabling LLMs to know when they have veered off distribution might work. That is, the LLM doesn't have to know more of the world, it just has to know what it knows and stay there. A person who says "I don't know" is a lot more valuable than an overzealous one who spouts nonsense confidently. Encouraging an LLM to say that there is a disagreement about a topic rather than picking one lane is also a valuable way forward.

FpUser• 2 years ago

You post amounts to: in order to be smarter I need to increase my smartness. Great insight.

renegade-otter4• 2 years ago

It's statistical prediction. LLMs do not "understand" the world by definition.

Ask an image generator to make "an image of a woman sitting on a bus and reading a book".

Images will be either a horror show or at best full of weird details that do not match the real world - because it's not how any of this works.

It's a glorified auto-complete that only works due to the massive amounts of data it is trained on. Throw in any complex interactions it has not seen in the training data and it's all over.

Anotheroneagain3• 2 years ago

Your understanding is distorted by dealing mostly with psychotic people. Iron poisoning (well within the supposed healthy levels) and lead deficiency cause schizophrenia; normal people look childlike or "not intelligent" to the affected.

someplaceguy5• 2 years ago

because P != NP therefore LLMs will hallucinate answers to NP-complete problems.

I haven't read the paper, but that sounds like it would only be true if the definition of "hallucinating" is giving a wrong answer, but that's not how it's commonly understood.

When people refer to LLMs hallucinating, they are indeed referring to an LLM giving a wrong (and confident) answer. However, not all wrong answers are hallucinations.

An LLM could answer "I don't know" when asked whether a certain program halts and yet you wouldn't call that hallucinating. However, it sounds like the paper authors would consider "I don't know" to be a hallucinating answer, if their argument is that LLMs can't always correctly solve an NP-complete problem. But again, I haven't read the paper.

Fripplebubby3• 2 years ago

Yes, I think you're right. I think one way to phrase the authors' argument is:

* There is a class of problems which are harder than polynomial time complexity to solve, but are not np-complete

* LLMs will generate an "answer" in formal language to this class of problems posed to it

* LLMs can at most solve problems with polynomial time complexity due to their fundamental design and principles

* Therefore, LLMs cannot solve > polynomial problems and not np-complete problems either

All of which I buy completely. But I think what people are more interested in is, why is it that the LLM gives an answer when we can prove that it cannot answer this problem correctly? And perhaps that is more related to the commonsense notion of hallucination than I first gave it credit for. Maybe the reason that an LLM gives a formal language answer is the same reason it gives a hallucinatory answer in natural language. But I don't think the paper sheds light on that question

fauigerzigerk3• 2 years ago

Does the paper distinguish between hallucination and approximation?

If LLMs could be trained to approximate NP-complete functions rather than making stuff up, that would be good enough in many contexts. I guess it's what humans would do.

thargor90• 2 years ago

You cannot approximate NP-complete functions. If you could approximate them with a practically useful limited error and at most P effort you would have solved P=NP. (disclaimer my computer science classes have been a long time ago)

kenjackson• 2 years ago

Last I’d heard it was still open if P != NP. And most questions I’ve seen hallucinations on are not NP-Complete.

Animats• 2 years ago

Yes. It looks like they introduce infinities and then run into the halting problem for infinities. That may not be helpful.

The place where this argument gets into trouble is where it says "we define hallucination in a formal world where all we care about is a computable ground truth function f on S." This demands a reliable, computable predicate for truth. That alone is probably not possible.

If, however, we are willing to accept a ground truth function with outputs

    - True
    - False
    - Unknown
    - Resource limit exceeded

that problem can be avoided. Now the goal is manageable - return True or False only when those results are valid, and try to reduce the fraction of useful queries for which Unknown and Resource Limit Exceeded are returned.

The same problem comes up in program verification systems, and has been dealt with in the same way for decades. Sometimes, deciding if something is true is too much work.

Fripplebubby3• 2 years ago

Well put. Overall this paper feels very Gödel Incompleteness for LLMs which is _interesting_ and perhaps even valuable to somebody, but because it attaches itself to this hot query 'hallucination', I think some people are finding themselves searching this paper for information it does not contain.

samatman3• 2 years ago

Hallucination is a misnomer in LLMs and it depresses me that it has solidified as terminology.

When humans do this, we call it confabulation. This is a psychiatric symptom where the sufferer can't tell that they're lying, but fills in the gaps in their knowledge with bullshit which they make up on the spot. Hallucination is an entirely different symptom.

And no, confabulation isn't a normal thing which humans do, and I don't see how that fact could have anything to do with P != NP. A normal person is aware of the limits of their knowledge, for whatever reason, LLMs are not.

navane• 2 years ago

When you talk to your mom and you remember something happening one way, and she remembers it another way, but you both insist you remember it correctly, one of you is doing what the LLM is doing (filling up gaps of knowledge with bull shit). And even when later you talk about this on meta level, no one calls this confabulation because no one uses that word. Also this is not a psychiatric syndrome, it's just people making shit up, inadvertently, to tell a coherent story without holes. It very much sounds you did the same. Everyone does this all the time.

pixl977• 2 years ago

confabulation isn't a normal thing which humans do
A normal person is aware of the limits of their knowledge, for whatever reason, LLMs are not.

Eh, both of these things are far more complicated. People perform minor confabulations all the time. Now, there is a medical term for confabulation to about a more serious medical condition that involves high rates of this occurring coupled with dementia, and would be the less common form. We know with things like eye witness testimony people turn into confabulatory bullshit spewing devices very quickly, though likely due to different mechanisms like recency bias and over writing memories by thinking about them.

Coupled with that, people are very apt to lie about things they do know and can do for a multitude of reasons and attempting to teach an LLM to say "I don't know" when it doesn't know something, versus it just lying to you and saying it doesn't know will be problematic. Just see ChatGPT getting lazy in some of its releases for backfire effects like this.

ThrustVectoring• 2 years ago

The human visual system regularly fills in what it expects to see in between saccades and outside of focus points. Confabulating missing details is a feature, not a bug.

mr_toad4• 2 years ago

I suspect it’s just an incomplete memory and no “filter”. LLMs aren’t self-aware enough to judge their own confidence in their responses, so they don’t know when to shut up.

caeril• 2 years ago

And no, confabulation isn't a normal thing which humans do

This is entirely wrong. Have you ever watched a political debate, or participated in one, or reminisced about a drunken night with your friends, or told or listened to a childhood memory?

Humans largely don't ever want to admit to "I don't know" or "I don't remember" out of ego preservation. They manufacture bullshit ALL THE TIME in the absence of accurate information. There is nothing rare about this at all.

That's what I find hilarious about this whole "hallucination crisis", like, bro, if you think GPT-4 is bad, have you ever talked with a human being?

wedesoft• 2 years ago

Maybe one should use something like energy-based out of distribution detection to prevent LLMs from making bold statements even when the confidence should be low. https://arxiv.org/abs/2010.03759

bitwize• 2 years ago

but rather are hallucinatory for some other interesting reason.

In improv theater, the actor's job is to come up with plausible interactions. They are free to make shit up as they go along, hence improv, but they have to keep their inventions plausible to what had just happened before. So in improv if someone asks you "What is an eggplant?" it is perfectly okay to say "An eggplant is what you get when you genetically splice together an egg and a cucumber" or similar. It's nonsense but it's nonsense that follows nicely from what just came before.

Large language models, especially interactive ones, are a kind of improv theater by machine: the machine outputs something statistically plausible to what had just come before; what "statistically plausible" means is based on the data about human conversations that came from the internet. But if there are gaps in the data, or the data lacks a specific answer that seems to statistically dominate, it seems like giving a definitive answer is more plausible in the language model than saying "I don't know", so the machine selects definitive, but wrong, answers.

ninetyninenine26• 2 years ago

Fiction and story writing is hallucination. It is the opposite of a stochastic parrot.

We've achieved both extremes of AI. Computers can be both logical machines and hallucinators. Our goal is to create a machine that can be both at the same time and can differentiate between both.

That's the key. Hallucination is important but the key is for the computer to be self aware about when it's hallucinating.

Of course it's a hard problem but even humans hallucinate massively. Just look at religion. Only one religion can be right or none, that must mean,logically speaking all other religions are hallucinations.

beardedwizard8• 2 years ago

It is not the opposite of stochastic parrot, it is exactly the same thing only the predictions are worse due to sparse training data.

elicksaur• 2 years ago

Comparing religion to LLM mistakes is a case of the very prevalent anthropomorphism in society currently. I fear this misunderstanding and conflation will prevent us actually improving the tech.

Coming to an incorrect explanation such as, “Helios pulls the Sun across the sky every day,” is categorically different than a math program incorrectly returning the most likely next token in a sequence. LLMs don’t have beliefs at all.

Helios answers a question “Why does the Sun rise?” Holding such a belief shows a logical understanding that some force must be doing this, but due to a lack of knowledge of the world, the person comes up with an incorrect explanation.

LLMs can’t pose and reason about such questions. It is just not the same class of “hallucinations.” Assuming we’ve figured out cognition via word prediction is going to get us nowhere fast in the long term.

ninetyninenine26• 2 years ago

No. I never said we figured out cognition.

The LLM is still a black box feed forward network. It is the intricacies of how signals interact with each other in this network that we don't fully understand.

Word prediction and curve fitting are high level concepts we used to build an intelligence we don't fully understand.

Also belief and understanding are orthogonal concepts. I can believe something I don't understand and I can understand something I don't belief.

My claim here is that LLMs understand things from a certain aspect because LLMs can produce output indistinguishable from understanding.

Also because both the human brain and the LLM are black boxes there is no other metric we can use to gauge the level of understanding an LLM has other than comparing it's inputs and outputs to the human brain.

MauranKilom• 2 years ago

even humans hallucinate massively

Simpler example: Dreams.

ninetyninenine26• 2 years ago

Yeah good point. But dreams are easily distinguishable from reality.

Religion is often indistinguishable from truth and reality to those who hallucinate it.

AlexandrB2• 2 years ago

Most humans are extremely aware of the difference between dreams and reality. If LLMs had similar awareness when they hallucinated there wouldn't be a problem.

timeon• 2 years ago

Just look at religion.

This is bit off-topic but what I see as one of driving force behind existence of religions is need for personification. It seems easier for human to interact with the world and its elements by communicating with it as it was familiar parson-like entity.

Now when we talk about LLMs and AI in general, there is often personification as well.

ninetyninenine26• 2 years ago

LLMs are trained to actually imitate human understanding deliberately. The data is human and the high level training is defined as the most likely word prediction of human output data.

So not surprising to find aspects of personification in the LLM. It is trained on US.

NateEag• 2 years ago

Only one religion can be right or none, that must mean,logically speaking all other religions are hallucinations.

There are some mistakes in this sentence.

It is possible (if unlikely) that multiple religions accurately describe some aspects of the world, while being mistaken about others. That is, treating rigorous complete "correctness" as the only useful state a religion could have is very misleading. Newtonian physics and special relativity both fail to predict some observed phenomena, but they're still both useful (and not every religion claims rigorous perfect correctness, even if some do).

Even if some religions can be shown to be wrong, that doesn't automatically mean that they're hallucinations. People can believe things for plausible reasons and be wrong about them.

People can also have reasonable stances like "I cannot prove this is true, and would not try to, but my subjective personal experience of visions of God persuade me it's probably real."

That seems very different to me from an LLM hallucinating a paper from whole cloth out of the blue.

wseqyrku2• 2 years ago

So they always hallucinate, it's just sometimes good enough?

throwawaaarrgh• 2 years ago

LLMs literally just place words one in front of another based on a probability and the "goodness" of training data. Of course it's going to make stuff up.

graemebenzie2• 2 years ago

I think part of understanding is filling in the gaps between facts. AIs can't recognize when that gap is too large

graemebenzie2• 2 years ago

I think understanding comes from filling the gaps between the facts. AIs can't tell when the gap between points of knowledge is too large to interpolate.

valine• 2 years ago

This paper is arguing that it’s impossible for an LLM to know the answer to every question, therefore it’s impossible to eliminate hallucination.

It’s easy to imagine an LLM that responds “I don’t know” to all questions. An LLM like that isn’t very useful, but it also doesn’t hallucinate. Eliminating hallucination by teaching it to recognize what it doesn’t know is probably a more sane approach than teaching an LLM to know literally everything in the universe.

zyklonix• 2 years ago

Hallucinations are essential for divergent thinking. Not everything is solved following goal driven approaches. Check out DreamGPT: https://github.com/DivergentAI/dreamGPT

zuminator• 2 years ago

This is sort of like the compression algorithm "problem." For the overwhelming majority of inputs, compression algorithms don't compress, and it can be proven that on average they don't work. But we're not really interested in compressing things on average. What we use compression for amounts to edge cases of highly regularized or repeatable data.

Thus the fact that LLMs can be proven in general to hallucinate doesn't necessarily imply that they must hallucinate in the types of situations for which we use them for. The paper itself discusses a number of mitigating strategies -- such as supplementing their training data with current information or using multiple LLMs to vote on the accuracy of the outcome -- only to basically brush them aside and advise not to use LLMs in any sort of critical situation. And that's probably true enough today, but in the future I think these strategies will greatly reduce the severity of these hallucinations. Just as we as human beings have developed strategies to reduce our reliance on pure memory.

This reminds me of a deposition I had to give a number of years back. One of the lawyers asked me if I remembered how the plaintiff and I came to discuss a certain accusation leveled at him by the defendant. And I confidently stated, "Sure, he and I used to have various conversations about the issue and one day he the plaintiff brought up this thing that defendant said to him." And the lawyer says, if you want to, you can refer to your phone text log to refresh your memory. Then I looked at my phone, and the truth was that I myself had spoken to the defendant, and she told me the accusation, and then I went and shared it with the plaintiff. So, I basically remembered the situation exactly backwards, i.e., a hallucination, which I was able to repair by referring to real world information instead of just my memory.

educaysean• 2 years ago

Is it a theoretical distinction of "we can't get to 0%, but we can virtually trivialize it by reducing its frequency down to to 1x10^-8%" type of scenario? Or is it something that requires an external layer of control?

DinaCoder992• 2 years ago

"Hallucination" implies perception of non-real things, not generation of phrases that map poorly to reality (or are simply incoherent). It seems like a really bad term for this phenomenon.

TheBlight• 2 years ago

"Bullsh***ing" seems more apt.

earthwalker99• 2 years ago

It makes being wrong sound impressive and mysterious, so it's here to stay.

bonzaidrinkingb6• 2 years ago

We use "confabulation".

It is a feature, not a bug.

"confabulation" could be "solved" when LLMs realize they are uncertain on a reply and making things up. But that requires the LLM saying "I don't know" and rewarding that more than a wrong guess. That requires a change in loss functions and not even sure if all users desire that.

nedt7• 2 years ago

Well humans believe that vacination either kills people or gives them chips for tracking and the top politicians are lizard people drinking the blood of children kept in caves and they had to fake a pandemic to get them out. I'd say an A.I. hallucinating isn't that far off from real humans. It's rather the recipient that needs to interpret any response from either.

katehikes88• 2 years ago

Have you considered that parts of what you said might be true but you ridicule it only because you associate with the others might be untrue and maybe even ridiculous?

nedt7• 2 years ago

It might be true, or not. It might be ridiculous, or not. With _it_ being a message from a human or an AI. A hallucination is not so much a problem as long as there is not blind trust or a single source of truth. And oh boy would I like to be pure ridiculous or satirical with the example of what humans are believing.

bonzaidrinkingb6• 2 years ago

Around 12k fatal outcomes have been reported in the EU after vaccination, but it is not certain in all cases that vaccines were the cause.

The vaccine tracking chips come from two Microsoft (-affiliate) patents, one about using chips to track body activity to reward in cryptocurrency, and another about putting a vaccine passport chip in the hands of African immigrants. That vaccines contain tracking chips is a fabricated conspiracy to ridicule and obfuscate.

Lizard people is often an anti-semitic dog whistle.

Rich elites use blood transfusions of young people to combat aging and age-related disease.

Children have been kept in cages and feral children have lived in caves.

You likely made up the part about faking a pandemic to get children out of caves, unless you can point to discussion about these beliefs.

Real humans do hallucinate all the time.

dragonwriter4• 2 years ago

Real humans do hallucinate all the time.

No, they don't hallucinate “all the time”, but LLM “hallucination” is a bad metaphor, as the phenomenon is more like confabulation than hallucination.

Humans also don’t confabulate all the time, either, though.

nedt7• 2 years ago

In Germany and Austria we have those Querdenker telegram channels. All examples I‘ve given are coming from there. I‘d really like to say I‘ve made it up. But all you did with my message is also what I‘d do with AI output. It can be trained on wrong data, not understanding the question or make stuff up. Just like a human.

nedt7• 2 years ago

Here out from the German wikipedia about the lockdown being used to cover up the use of children for their blood: "According to the initial interpretation, the mass quarantine (the "lockdown") does not serve to combat the pandemic, but is intended to provide Trump and his allies with an excuse to free countless children from torture chambers, where adrenochrome is being withdrawn en masse on behalf of the elite." – translated via Google translate, but source is here with Die Zeit as source https://de.wikipedia.org/wiki/QAnon#cite_ref-29

nedt7• 2 years ago

Ok I take it back. Hallucination is a problem. Seeing the downvotes and comments here it does seem to be hard to see what is made up and what is just fuxxed up humans. The will to believe that humans can't be that stupid is bigger than I thought and same evaluation might lead to an AI response taken as truth if it's just calm enough and sounds plausible.

whycome• 2 years ago

Maybe this is why we need to force LLMs to dream. To get all the hallucinations out of the way during ‘down time’ and clean things up for ‘waking hours’.

emrah• 2 years ago

There is a fine line between hallucination and creativity. When we know the exact answer and seek it, we label misfires "hallucination".

Is it that the answer was not in the data set? Or that the LLM chose not to use it? Or something else?

I find it ironic that people claim LLMs are creative and then tryy to eliminate hallucinations. Maybe we need an explicit switch to turn on/off creative elements in general

dorkwood• 2 years ago

These scientists could use a bit of coaching from HN's resident LLM experts. Don't they know that if you write a detailed enough prompt, and ask the LLM to explain itself each step along the way, that you can eliminate hallucination?

Hallucination is inevitable: An innate limitation of large language models

Related Stories