On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs

by bilsbie • 7 months ago

30 44 7 months ago

buppermint• 7 months ago

From a quick read, this is cool but maybe a little overstated. From Figure 3, completely suppressing these neurons only reduces hallucinations by like ~5% compared to their normal state.

Table 1 is even more odd, H-neurons predicts hallucination ~75% of the time but a similar % of random neurons predict hallucinations ~60% of the time, which doesn't seem like a huge difference to me.

lukeinator42• 7 months ago

It is fairly well established that neurons in these artificial neural networks are polysemantic and information is represented in directions in the activation embedding space rather than neurons independently representing information (which is why anthropic is doing things like training sparse autoencoders). I haven't read the paper in depth but it seems like it is based on a fundamental misunderstanding about neurons in ANNs vs the brain.

readthenotes1• 7 months ago

It may be that there are just a few misfirings that dramatically degrade the results. Strokes in the human brain causing a few neurons to die can have outsized affects in the body...

GaggiX• 7 months ago

The people talking about semantics in the comment section seems to completely ignore the positive correlation of LLMs between accuracy and stated confidence, this is called calibration and this "old" blog post from a year ago already showed it, LLMs can know what they know: https://openai.com/index/introducing-simpleqa/

jurystillout26• 7 months ago

Many people seem to be claiming that "LLMs do what humans do / humans also hallucinate", as if the process of human knowledge is identical to the purely semantic knowledge of LLMs.

No. Human beings have experiential, embodied, temporal knowledge of the world through our senses. That is why we can, say, empirically know something, which is vastly different than semantically or logically knowing something. Yes, human beings also have probabalistic ways of understanding the world and interacting with others. We have many other forms of knowledge as well and the LLM way of interpreting data is by no means the primary way in which we feel confident that something is true or false.

That said, I don't get up in arms about the term "hallucination", although I prefer the term confabulation per neuroscientist Anil Seth. Many clunky metaphors are now mainstream, and as long as the engineers and researchers who study these kinds of things are ok with that, that's the most important thing.

But what I think all these people who dismiss objections to the term as "arguing semantics" are missing is the fundamental point: LLMs have no intent, and they have no way of distinguishing what data is empirically true or not. This is why the framing, not just the semantics, of this piece is flawed. "Hallucinations" is a feature of LLMs that exists at the very conceptual level, not as a design flaw of current models. They have pattern recognition, which gets us very far in terms of knowing things, but people who only rely on such methods of knowing are most often referred to as conspiracy theorists.

meowface• 7 months ago

On one hand this is true. On the other hand it does seem possible, in theory, that through enough post-training and other measures they could become a bit closer to human minds and not just be token guessers.

The human brain may at its fundamental level operate on the principles of predictive processing (https://slatestarcodex.com/2017/09/05/book-review-surfing-un...). It might be that it has many layers surrounding that raw predictive core which develop us into epistemological beings. The LLMs we see today may be in the very early stages of a similar sort of (artificial) evolution.

The reflexiveness with which even top models like Opus 4.5 will sometimes seamlessly confabulate things definitely does make it seem like it is a very deep problem, but I don't think it's necessarily unsolvable. I used to be among the vast majority of people who thought LLMs were not sufficient to get us to AGI/ASI, but I'm increasingly starting to feel that piling enough hacks atop LLMs might really be what gets there before anything else.

dang2• 7 months ago

[stub for offtopicness]

[submitters: one reason for not editorializing titles is it makes the threads be about that!]

bowsamic• 7 months ago

“Physical”? I don’t think that’s the right word to use

Miraltar2• 7 months ago

Agreed, the original title is better (and typo free) although a bit long.

H-Neurons: On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs

ZeroConcerns• 7 months ago

Yeah, actual title is "H-Neurons: On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs"

But regardless of title this is all highly dubious...

airhangerf15• 7 months ago

LLMs don't "hallucinate" or "lie." They have no intent. They're Weighted Random Word Generator Machines. They're train mathematically to create series of tokens. Whenever they get something "right," it's literally by accident. If you get that rate of accidental rightness up to 80%, and people suddenly thing the random word generator is some kind of oracle. It's not. It's a large model with an embedded space, tokens and a whole series of computationally expensive perceptron and attention blocks that generate output.

The title/introduction is very baited, because it implies some "physical" connection to hallucinations in biological organism, but it's focused on trying to single out certain parts of the model. LLMs are absolutely nothing at all like a biological system, of which our brains are orders of magnitudes more complex than the machines we've built that we no longer fully understand. Believing in these LLMs as being some next stage in understanding intelligence is hubris.

pektezol• 7 months ago

It’s been more than 3 years and people still can’t understand this.

amelius• 7 months ago

You're reasoning at the wrong abstraction level. This will not get you very far.

crazygringo2• 7 months ago

LLMs don't "hallucinate" or "lie." They have no intent.

You're just arguing about semantics. It doesn't matter in any substantial way. Ultimately, we need a word to distinguish factual output from confidently asserted erroneous output. We use the word "hallucinate". If we used a different word, it wouldn't make any difference -- the observable difference remains the same. "Hallucinate" is the word that has emerged, and it is now by overwhelming consensus the correct word.

Whenever they get something "right," it's literally by accident.

This is obviously false. A great deal of training goes into making sure they usually get things right. If an infinite number of monkeys on typewriters get something right, that's by accident. Not LLM's.

markus_zhang• 7 months ago

Maybe humans are just random word generators too? Just more sophisticated ones. But I think I read some pieces that we are not random word generators but something better. Forgot the word.

gchamonlive• 7 months ago

I think you are reading too much into the title. "Neuron" is a totally valid way of refering to the perception unit in such models. It's got nothing to do with biology apart from the name.

lambdaone3• 7 months ago

"Hallucinate" is a term of art, and does not imply a philosophical commitment to whether LLMs have minds. "Confabulation" might be a more appropriate term.

What is indisputable is that LLMs, even though they are 'just' word generators, are remarkably good at generating factual statements and accurate answers to problems, yet also regrettably inclined to generating apparenly equally confident counterfactual statements and bogus answers. That's all that 'hallucination' means in this context.

If this work can be replicated, it may offer a way to greatly improve the signal-to-bullshit ratio of LLMs, and that will be both impressive and very useful if true.

2026iknewit4• 7 months ago

This is still not true.

"Whenever they get something "right," it's literally by accident." "the random word generator"

First of, the input is not random at all which allows the question how random the output is.

Second, it compresses data which has an impact on that data. Probably cleaning or adjustment which should reduce 'random' even more. It compresses data from us into concepts. A high level concept is more robust than 'random'.

Thinking or reasoning models are also finetuning the response by walking the hyperspace and basically collecting and strengthening data.

We as humans do very similiar things and no one is calling us just random word predictors...

And because of this, "hallucinations -- plausible but factually incorrect outputs" is an absolut accurate description of what an LLM does when it response with a low probability output.

Humans also do this often enough btw.

Please stop saying an LLM is just a random word predictor.

dpweb• 7 months ago

We don't understand the brain. We fully understand what LLM are doing, humans built them. The idea we don't understand what LLMs are doing is magical. Magical is good for clicks and fundraising.

dudu24• 7 months ago

I'm tired of this pseudointellectual reductionist response. It's not "literally by accident" when they're trained to do something, as if we are not also machines that generate next actions based on learned neural weights and abstract (embedded) representations. Your issue is with semantics rather than content.

Obviously "hallucinate" and "lie" are metaphors. Get over it. These are still emergent structures that we have a lot to learn from by studying. But I suppose any attempt by researchers to do so should be disregarded because Person On The Internet has watched the 3blue1brown series on Neural Nets and knows better. We know the basic laws of physics, but spend lifetimes studying their emergent behaviors. This is really no different.

cyost• 7 months ago

Somewhat on the subject: here's a neuroscientist reflecting on our failure to model a worm's brain, a "mere" 302 neurons (3 parts, this one is the first). https://ccli.substack.com/p/the-biggest-mystery-in-neuroscie...

Biological systems are hard.

jrm4• 7 months ago

There is no such thing as a "hallucination" that could be isolated from "not a hallucination" in a provable systematic way because all they do is hallucinate.

I'm extremely comfortable calling this paper complete and utter bullshit (or, I suppose if I'm being charitable, extremely poorly titled) from the title alone.

eurekin• 7 months ago

We are in the vibe science era, it seems

2026iknewit4• 7 months ago

The Input of an LLM is real data. The n-dimensional space an LLM works in is a reflection of this. Statistical probably speaking there should be a way of knowing when an LLM is confident vs. when not.

This type of research is absolut valid.

An LLM is not just hallucinate.

lambdaone3• 7 months ago

Arguably, all we do is something similar to hallucination; it's just that hundreds of millions of years have selected against brains that generate internal states that lead to counter-survival behavior.

I recently almost fell on a tram as it accelerated suddenly; my arm reached out for a stanchion that was out of my vision, so rapidly I wasn't aware of what I was doing before it had happened. All of this occurred using subconscious processes, based on a non-physical internal mental model of something I literally couldn't see at the moment it happened. Consciousness is over-rated; I believe Thomas Metzinger's work on consciousness (specifically, the illusion of consciousness) captures something really important about the nature of how our minds really work.

lambdaone3• 7 months ago

The title as posted is misleading and sensational: we should use the actual paper title, "H-Neurons: On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs", which is far less sensational.

dijksterhuis• 7 months ago

please use the original title, unless it is misleading or linkbait; don't editorialize.

https://news.ycombinator.com/newsguidelines.html

dang2• 7 months ago

Yes, changed above now. (Submitted title was "Origin of Hallucination in LLMs, The physical source of hallucinations has found")

Submitters: If you want to say what you think is important about an article, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...

On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs

Related Stories