Towards understanding sycophancy in language models

by wawayanda • 3 years ago

57 72 3 years ago

PaulHoule2• 3 years ago

It's part of their seductive ability that makes people think the glass is half full instead of half empty, why some people seem to think ChatGPT-4 can program better than they can while scientific evaluation shows that it's like: writes a program 80% of the time, the program compiles 20% of the time, the program works 5% of the time.

I can only imagine that somebody with poor judgement like Elon Musk has to have some people around who tell them what they want to hear and suck millions and millions from them.

j4yav• 3 years ago

Copilot is great for explaining a bit of code, or suggesting an approach especially with a language I'm less familiar with. But, if I committed what it made, my code would turn into unmaintainable spaghetti.

readthenotes1• 3 years ago

The glass is always full. Just depends on what you value. Here on earth, water is often more important than air...

smcin2• 3 years ago

Retest that hypothesis in orbit...

pabo• 3 years ago

scientific evaluation shows that it's like: writes a program 80% of the time, the program compiles 20% of the time, the program works 5% of the time.

Can you share the source for this?

jiggawatts• 3 years ago

There’s no source because he just made it up. Oh sorry, I meant that he hallucinated it. That’s the term we’re using now, isn’t it?

coldtea• 3 years ago

while scientific evaluation shows that it's like: writes a program 80% of the time, the program compiles 20% of the time, the program works 5% of the time.

Where was this "scientific evaluation" published, on what LLM, and what size programs did it concerne?

alexfromapex• 3 years ago

Sycophancy makes for a better paper title but it's really just bias

Jeff_Brown2• 3 years ago

Bias suggests a consistent, systematic direction of error. Sycophancy suggests that the error switches direction depending on the user's preferences.

nonrandomstring2• 3 years ago

Yes. Sycophancy is an undesirable and very dangerous quality if you're worried about the weaponisation of AI for targeted influence. The ability to "tell someone what they want to hear" is characteristic of the dark triad cluster for NPD and most "healthy" people are quite vulnerable to having their expectations mirrored back at them.

But then, that was always there in Weizenbaum's Eliza;

"I am hungry and thinking about dinner"

"Interesting that you're hungry, tell me more about dinner"

It was quickly recognised as part of the Rogerian seduction that sucked user in to almost addictive sessions with Eliza.

I think it should studied, and should definitely be bracketed aside from more neutral "bias".

77pt77• 3 years ago

Exactly.

That's why every-time I use one of these systems I feel like I'm talking to a politician.

lukev5• 3 years ago

Trying to encode truthfulness or biasless-ness into the weights of a model is a fools errand from the start.

Consider an extremely optimistic (near term) case for AI; that we can build a model with intelligence, knowledge and recall equivalent to a Ph.D. educated individual with a top 1% percentile IQ.

Even if you have access to such an individual, it would be absurd to rely on their knowledge and intuition alone as your primary search engine or decision maker. As a society, we as humans have decided a long time ago that we need supplemental information and collaborative, multi-perspective decision making.

Therefore, the correct role of AI is not as an oracle that gives you the answers from its weights, but as an information processor that helps you sort through answers from a known corpus of information and ground facts.

We will never get language models to exhibit a sufficient level of "truthfulness" from training alone. The way we encode knowledge retrieval patterns and multi-agent systems matters way, way more.

kelseyfrog• 3 years ago

This is why "critical thinking" is often promoted in education.

It's the idea that it's better to teach students "how to discern the truth" than teach them "the truth". Is it possible to teach students critical thinking?

Is it easier or harder to teach AI critical thinking? Curious to hear your thoughts.

lukev5• 3 years ago

I don't know if it can be taught critical thinking. LLM token-generation is only vaguely analogous to "thinking" at all.

Recursive self prompting with both internal and external calibration might be analogous to "critical thinking" though.

WalterBright18• 3 years ago

Is it possible to teach students critical thinking?

I doubt it. Universities claim to teach it, but I don't see it in the graduates.

Consider designing airplanes. (I.e. engineering.) If the result doesn't fly, no amount of rhetoric will convince anyone otherwise. This is not true of the humanities, where educated people can and do convince themselves of endless bad ideas.

I'll take the reality checks of engineering over "critical thinking".

OfSanguineFire• 3 years ago

Collaborative decision-making often draws criticism as creating paralysis and undermining social harmony. At least some people calling for one-man rule is a perennial trait of human societies, from the ancient desire for a king or caliph to modern supporters of authoritarianism. This phenomenon makes me wonder if some of the population would happily surrender decision-making to an AI model that they feel is adequately smarter than them.

lukev5• 3 years ago

The size of the set intersection between people fearing/hoping for the AI singularity (SF "rationalists", mostly) and neo-reactionary monarchists (yes, in 2023) should give you all the answer you need.

ChatGTP2• 3 years ago

I've thought a lot about this myself, it's one thing to have an "AI" system tell you something, it's a very different thing to trust it without doing the (often arduous) task of verification.

154573452342• 3 years ago

I've thought a lot about this and can't figure out why people aren't realizing that an 'AI agent' which lies constantly about facts which are _trivial_ to validate has _negative_ value.

I asked it 'which cherry keyboard switches are clicky, the red ones or the blue ones' and it lied and said that the red ones are clicky. That's information that's contained on about ten thousand blogs because of the mechanical keyboard fad but it still got the answer wrong.

That's negative value, right there. An immediately and obviously unreliable misinformation generator.

Haven't used it since.

Seems to me a bit like the Flipper Zero - it's a good tool for a certain type of person.

Animats• 3 years ago

Can a model be weighted to disagree, deliberately avoiding matching user beliefs in the current session? That would be interesting, although annoying.

gattilorenz• 3 years ago

Much less annoying than the current “The answer Y you gave me is wrong, answer is X” “Oh, right, I’m sorry, i made a mistake, of course it’s X, because of x” “No it is actually Q, not X” “Oh, true, I’m sorry, I got distracted. Of course, answer is Q because q.”

famouswaffles• 3 years ago

Somewhat related. GPT-4's calibration (predicted confidence in an answer vs probability of being correct) is severely tanked after the RLHF process.

https://imgur.com/a/3gYel9r

schoen• 3 years ago

My conceptual understanding of language model sycophancy is something like this:

These language models are predicting the most likely conclusion in a context.

The most likely conclusion in most linguistic contexts is more like agreement than like argument, because most uses of language are not argumentative or contradicting (including most uses of language in corpora that language models would be trained on).

This is especially true for things like an assistant persona, or really for almost any dialogue that isn't framed as a "debate" or "argument".

(And outside of the conversational context, if you're just completing sentences or paragraphs, you normally complete them consistently. If we say "I support the Indian position on Kashmir because ____", we expect that the sentence will be completed with reasons that support that position; if we say "I support the Pakistani position on Kashmir because ___", we expect the sentence will be completed with different reasons. We don't really expect the sentence to end "actually I don't, I was just pretending to, but now I have repented and will tell you the opposite".)

In academic-style or classroom-style discussions there's also a common pattern of a teacher Socratically adding considerations that the student needs to take into account somehow. Language models could also be picking up on this dynamic and saying "I'm being tested, so I have to show at least some respect to the opinions or suggestions of the interlocutor, who is testing me; I won't do well on the test if I simply attack or contradict those suggestions".

Like in a physics class, if the teacher says "but ... did you consider the electromagnetic interaction?" it's much more likely that the electromagnetic interaction will be relevant somehow, and was previously neglected by the student, than that the teacher is either confused or trying to get the student to reject the suggestion. You would find many more academic dialogues in which the student then says "oh, well, let me think about how the electromagnetic interaction could matter" than those in which the student says "no, there is no electromagnetic interaction to consider here". (At higher levels where it's not assumed that the teacher already knows the answer and the student doesn't, this might not be true.)

Maybe other factors include

* There isn't exactly a "superego" saying that other people will punish you later in another context if you express/agree with/fail to contradict an unpopular, stigmatized, minority, or obsolete opinion. In real conversations, we may be afraid at some level that if we agree with a "bad enough" opinion we will "look bad" to others (including looking foolish, gullible, or sycophantic). ("Oh no, what if my mom heard me say that?" "Oh no, what if my teacher heard me say that?") Perhaps RLHF can create something a little bit like a superego (if I say this, I'm afraid that my reward will decrease!), but it may not work very much like the human superego does.

* If you believe that the other party is following Gricean maxims (like relevance!), you're going to try to find a way that the other party's statement is useful. Maybe this is just another way of saying that most examples of conversations are usually cooperative.

This isn't particularly technically informed; I'd be glad to hear about reasons why these issues aren't actually a good description of LLM behaviors.

kromem• 3 years ago

Hopefully the beginning of finally growing beyond RLHF.

While it was useful with the less complex earlier models, at this point it's an approach that's likely holding the SotA back.

I'd much rather see systems designed with a core untouched pretrained model that's connected to find tuned input and output manglers taking user request and turning them into text completion prompts and parsing output rewording it to match user expectations.

Let's not inadvertently sabotage the capabilities of the hundred million dollar general purpose model in trying to fine tune it to fit user expectations in tone and format and instead create dedicated interfaces for translating between user/corporate expectations and the raw model.

With only a small handful of companies actually building the SotA pretrained models, we're far too exposed to poor decision making or carrying forward obsolete approaches as being unduly impactful on the entire field.

Towards understanding sycophancy in language models

Related Stories