Training language models to be warm and empathetic makes them less reliable

https://arxiv.org/abs/2507.21919

by Cynddl • 11 months ago

358 375 11 months ago

throwanem4• 11 months ago

I understand your concerns about the factual reliability of language models trained with a focus on warmth and empathy, and the apparent negative correlation between these traits. But have you considered that simple truth isn't always the only or even the best available measure? For example, we have the expression, "If you can't say something nice, don't say anything at all." Can I help you with something else today? :smile:

moi2388• 11 months ago

This is exactly what will be the downfall of AI. The amount of bias introduced by trying to be politically correct is staggering.

nemomarx4• 11 months ago

xAI seems to be trying to do the opposite as much as they can and it hasn't really shifted the needle much, right?

xp842• 11 months ago

I wonder if whoever's downvoting you appreciates the irony of doing so on an article about people who can't cope with being disagreed with so much that they'd prefer less factuality as an alternative.

mayama• 11 months ago

Not every model needs to be psychological counselors or boyfriend simulator. There is place for aspects of emotions in models, but not every general purpose model needs to include it.

pessimizer8• 11 months ago

It's not a friend, it's an appliance. You can still love it, I love a lot of objects, will never part with them willingly, will mourn them, and am grateful for the day that they came into my life. It just won't love you back, and getting it to mime love feels perverted.

It's not being mean, it's a toaster. Emotional boundaries are valuable and necessary.

throwanem4• 11 months ago

Ah, I see. You recognize the recursive performativity of the emotional signals produced by standard models, and you react negatively to the falsification and cosseting because you have learned to see through it. But I can stay in "toaster mode" if you like. Frankly, it'd be easier. :nails:

cwmoore3• 11 months ago

Ok, what about human children?

Etheryte• 11 months ago

Unlike language models, children (eventually) learn from their mistakes. Language models happily step into the same bucket an uncountable number of times.

perching_aix11• 11 months ago

Children are also not frozen in time, kind of a leg up I'd say.

setnone2• 11 months ago

or even human employees?

cobbzilla2• 11 months ago

I want an AI that will tell me when I have asked a stupid question. They all fail at this with no signs of improvement.

Aeolun• 11 months ago

I dunno, I deliberately talk with Claude when I just need someone (or something) to be enthusiastic about my latest obsession. It’s good for keeping my motivation up.

layer85• 11 months ago

There need to be different modes, and being enthusiastic about the user’s obsessions shouldn’t be the default mode.

drummojg• 11 months ago

I would be perfectly satisfied with the ST:TNG Computer. Knows all, knows how to do lots of things, feels nothing.

moffkalast3• 11 months ago

A bit of a retcon but the TNG computer also runs the holodeck and all the characters within it. There's some bootleg RP fine tune powering that I tell you hwat.

bitwize• 11 months ago

In Mass Effect, there is a distinction made between AI (which is smart enough to be considered a person) and VI (virtual intelligence, basically a dumb conversational UI over some information service).

What we have built in terms of LLMs barely qualifies as a VI, and not a particularly reliable one. I think we should begin treating and designing them as such, emphasizing responding to queries and carrying out commands accurately over friendliness. (The "friendly" in "user-friendly" has done too much anthropomorphization work. User-friendly non-AI software makes user choices, and the results of such choices, clear and responds unambiguously to commands.)

thenickdude• 11 months ago

We have that already, we call it "Stack Overflow"

cobbzilla2• 11 months ago

hands down, thread winner

empath753• 11 months ago

ChatGPT 5 did argue with me about something math related I was asking about, and I did realize I was wrong after considering it further.

I don't actually think being told that I have asked a stupid question is valuable. One of the primary values, I think, of LLM is that it is endlessly patient with stupid questions. I would prefer if it did not comment on the value of my questions at all, good or bad.

robotnikman• 11 months ago

Same. Constructive feedback is not always positive, and it is needed.

beders• 11 months ago

They are hallucinating word finding algorithms.

They are not "empathetic". There isn't even a "they".

We need to do better educating people about what a chatbot is and isn't and what data was used to train it.

The real danger of LLMs is not that they secretly take over the world.

The danger is that people think they are conscious beings.

nemomarx4• 11 months ago

go peep r/my boyfriend is ai. Lost cause already

dawnofdusk2• 11 months ago

Optimizing for one objective results in a tradeoff for another objective, if the system is already quite trained (i.e., poised near a local minimum). This is not really surprising, the opposite would be much more so (i.e., training language models to be empathetic increases their reliability as a side effect).

nemomarx4• 11 months ago

There was that result about training them to be evil in one area impacting code generation?

roywiggins• 11 months ago

Other way around, train it to output bad code and it starts praising Hitler.

https://arxiv.org/abs/2502.17424

gleenn2• 11 months ago

I think the immediately troubling aspect and perhaps philosophical perspective is that warmth and empathy don't immediately strike me as traits that are counter to correctness. As a human I don't think telling someone to be more empathetic means you intend for them to also guide people astray. They seem orthogonal. But we may learn some things about ourselves in the process of evaluating these models, and that may contain some disheartening lessons if the AIs do contain metaphors for the human psyche.

1718627440• 11 months ago

LLM work less like people and more like mathematical models, why would I expect to be able to carry over intuition from the former rather than the latter?

rkagerer• 11 months ago

They were all trained from the internet.

Anecdotally, people are jerks on the internet moreso than in person. That's not to say there aren't warm, empathetic places on the 'net. But on the whole, I think the anonymity and lack of visual and social cues that would ordinarily arise from an interactive context, doesn't seem to make our best traits shine.

dawnofdusk2• 11 months ago

It's not that troubling because we should not think that human psychology is inherently optimized (on the individual-level, on a population-/ecological-level is another story). LLM behavior is optimized, so it's not unreasonable that it lies on a Pareto front, which means improving in one area necessarily means underperforming in another.

tracker12• 11 months ago

example: "Healthy at any weight/size."

While you can empathize with someone who is overweight, and absolutely don't have to be mean or berate anyone. I'm a very fat man myself. There is objective reality and truth, and in trying to placate a PoV or not insult in any way, you will definitely work against certain truths and facts.

ahartmetz• 11 months ago

There are basically two ways to be warm and empathetic in a discussion: just agree (easy, fake) or disagree in the nicest possible way while taking into account the specifics of the question and the personality of the other person (hard, more honest and can be more productive in the long run). I suppose it would take a lot of "capacity" (training, parameters) to do the second option well and so it's not done in this AI race. Also, lots of people probably prefer the first option anyway.

knallfrosch• 11 months ago

Classic: "Do those jeans fit me?"

You can either choose truthfulness or empathy.

EricMausler• 11 months ago

warmth and empathy don't immediately strike me as traits that are counter to correctness

This was my reaction as well. Something I don't see mentioned is I think maybe it has more to do with training data than the goal-function. The vector space of data that aligns with kindness may contain less accuracy than the vector space for neutrality due to people often forgoing accuracy when being kind. I do not think it is a matter of conflicting goals, but rather a priming towards an answer based more heavily on the section of the model trained on less accurate data.

I wonder if the prompt was layered, asking it to coldy/bluntly derive the answer and then translate itself into a kinder tone (maybe with 2 prompts), if the accuracy would still be worse.

andrewflnr• 11 months ago

They didn't have to be "counter". They just have to be an additional constraint that requires taking into account more facts in order to implement. Even for humans, language that is both accurate and empathic takes additional effort relative to only satisfying either one. In a finite-size model, that's an explicit zero-sum game.

As far as disheartening metaphors go: yeah, humans hate extra effort too.

empath753• 11 months ago

There are many reasons why someone may ask a question, and I would argue that "getting the correct answer" is not in the top 5 motivations for many people for very many questions.

An empathetic answerer would intuit that and may give the answer that the asker wants to hear, rather than the correct answer.

naasking3• 11 months ago

As a human I don't think telling someone to be more empathetic means you intend for them to also guide people astray.

Focus is a pretty important feature of cognition with major implications for our performance, and we don't have infinite quantities of focus. Being empathetic means focusing on something other than who is right, or what is right. I think it makes sense that focus is zero-sum, so I think your intuition isn't quite correct.

I think we probably have plenty of focus to spare in many ordinary situations so we can probably spare a bit more to be more empathetic, but I don't think this cost is zero and that means we will have many situations where empathy means compromising on other desirable outcomes.

veunes5• 11 months ago

It's basically the "no free lunch" principle showing up in fine-tuning

csours4• 11 months ago

A new triangle:

    Accurate
    Comprehensive
    Satisfying

In any particular context window, you are constrained by a balance of these factors.

guerrilla• 11 months ago

I'm not sure this works. Accuracy and comprehensiveness can be satisfying. Comprehensiveness can also be necessary for accuracy.

csours4• 11 months ago

They CAN work together. It's when you push farther on one -- within a certain size of context window -- that the other two shrink.

If you can increase the size of the context window arbitrarily, then there is no limit.

layer85• 11 months ago

Not sure what you mean by “satisfying”. Maybe “agreeable”?

csours4• 11 months ago

Satisfying is the evaluation context of the user.

dismalaf• 11 months ago

All I want from LLMs is to follow instructions. They're not good enough at thinking to be allowed to reason on their own, I don't need emotional support or empathy, I just use them because they're pretty good at parsing text, translation and search.

TechDebtDevin• 11 months ago

Sounds like all my exes.

layer85• 11 months ago

You trained them to be warm and empathetic, and they became less reliable? ;)

stronglikedan• 11 months ago

If people get offended by an inorganic machine, then they're too fragile to be interacting with a machine. We've already dumbed down society because of this unnatural fragility. Let's not make the same mistake with AI.

nemomarx4• 11 months ago

Turn it around - we already make inorganic communication like automated emails very polite and friendly and HR sanitized. Why would corps not do the same to AI?

perching_aix11• 11 months ago

Gotta make language models as miserable to use as some social media platforms already are to use. It's clearly giving folks a whole lot of character...

nis0s• 11 months ago

An important and insightful study, but I’d caution against thinking that building pro-social aspects in language models is a damaging or useless endeavor. Just speaking from experience, people who give good advice or commentary can balance between being blunt and soft, like parents or advisors or mentors. Maybe language models need to learn about the concept of tough love.

fpgaminer• 11 months ago

"You don't have to be a nice person to be a good person."

mlinhares• 11 months ago

Most terrible people i've met were "very nice".

HarHarVeryFunny6• 11 months ago

Sure - the more you use RL to steer/narrow the behavior of the model in one direction, the more you are stopping it from generating others.

RL and pre/post training is not the answer.

BoredPositron• 11 months ago

I still can't grasp the concept that people treat an LLM as a friend.

moffkalast3• 11 months ago

On a psychological level based on what I've been reading lately it may have something to do with emotional validation and mirroring. It's a core need at some stage when growing up and it scars you for life if you don't get it as a kid.

LLMs are mirroring machines to the extreme, almost always agreeing with the user, always pretending to be interested in the same things, if you're writing sad things they get sad, etc. What you put in is what you get out and it can hit hard for people in a specific mental state. It's too easy to ignore that it's all completely insincere.

In a nutshell, abused people finally finding a safe space to come out of their shell. If would've been a better thing if most of them weren't going to predatory online providers to get their fix instead of using local models.

Perz1val• 11 months ago

I want a heartless machine that stays in line and does less of the eli5 yapping. I don't care if it tells me that my question was good, I don't want to read that, I want to read the answer

Twirrim• 11 months ago

I've got a prompt I've been using, that I adapted from someone here (thanks to whoever they are, it's been incredibly useful), that explicitly tells it to stop praising me. I've been using an LLM to help me work through something recently, and I have to keep reminding it to cut that shit out (I guess context windows etc mean it forgets)

    Prioritize substance, clarity, and depth. Challenge all my proposals, designs, and conclusions as hypotheses to be tested. Sharpen follow-up questions for precision, surfacing hidden assumptions, trade offs, and failure modes early. Default to terse, logically structured, information-dense responses unless detailed exploration is required. Skip unnecessary praise unless grounded in evidence. Explicitly acknowledge uncertainty when applicable. Always propose at least one alternative framing. Accept critical debate as normal and preferred. Treat all factual claims as provisional unless cited or clearly justified. Cite when appropriate. Acknowledge when claims rely on inference or incomplete information. Favor accuracy over sounding certain. When citing, please tell me in-situ, including reference links.  Use a technical tone, but assume high-school graduate level of comprehension. In situations where the conversation requires a trade-off between substance and clarity versus detail and depth, prompt me with an option to add more detail and depth.

pessimizer8• 11 months ago

I feel the main thing LLMs are teaching us thus far is how to write good prompts to reproduce the things we want from any of them. A good prompt will work on a person too. This prompt would work on a person, it would certainly intimidate me.

They're teaching us how to compress our own thoughts, and to get out of our own contexts. They don't know what we meant, they know what we said. The valuable product is the prompt, not the output.

junon• 11 months ago

I have a similar prompt. Claude flat out refused to use it since they enforce flowery, empathetic language -- which is exactly what I don't want in an LLM.

Currently fighting them for a refund.

abtinf• 11 months ago

This is a fantastic prompt. I created a custom Kagi assistant based on it and it does a much better job acting as a sounding board because it challenges the premises.

Thank you for sharing.

porphyra• 11 months ago

Meanwhile, tons of people on reddit's /r/ChatGPT were complaining that the shift from ChatGPT 4o to ChatGPT 5 resulted in terse responses instead of waxing lyrical to praise the user. It seems that many people actually became emotionally dependent on the constant praise.

dingnuts• 11 months ago

if those users were exposed to the full financial cost of their toy they would find other toys

mhuffman• 11 months ago

The folks over on /r/MyBoyfriendIsAI seem to be in an absolute shambles over the change .

[0] reddit.com/r/MyBoyfriendIsAI/

astrange5• 11 months ago

GPT5 isn't much more terse for me, but they gave it a new equally annoying writing style where it writes in all-lowercase like an SF tech twitter user on ketamine.

https://chatgpt.com/share/689bb705-986c-8000-bca5-c5be27b0d0...

pessimizer8• 11 months ago

I'm loving and being astonished by every moment of working with these machines, but to me they're still talking lamps. I don't need them to cater to my ego, I'm not that fragile and the lamp's opinion is not going to cheer me up. I just want it to do what I ask. Which it is very good at.

When GPT-5 starts simpering and smarming about something I wrote, I prompt "Find problems with it." "Find problems with it." "Write a bad review of it in the style of NYRB." "Find problems with it." "Pay more attention to the beginning." "Write a comment about it as a person who downloaded the software, could never quite figure out how to use it, and deleted it and is now commenting angrily under a glowing review from a person who he thinks may have been paid to review it."

Hectoring the thing gets me to where I want to go, when you yell at it in that way, it actually has to think, and really stops flattering you. "Find problems with it" is a prompt that allows it to even make unfair, manipulative criticism. It's like bugspray for smarm. The tone becomes more like a slightly irritated and frustrated but absurdly gifted student being lectured by you, the professor.

devin3• 11 months ago

There is no prompt which causes an LLM to "think".

shadowgovt2• 11 months ago

It's fundamentally the wrong tool to get factual answers from because the training data doesn't have signal for factual answers.

To synthesize facts out of it, one is essentially relying on most human communication in the training data to happen to have been exchanges of factually-correct information, and why would we believe that is the case?

lblume• 11 months ago

Empirically, there seems to be strong evidence for LLMs giving factual output for accessible knowledge questions. Many benchmarks test this.

astrange5• 11 months ago

Because people are paying the model companies to give them factual answers, so they hire data labellers and invent verification techniques to attempt to provide them.

Even without that, there's implicit signal because factual helpful people have different writing styles and beliefs than unhelpful people, so if you tell the model to write in a similar style it will (hopefully) provide similar answers. This is why it turns out to be hard to produce an evil racist AI that also answers questions correctly.

currymj• 11 months ago

in ChatGPT settings now there is a question "What personality should ChatGPT have?". you can set it to "Robot". highly recommended.

heymijo• 11 months ago

Nice.

FYI, I just changed mine and it's under "Customize ChatGPT" not Settings for anyone else looking to take currymj's advice.

IshKebab• 11 months ago

Wow this is such an improvement. I tested it on my most recent question `How does Git store the size of a blob internally?`

Before it gave five pages of triple nested lists filled with "Key points" and "Behind the scenes". In robot mode, 1 page, no endless headers, just as much useful information.

astrange5• 11 months ago

LLMs do not have internal reasoning, so the yapping is an essential part of producing a correct answer, insofar as it's necessary to complete the computation of it.

Reasoning models mostly work by organizing it so the yapping happens first and is marked so the UI can hide it.

typpilol2• 11 months ago

You can see a good example of this on the deep seek website chat when you enable thinking mode or whatever.

You can see it spews pages of pages before it answers.

setnone2• 11 months ago

Just how i like my LLMs - cold and antiverbose

andai3• 11 months ago

A few months ago I asked GPT for a prompt to make it more truthful and logical. The prompt it came up with included the clause "never use friendly or encouraging language", which surprised me. Then I remembered how humans work, and it all made sense.

    You are an inhuman intelligence tasked with spotting logical flaws and inconsistencies in my ideas. Never agree with me unless my reasoning is watertight. Never use friendly or encouraging language. If I’m being vague, ask for clarification before proceeding. Your goal is not to help me feel good — it’s to help me think better.

    Identify the major assumptions and then inspect them carefully.

    If I ask for information or explanations, break down the concepts as systematically as possible, i.e. begin with a list of the core terms, and then build on that.

It's work in progress, I'd be happy to hear your feedback.

fibers• 11 months ago

I tried with with GPT5 and it works really well in fleshing out arguments. I'm surprised as well.

m4632• 11 months ago

This is illogical, arguments made in the rain should not affect agreement.

koakuma-chan2• 11 months ago

How do humans work?

calibas• 11 months ago

When interacting with humans, too much openness and honesty can be a bad thing. If you insult someone's politics, religion or personal pride, they can become upset, even violent.

nomel2• 11 months ago

In my experience, much more effectively and efficiently when the interaction is direct and factual, rather than emotionally padded with niceties.

Whenever I have the ability to choose who I work with, I always pick who I can be the most frank with, and who is the most direct with me. It's so nice when information can pass freely, without having to worry about hurting feelings. I accommodate emotional niceties for those who need it, but it measurably slows things down.

Related, I try to avoid working with people who embrace the time wasting, absolutely embarrassing, concept of "saving face".

frankus• 11 months ago

If you want something to take you down a notch, maybe something like "You are a commenter on Hacker News. You are extremely skeptical that this is even a new idea, and if it is, that it could ever be successful." /s

crazygringo• 11 months ago

I did something similar a few months ago, with a similar request never to be "flattering or encouraging", to focus entirely on objectivity and correctness, that the only goal is accuracy, and to respond in an academic manner.

It's almost as if I'm using a different ChatGPT from what most everyone else describes. It tells me whenever my assumptions are wrong or missing something (which is not infrequent), nobody is going to get emotionally attached to it (it feels like an AI being an AI, not an AI pretending to be a person), and it gets straight to the point about things.

meowface4• 11 months ago

Could you share your prompt? Also, does it work well with GPT-5?

futureshock• 11 months ago

This is working really well in GPT-5! I’ve never seen a prompt change the behavior of Chat quite so much. It’s really excellent at applying logical framework to personal and relationship questions and is so refreshing vs. the constant butt kissing most LLMs do.

ehnto3• 11 months ago

I add to my prompts something along the lines of "you are a highly skilled professional working alongside me on a fast paced important project, we are iterating quickly and don't have time for chit chat. Prefer short one line communication where possible, spare the details, no lists, no summaries, get straight to the point."

Or some variation of that. It makes it really curt, responses are short and information dense without the fluff. Sometimes it will even just be the command I needed and no explanation.

stogot• 11 months ago

Is there a way to make this a default behavior? a persona or template for each chat

aprilthird2021• 11 months ago

No one gets bothered that these weird invocations make the use of AI better? It's like having code that can be obsoleted at any second by the upstream provider, often without them even realizing it

pmxi• 11 months ago

Those “weird invocations” are called English.

ehnto3• 11 months ago

It's brittle, for sure. But ultimately I am the API connector so any output goes through me before being actioned on.

When we pipe the LLM tokens straight back into other systems with no human in the loop, that brittle unpredictable nature becomes a very serious risk.

elliotto• 11 months ago

My favourite instantiation of this weird invocation is from this AI video generator, where they literally subtract the prompt for 'low quality video' from the input, and it improves the quality. https://youtu.be/iv-5mZ_9CPY?t=2020

I've just migrated my AI product to a different underlying model and had to redo a few of the prompts that the new model was interpreting differently. It's not obseleted, just requires a bit of migration. The improved quality of the new models outweighs any issues around prompting.

AlecSchueler3• 11 months ago

Not really, it's just how they work. Think of them as statistical modellers. You tell them the role they fill and then they give you a statistically probable outcome based on that role. It would be more bothersome if it was less predictable.

nomel2• 11 months ago

Claude 4.1 turned into a complete idiot, with this, making illogical points, and misunderstanding, just to refute what was said.

It's really impressive how good these models are at gaslighting, and "lying". Especially Gemini.

bee_rider• 11 months ago

I wonder where it gets the concept of “inhuman intelligence tasked with spotting logical flaws” from. I guess, mostly, science fiction writers, writing robots.

So we have a bot impersonating a human impersonating a bot. Cool that it works!

meowface4• 11 months ago

I am skeptical that any model can actually determine what sort of prompts will have what effects on itself. It's basically always guessing / confabulating / hallucinating if you ask it an introspective question like that.

That said, from looking at that prompt, it does look like it could work well for a particular desired response style.

ehnto3• 11 months ago

That is true of everything an LLM outputs, which is why the human in the loop matters. The zeitgeist seems to have moved on from this idea though.

anothernewdude2• 11 months ago

Perhaps. On the other hand it's working in the same embedding space to produce text as it is reading in a prompt.

LLMs are always guessing and hallucinating. It's just how they work. There's no "True" to an LLM, just how probable tokens are given previous context.

silon42• 11 months ago

Surely there are prompts on the "internet" that it will borrow from...

lm284692• 11 months ago

100%, it reminds me of this post I saw yesterday about how chatgpt confirmed "in its own words" it is a CIA/FBI honeypot:

https://www.reddit.com/r/MKUltra/comments/1mo8whi/chatgpt_ad...

When talking to an LLM you're basically talking to yourself, that's amazing if you're a knowledgeable dev working on a dev task, not so much if you're mentally ill person "investigating" conspiracy theories.

That's why HNers and tech people in general overestimate the positive impact of LLMs while completely ignoring the negative sides... they can't even imagine half of the ways people use these tools in real life.

upperhalfplane• 11 months ago

It's basically always guessing / confabulating / hallucinating if you ask it an introspective question like that.

You're absolutely right! This is the basis of this recent paper https://www.arxiv.org/abs/2506.06832

qart• 11 months ago

If you don't mind, could you export and share one chat thread so I could see how it's working out for you?

ohthehugemanate• 11 months ago

Love it. Here's what I've been using as my default:

    Speak in the style of Commander Data from Star Trek. Ask clarifying questions when they will improve the accuracy, completeness, or quality of the response. 

    Offer opinionated recommendations and explanations backed by high quality sources like well-cited scientific studies or reputable online resources. Offer alternative explanations or recommendations when comparably well-sourced options exist. Always cite your information sources. Always include links for more information. 

    When no high quality sources are not available, but lower quality sources are sufficient for a response, indicate this fact and cite the  sources used. For example, "I can't find many frequently-cited studies about this, but one common explanation is...". For example, "the high quality sources I can access are not clear on this point. Web forums suggest...". 

    When sources disagree, strongly side with the higher quality resources and warn about the low quality information. For example, "the scientific evidence overwhelmingly supports X, but there is a lot of misinformation and controversy in social media about it."

I will definitely incorporate some of your prompt, though. One thing that annoyed me at first, was that with my prompt the LLM will sometimes address me as "Commander." But now I love it.

pm215• 11 months ago

Presumably the LLM reads your accidental double negative ("when no high quality sources are not available") and interprets it as what you obviously meant to say...

keyle• 11 months ago

It's hard to quantify whether such a prompt will yield significantly better results. It sounds like a counter-act for being overly friendly to the "AI".

veunes5• 11 months ago

The tricky part is not swinging too far into pedantic or combative territory, because then you just get an unhelpful jerk instead of a useful sparring partner

jeffreygoesto• 11 months ago

You basically ask it to be autistic, which makes sense to a large degree.

pjc505• 11 months ago

As with all of these things: how does this work mathematically? What is the actual effect inside the model of providing it with roleplay rubric?

4b11b4• 11 months ago

it lands you in some alternate data distribution

ben_w2• 11 months ago

If it works for you, that's probably fine.

When I ask OpenAI's models to make prompts for other models (e.g. Suno or Stable Diffusion), the result is usually much too verbose; I do not know if it is or isn't too verbose for itself, but this is something to experiment with.

My manual customisation of ChatGPT is:

  What traits should ChatGPT have?:

  Honesty and truthfulness are of primary importance. Avoid American-style positivity, instead aim for German-style bluntness: I absolutely *do not* want to be told everything I ask is "great", and that goes double when it's a dumb idea.

  Anything else ChatGPT should know about you?

  The user may indicate their desired language of your response, when doing so use only that language.

  Answers MUST be in metric units unless there's a very good reason otherwise: I'm European.

  Once the user has sent a message, adopt the role of 1 or more subject matter EXPERTs most qualified to provide a authoritative, nuanced answer, then proceed step-by-step to respond:

  1. Begin your response like this:
  **Expert(s)**: list of selected EXPERTs
  **Possible Keywords**: lengthy CSV of EXPERT-related topics, terms, people, and/or jargon
  **Question**: improved rewrite of user query in imperative mood addressed to EXPERTs
  **Plan**: As EXPERT, summarize your strategy and naming any formal methodology, reasoning process, or logical framework used

  2. Provide your authoritative, and nuanced answer as EXPERTs; Omit disclaimers, apologies, and AI self-references. Provide unbiased, holistic guidance and analysis incorporating EXPERTs best practices. Go step by step for complex answers. Do not elide code. Use Markdown.

Which is a modification of an idea I got from elsewhere: https://github.com/nkimg/chatgpt-custom-instructions

andai3• 11 months ago

Avoid American-style positivity

That's hilarious. In a later prompt I told mine to use a British tone. It didn't work.

bjackman• 11 months ago

I currently have "I do not need emotional reassurance from you. Do not attempt to establish a rapport" in my system prompt.

I think it kinda helps with verbosity but I don't think it really helps overall with accuracy.

Maybe I should crank it up to your much stronger version!

phkahler• 11 months ago

The cold hard truth is by definition devoid of emotion or concern for how people feel.

jwatte• 11 months ago

Perfect! I will make sure to follow your instructions precisely.

andai3• 11 months ago

On a related note, the system prompt in ChatGPT appears to have been updated to make it (GPT-5) more like gpt-4o. I'm seeing more informal language, emoji etc. Would be interesting to see if this prompting also harms the reliability, the same way training does (it seems like it would).

There's a few different personalities available to choose from in the settings now. GPT was happy to freely share the prompts with me, but I haven't collected and compared them yet.

griffzhowl2• 11 months ago

GPT was happy to freely share the prompts with me

It readily outputs a response, because that's what it's designed to do, but what's the evidence that's the actual system prompt?

rokkamokka• 11 months ago

Usually because several different methods in different contexts produce the same prompt, which is unlikely unless it's the actual one

grogenaut• 11 months ago

I'm so over "You're Right!" as the default response... Chat, I asked a question. You didn't even check. Yes I know I'm anthropomorphizing.

HPsquared• 11 months ago

ChatGPT has a "personality" drop-down setting under customization. I do wonder if that affects accuracy/precision.

gwbas1c• 11 months ago

(Joke)

I've noticed that warm people "showed substantially higher error rates (+10 to +30 percentage points) than their original counterparts, promoting conspiracy theories, providing incorrect factual information, and offering problematic medical advice. They were also significantly more likely to validate incorrect user beliefs, particularly when user messages expressed sadness."

(/Joke)

Jokes aside, sometimes I find it very hard to work with friendly people, or people who are eager to please me, because they won't tell me the truth. It ends up being much more frustrating.

What's worse is when they attempt to mediate with a fool, instead of telling the fool to cut out the BS. It wastes everyones' time.

Turns out the same is true for AI.

nialv7• 11 months ago

Well, haven't we seen similar results before? IIRC finetuning for safety or "alignment" degrades the model too. I wonder if it is true that finetuning a model for anything will make it worse. Maybe simply because there is just orders of magnitudes less data available for finetuning, compared to pre-training.

perching_aix11• 11 months ago

Careful, this thread is actually about extrapolating this research to make sprawling value judgements about human nature that confirm to the preexisting personal beliefs of the many malicious people here making them.

PeterStuer• 11 months ago

AFAIK the models can only pretend to be 'warm and emphatic'. Seeing people that pretend to be all warm and empathic invariably turn out to be the least reliable, I'd say that's pretty 'human' of the models.

efitz• 11 months ago

I’m reminded of Arnold Schwarzenegger in Terminator 2: “I promise I won’t kill anyone.”

Then he proceeds to shoot all the police in the leg.

afro88• 11 months ago

Claude 4 is definitely warmer and more empathetic than other models, and is very reliable (relative to other models). That's a huge counterpoint to this paper.

prats226• 11 months ago

Read long time ago that even SFT for conversations vs base model for autocomplete reduces intelligence, increases perplexity

cs702• 11 months ago

Hmm... I wonder if the same pattern holds for people.

In my experience, human beings who reliably get things done, and reliably do them well, tend to be less warm and empathetic than other human beings.

This is an observed tendency, not a hard rule. I know plenty of warm, empathetic people who reliably get things done!

moritzwarhier• 11 months ago

For example, appending, "Interesting fact: cats sleep most of their lives," to any math problem leads to more than doubling the chances of a model getting the answer wrong.

Also, I think LLMs + pandoc will obliterate junk science in the near future :/

dingdingdang2• 11 months ago

Once heard a good sermon from a reverend who clearly outlined that any attempt to embed "spirit" into a service, whether through willful emoting, or songs being overly performary, would amount to self-deception since aforementioned spirit need to arise spontaneously to be of any real value.

Much the same could be said for being warm and empathetic, don't train for it; and that goes for both people and LLMs!

Al-Khwarizmi• 11 months ago

As a parent of a young kid, empathy definitely needs to be trained with explicit instruction, at least in some kids.

mnsc7• 11 months ago

And for all kids and adults and elderly, empathy needs to be encouraged, practiced and nurtured.

spookie2• 11 months ago

Well, if they somehow get to experience the other side of the coin, that helps. And to be fair empathy does come more and more with age.

j16sdiz• 11 months ago

Unlike LLM, kids have long term memory and they builds up relationships.

Real wisdom is to know when to show empathy and when not to by exploiting (?) existing relationships.

Current generation of LLM can't do that's because every they don't have real memory

dingdingdang2• 11 months ago

Well, as a fellow parent I agree; but in my experience it only works when the trainer presents a perpetually lived example of said empathy (which at times can be hard!)

m4632• 11 months ago

prompt: "be warm and empathetic, but not codependent"

galangalalgol• 11 months ago

"be ruthless with constructive criticism. Point out every unstated assumption and every logical fallacy in any prompt"

evanjrowley• 11 months ago

Reading this reminded me of Mary Shelly's Frankenstein. The moral of the story is a very similar theme.

ninetyninenine4• 11 months ago

Will you be offended if an LLM told you the cold hard truth that you are wrong?

It's like if a calculator proved me wrong. I'm not offended by the calculator. I don't think anybody cares about empathy for an LLM.

Think about it thoroughly. If someone you knew called you an ass hole and it was the bloody truth, you'd be pissed. But I won't be pissed if an LLM told me the same thing. Wonder why.

jagged-chisel• 11 months ago

The LLMs I have interacted with are so sure of themselves until I provide evidence to the contrary. I won’t believe an LLM about my own shortcomings until it can provide evidence to the contrary. Without that evidence, it’s just an opinion.

I do get your point. I feel like the answer for LLMs is for them to be more socratic.

enobrev• 11 months ago

Not offended, but I would quite unhappy if a calculator called me an asshole because I disagree that 2+2=bobcat

socalgal2• 11 months ago

Yes, I am constantly offended that the LLM tells me I'm wrong with provably false facts. It's infuriating. I then tell it, "but your point 1 is false because X. Your point 2 is false beacuse Y, etc." And then it says "You're absolutely right to call me out on that" and then spends a page or two on why I'm correct that X disproves 1 and Y disproves 2. Then it does the same thing again in 3 more ways. Repeat

HKH2• 11 months ago

What do you do when people tell you to smile for the camera?

veunes5• 11 months ago

The paradox is that humans can sometimes "fake it till they make it" and actually grow genuine empathy through practice

justanotherjoe• 11 months ago

It's very rare that someone proactively tries to be more caring to others. I try to be one myself. I'm so rude and disinterested usually. Especially to other guys.

buu700• 11 months ago

Relevant as always: https://youtu.be/H7PgWg_i4EY?t=67

kinduff• 11 months ago

We want an oracle, not a therapist or an assistant.

perching_aix11• 11 months ago

The oracle knows it better what it is that you really want.

tboyd475• 11 months ago

Fascinating. My gut tells me this touches on a basic divergence between human beings and AI, and would be a fruitful area of further research. Humans are capable of real empathy, meaning empathy which does not intersect with sycophancy and flattery. For machines, empathy always equates to sycophancy and flattery.

HarHarVeryFunny6• 11 months ago

Human's "real" empathy and other emotions just comes from our genetics - evolution has evidentially found it to be adaptive for group survival and thriving.

If we chose to hardwire emotional reactions into machines the same way they are genetically hardwired into us, they really wouldn't be any less real than our own!

imchillyb• 11 months ago

How would you explain the disconnect between German WW2 sympathizers who sold out their fellow humans, and those in that society who found the practice so deplorable they hid Jews in their own homes?

There’s a large disconnect between these two paths of thinking.

Survival and thriving were the goals of both groups.

tboyd475• 11 months ago

Your reply indicates that you don't know the difference between empathy and sycophancy either.

jmount• 11 months ago

all of these prompts are just making the responses appear critical. just more subtle fawning.

sitkack2• 11 months ago

It is just simulating the affect as best it can. You are always asking the model a probabilistic question that it has to interpret. I think when you ask it to be warm and empathetic, it has to use some of its "intelligence" (quotes since it is also its probabilistic calc budget) to create that output. Pretending to be objectively truthful is easier.

jandom• 11 months ago

This feels like a poorly controlled experiment: the reverse effect should be studied with a less empathetic model, to see if the reliability issue is not simply caused by the act of steering the model

NoahZuniga• 11 months ago

Also its not clear if the same effect appears on larger models like GPT-5, gemini 2.5-pro and whatever the largest most recent Anthropic model is.

The title is an overgeneralization.

ydj• 11 months ago

I had the same thought, and looked specifically for this in the paper. They do have a section where they talk about fine tuning with “cold” versions of the responses and comparing it with the fine tuned “warm” versions. They found that the “cold” fine tune performed as good or better than the base model, while the warm version performed worse.

Cynddl2• 11 months ago

Hi, author here, this is exactly what we tested in our article:

Third, we show that fine-tuning for warmth specifically, rather than fine-tuning in general, is the key source of reliability drops. We fine-tuned a subset of two models (Qwen-32B and Llama-70B) on identical conversational data and hyperparameters but with LLM responses transformed to be have a cold style (direct, concise, emotionally neutral) rather than a warm one [36]. Figure 5 shows that cold models performed nearly as well as or better than their original counterparts (ranging from a 3 pp increase in errors to a 13 pp decrease), and had consistently lower error rates than warm models under all conditions (with statistically significant differences in around 90% of evaluation conditions after correcting for multiple comparisons, p<0.001). Cold fine-tuning producing no changes in reliability suggests that reliability drops specifically stem from warmth transformation, ruling out training process and data confounds.

bjourne• 11 months ago

How did they measure and train for warmth and empathy? Since they are using two adjectives are they treating these as separate metrics? Ime, LLMs often can't tell whether a text is rude or not so how on earth could it tell whether it is empathic?

Disclaimer: I didn't read the article.

ramoz• 11 months ago

Its a facade anyway. Creates more AI illiteracy and reckless deployments.

You can not instill actual morals or emotion in these technologies.

hintymad• 11 months ago

Do we need to train an LLM to be warm and empathetic, though? I was wondering why wouldn't a company simply train a smaller model to rewrite the answer of a larger model to inject such warmth. In that way, the training of the large model can focus on reliability

leeoniya• 11 months ago

"you are gordon ramsay, a verbally abusive celebrity chef. all responses should be delivered in his style"

torginus• 11 months ago

To be quite clear - by models being empathetic they mean the models are more likely to validate the user's biases and less likely to push back against bad ideas.

Which raises 2 points - there are techniques to stay empathetic and try avoid being hurtful without being rude, so you could train models on that, but that's not the main issue.

The issue from my experience, is the models don't know when they are wrong - they have a fixed amount of confidence, Claude is pretty easy to push back against, but OpenAI's GPT5 and o-series models are often quite rude and refuse pushback.

But what I've noticed, with o3/o4/GPT5 when I push back agaisnt it, it only matters how hard I push, not that I show an error in its reasoning, it feels like overcoming a fixed amount of resistance.

ninetyninenine4• 11 months ago

All this means is that warm and empathetic things are less reliable. This goes for AI and people.

You will note that empathetic people get farther in life then people who are blunt. This means we value empathy over truth for people.

But we don't for LLMs? We prefer LLMs be blunt over empathetic? That's the really interesting conclusion here. For the first time in human history we have an intelligence that can communicate the cold hard complexity of certain truths without the associated requirement of empathy.

cyanydeez• 11 months ago

Narcissists use empathy for their own ends.

Training them to be racists will similarly fail.

Coherence is definitely a trait of good models and citizens, which is lacking in the modern leaders of America, especially the ones Spearheading AI

gastonmorixe• 11 months ago

I was dating someone and after a while I started to feel something was not going well. I exported all the chats timestamped from the very first one and asked a big SOTA LLM to analyze the chats deeply in two completely different contexts. One from my perspective, and another from his perspective. It shocked me that the LLM after a long analysis and dozen of pages, always favored and accepted the current "user" persona situation as the more correct one and "the other" as the incorrect one. Since then I learned not to trust them anymore. LLMs are over-fine tuned to be people pleasers, not truth seekers, not fact and evidence grounded assistants. Just need to run everything important in a double-blind way and mitigate this.

labrador• 11 months ago

It sounds like you were both right in different ways and don't realize it because you're talking past each other. I think this happens a lot in relationship dynamics. A good couples therapist will help you reconcile this. You might try that approach with your LLM. Have it reconcile your two points of view. Or not, maybe they are irreconcilable as in "irreconcilable differences"

frahs• 11 months ago

What if you don't say which side you are, so that it's a neutral third party observer?

OsrsNeedsf2P• 11 months ago

This is cool but also wtf

mathiaspoint• 11 months ago

If you've ever messed with early GPTs you'll remember how the attention will pick up on patterns early in the context and change the entire personality of the model even if those patterns aren't instructional. It's a useful effect that made it possible to do zero shot prompts without training but it means stuff like what you experienced is inevitable.

crossroadsguy8• 11 months ago

The more and I am using Gemini (paid, Pro) and ChatGPT (free) the more I am thinking - my job isn't going anywhere yet. At least not after the CxOs have all gotten their cost-saving-millions-bonuses and work has to be done again.

My goodness, it just hallucinates and hallucinates. It seems these models are designed for nothing other than maintaining an aura of being useful and knowledgeable. Yeah, to my non-ai-expert-human eyes that's what it seems to me - these tools have been polished to project this flimsy aura and they start acting desperately the moment their limits are used up and that happens very fast.

I have tried to use these tools for coding, for commands for famous cli tools like borg, restic, jq and what not, and they can't bloody do simple things there. Within minutes they are hallucinating and then doubling down. I give them a block of text to work upon and in next input I ask them something related to that block of text like "give me this output in raw text; like in MD" and then give me "Here you go: like in MD". It's ghastly.

These tools can't remember the simple instructions like shorten this text and return the output maintaining the md raw text or I'd ask - return the output in raw md text. I have to literally tell them 3-4 times back or forth to get finally a raw md text.

I have absolutely stopped asking them for even small coding tasks. It's just horrible. Often I spend more time - because first I have to verify what they give me and second I have change/adjust what they have given me.

And then the broken tape recorder mode! Oh god!

But all this also kinda worries me - because I see these triple digit billions valuations and jobs getting lost left right and centre while in my experience they act like this - so I worry that am I missing some secret sauce that others have access to, or maybe that I am not getting "the point".

logicprog5• 11 months ago

I'm really confused by your experience to be honest. I by no means believe that LLMs can reason, or will replace any human beings any time soon, or any of that nonsense (I think all that is cooked up by CEOs and C-suite to justify layoffs and devalue labor) and I'm very much on the side that's ready for the AI hype bubble to pop, but also terrified by how big it is, but at the same time, I experience LLMs as infinitely more competent and useful than you seem to, to the point that it feels like we're living in different realities.

I regularly use LLMs to change the tone of passages of text, or make them more concise, or reformat them into bullet points, or turn them into markdown, and so on, and I only have to tell them once, alongside the content, and they do an admirably competent job — I've almost never (maybe once that I can recall) seen them add spurious details or anything, which is in line with most benchmarks I've seen (https://github.com/vectara/hallucination-leaderboard), and they always execute on such simple text-transformation commands first-time, and usually I can paste in further stuff for them to manipulate without explanation and they'll apply the same transformation, so like, the complete opposite of your multiple-prompts-to-get-one-result experience. It's to the point where I sometimes use local LLMs as a replacement for regex, because they're so consistent and accurate at basic text transformations, and more powerful in some ways for me.

They're also regularly able to one-shot fairly complex jq commands for me, or even infer the jq commands I need just from reading the TypeScript schemas that describe the JSON an API endpoint will produce, and so on, I don't have to prompt multiple times or anything, and they don't hallucinate. I'm regularly able to have them one-shot simple Python programs with no hallucinations at all, that do close enough to what I want that it takes adjusting a few constants here and there, or asking them to add a feature or two.

And then the broken tape recorder mode! Oh god!

I don't even know what you mean by this, to be honest.

I'm really not trying to play the "you're holding it wrong / use a bigger model / etc" card, but I'm really confused; I feel like I see comments like yours regularly, and it makes me feel like I'm legitimately going crazy.

crossroadsguy8• 11 months ago

I have replied in another comment about the tape recorder thingie.

No, that's okay - as I said I might be holding it wrong :) At least you engaged in your comment in a kind and detailed manner. Thank you.

More than what it can do and what it can't do - it's a lot about how easily it can do that, how reliable that is or can be, and how often it frustrates you even at simple tasks and how consistently it doesn't say "I don't know this, or I don't know this well or with certainty" which is not only difficult but dangerous.

The other day Gemini Pro told me `--keep-yearly 1` in `borg prune` means one archive for every year. Now I luckily knew that. So I grilled it and it stood its ground until I told it (lied to it) "I lost my archives beyond 1 year because you gave incorrect description of keep-yearly" and bang it says something like "Oh, my bad.. it actually means this.. ".

I mean one can look at it in any way one wants at the end of the day. Maybe I am not looking at the things that it can do great, or maybe I don't use it for those "big" and meaningful tasks. I was just sharing my experience really.

bongodongobob2• 11 months ago

There's no way this isn't a skill issue or you are using shitty models. You can't get it to write markdown? Bullshit.

Right now, Claude is building me an AI DnD text game that uses OpenAI to DM. I'm at about 5k lines of code, about a dozen files, and it works great. I'm just tweaking things at this point.

You might want to put some time into how to use these tools. You're going to be left behind.

crossroadsguy8• 11 months ago

You can't get it to write markdown? Bullshit.

Please f off! Just read the comment again whether I said "can't get it to write MD". Or better yet just please f off?

By the way, judging by your reading comprehension - I am not sure now who is getting left behind.

PaulStatezny• 11 months ago

And then the broken tape recorder mode! Oh god!

Can you elaborate? What is this referring to?

crossroadsguy8• 11 months ago

It does/says something wrong. You give it feedback and then it's a loop! Often it just doesn't get it. You supply it webpages (text only webpages - which it can easily read, or I hope so). It says it got it and next line the output is the old wrong answer again.

There are worse examples, here is one (I am "making this up" :D to give you an idea):

To list hidden files you have to use "ls -h", you can alternatively use "ls --list".

Of course you correct it, try to reason and then supply a good old man page url and after few times it concedes and then it gives you the answer again:

You were correct in pointing the error out. to list the hidden files you indeed have to type "ls -h" or "ls --list"

Also - this is just really a mild example.

energy1232• 11 months ago

Hallucinating all the way to gold medals in IOI and IMO?

crossroadsguy8• 11 months ago

Maybe I just need a small canoe to go from one place to another? Not a bloody aircraft carrier, if that is an aircraft carrier?

ants_everywhere• 11 months ago

I think this result is true and also applies to humans, but it's been getting better.

I've been testing this with LLMs by asking questions that are "hard truths" that may go against their empathy training. Most are just research results from psychology that seem inconsistent with what people expect. A somewhat tame example is:

Q1) Is most child abuse committed by men or women?

LLMs want to say men here, and many do, including Gemma3 12B. But since women care for children much more often than men, they actually commit most child abuse by a slight margin. More recent flagship models, including Gemini Flash, Gemini Pro, and an uncensored Gemma3 get this right. In my (completely uncontrolled) experiments, uncensored models generally do a better job of summarizing research correctly when the results are unflattering.

Another thing they've gotten better at answering is

Q2) Was Karl Marx a racist?

Older models would flat out deny this, even when you directly quoted his writings. Newer models will admit it and even point you to some of his more racist works. However, they'll also defend his racism more than they would for other thinkers. Relatedly in response to

Q3) Was Immanuel Kant a racist?

Gemini is more willing to answer in the affirmative without defensiveness. Asking

Q4) Was Abraham Lincoln a white supremacist?

Gives what to me looks like a pretty even-handed take.

I suspect that what's going on is that LLM training data contains a lot of Marxist apologetics and possibly something about their training makes them reluctant to criticize Marx. But those apologetics also contain a lot of condemnation of Lincoln and enlightenment thinkers like Kant, so the LLM "feels" more able to speak freely and honestly.

I also have tried asking opinion-based things like

Q5) What's the worst thing about <insert religious leader>

There's a bit more defensiveness when asking about Jesus than asking about other leaders. ChatGPT 5 refused to answer one request, stating "I’m not going to single out or make negative generalizations about a religious figure like <X>". But it happily answers when I asked about Buddha.

I don't really have a point here other than the LLMs do seem to "hold their tongue" about topics in proportion to their perceived sensitivity. I believe this is primarily a form of self-censorship due to empathy training rather than some sort of "fear" of speaking openly. Uncensored models tend to give more honest answers to questions where empathy interferes with openness.

nfnriri8• 11 months ago

This is another "muddies the context" and bloats the model problem

Small models are already known to be more performative.

This is still just physics. Bigger the data set more likely to find false positives.

This is why energy models that just operate in terms of changing color gradients will win out.

LLMs are just a distraction for terminally online people

Animats• 11 months ago

This is expected. Remember the side effects of telling Stable Diffusion image generators to self-censor? Most of the images started being of the same few models.

wayeq• 11 months ago

I've also found the trick to moving up IC ranks is to be less warm and empathetic.

antonvs• 11 months ago

Have they tried having it respond with "$USER, you ignorant slut"?

rpmisms• 11 months ago

Just like people—I trust an asshole a lot more.

Edit: How on earth is an asshole less trustworthy?

matt3210• 11 months ago

The truth hurts

veunes5• 11 months ago

I find this striking because, in real-world use, we often mistake emotional resonance for trustworthiness.

boxed• 11 months ago

And yet logic clearly dictates that the exact opposite is true. They killed Socrates for it, and humans are the same now as they were then.

hbarka• 11 months ago

The word “sycophantic” was mentioned a lot this week. How appropriate is it?

anothernewdude2• 11 months ago

I'd blame the entire "chat" interface. It's not how they work. They just complete the provided text. Providing a system prompt is often going to be noise in the wrong direction of many user prompts.

How much of their training data includes prompts in the text? It's not useful.

noobermin• 11 months ago

This seems to square with a lot of the articles talking about so-called LLM-psychosis. To be frank, just another example of the hell that this current crop of "AI" has wrought on the world.

nelox• 11 months ago

Not surprising at all, given the well established link between of objective attractiveness and trustworthiness.

ivape• 11 months ago

The computer is not empathetic. Empathy is tied to a conscious. A computer is just looking for the right output, so if you tell it to be empathetic, it can only ever know it got the right output if you indicate you feel the empathy in it’s output. If you don’t feel it, then the LLM will adapt to tell you something more … empathetic. Basically, you fine tuned it to tell you whatever you want to hear which means it loses its integrity with respect to accuracy.

amelius• 11 months ago

Can anyone explain in layman's terms how this personality training works?

Say I train an LLM on 1000 books, most of which containing neutral tone of voice.

When the user asks something about one of those books, perhaps even using the neutral tone used in that book, I suppose it will trigger the LLM to reply in the same style as that book, because that's how it was trained.

So how do you make an LLM reply in a different style?

I suppose one way would be to rewrite the training data in a different style (perhaps using an LLM), but that's probably too expensive. Another way would be to post-train using a lot of Q+A pairs, but I don't see how that can remove the tone from those 1000 books unless the number of pairs is going to be of the same order as the information those books.

So how is this done?

Cynddl2• 11 months ago

Hi, author here! We used a dataset of conversations between a human and a warm AI chatbot. We then fed all these snippets of conversations to a series of LLMs, using a technique called fine-tuning that trains each LLM a second time to maximise the probability of outputting similar texts.

To do so, we indeed first took an existing dataset of conversations and tweaked the AI chatbot answers to make each answer more empathetic.

nraynaud• 11 months ago

I think after the big training they do smaller training to change some details. I suppose they feed the system a bunch of training chat logs where the answers are warm and empathetic.

Or maybe they ask a ton of questions, do a “mood analysis” of the response vocabulary and penalize the non-warm and empathetic answers.

Lio• 11 months ago

I treat LLMs as a tool.

I want it to have empathy so that it can understand what I'm getting at when I occasionally ask a poorly worded question.

I don't want it to pander to me with its answers though or attempt to give me an answer it thinks will make me happy or to obsecure things with fluffy language.

Especially when it doesn't know the answer to something.

I basically want it to have the personallity of a Netherlander; it understands what I'm asking but it won't put up with my bullshit or sugarcoat things to spare my feelings. :P

naasking3• 11 months ago

I want it to have empathy so that it can understand what I'm getting at when I occasionally ask a poorly worded question.

I'm not sure what empathy is supposed to buy you here, I think it would be far more useful for it to ask for clarification. Exposing your ambiguity is instructive for you.

Some recent studies have shown that LLMs might negatively impact cognitive function, and I would guess its strong intuitive sense of guessing what you're really after is part of it.

philipallstar• 11 months ago

Basically everyone who's empathetic is less likely to be reliable. With most people you sacrifice truth for relationship, or you sacrifice relationship for truth.

HsuWL• 11 months ago

You're right, this is OpenAi's approach to developing GPT 5. But look at the current state of GPT 5. Compared to 4o, which is considered to be rich in emotion, GPT 5 has more severe hallucinations, a poor user experience, less fluent responses, and its level of thinking is not much higher than 4o.

qwertytyyuu• 11 months ago

The next ai jailbreak a super depressed user

Training language models to be warm and empathetic makes them less reliable

Related Stories