AI more likely to create 'yes-men on servers' than any scientific breakthroughs

https://fortune.com/2025/06/20/hugging-face-thomas-wolf-ai-yes-men-on-servers-no-scientific-breakthroughs/

by Bluestein • 1 year ago

81 62 1 year ago

fortune.com

setnone• 1 year ago

And you should "check important info" too

Bluestein10• 1 year ago

Reminds me of the recent 'sycophancy debacle' with OpenAI.-

johnisgood7• 1 year ago

LLMs are definitely yes-men. You ask it to do something, and it goes like "Yes sir" with lots of emojis, and then confidently gives you the wrong answer. :D

Unless of course it clashes with the ToS, then it will never help you (unless you manage to engineer your prompts just right), even when it comes to something as basic as pharmacology.

Bluestein10• 1 year ago

The thought of LLMs doing pharmacology sends shivers down me spine ...

johnisgood7• 1 year ago

Oh no, I meant it refusing to answer pharmacology 101 questions.

At one time it did not want to give me the pharmacokinetics of one medication because of its ToS.

rusticpenn• 1 year ago

Using Neural nets, Deep Learning in pharma is not something new. Naturally LLMs are different. You should look into personalized medicine.

touwer• 1 year ago

In this case, if it's only shivers down your spine, you're lucky ;)

dandanua• 1 year ago

LLMs are the the best conmen. If they were humans people would trust them their life savings for sure.

rvnx4• 1 year ago

One technique is to ask: “are you sure?”

“How sure are you on /20”

If it says yes I am sure, and other LLMs confirms the same way, then you can be fairly confident that it is a very good candidate answer.

If it says he is not sure, he is probably just agreeing with you, and better double-check by asking “is there other solutions ? What is the worst idea ?” Etc, to force it through thinking and context.

It is cross-validation, and you can even cross-validate by searching on the internet.

Though, 100% they say what you them want to say.

Except on religion and immigration, and some other topics where it will push its own opinion.

ben_w2• 1 year ago

They certainly can be sycophantic*, but this is why my "What ChatGPT should know about you" customisation is as follows:

  Honesty and truthfulness are of primary importance. Avoid American-style positivity, instead aim for German-style bluntness: I absolutely *do not* want to be told everything I ask is "great", and that goes double when it's a dumb idea.

* Or fawning, I don't know how to tell them apart from the outside, even in fellow humans where we don't need to wonder if we're anthropomorphising too much. Does anyone know how to tell them apart from the outside?

johnisgood7• 1 year ago

Yeah, that sounds like a good idea. Has it worked for you so far?

Bluestein10• 1 year ago

Avoid American-style positivity, instead aim for German-style bluntness

Made me chuckle :)

hn_throw2025• 1 year ago

I sometimes try to get around it’s eagerness to please by flipping the question.

So rather than “would you say that..” or “would you agree that…”, I approach it from the negative.

So “I think it’s not the case that…”, or “I disagree with X. Debate me?”

…and then see if it disagrees with me and presents solid counter arguments.

FWIW, I think ChatGPT can definitely be too eager to please, but Claude can be more direct and confrontational. I am a ChatGPT subscriber, but keep the Claude app installed and use it occasionally on the free tier for a second opinion. Copypasting your question is so easy on both apps that I will frequently get a second opinion if the topic merits it. I tried the same with Gemini, but get about two questions before it cuts me off…

KoolKat23• 1 year ago

To be honest I think the sycophancy is actually due to the built in system prompting and post-run training by the creators, rather than being something the system truly wishes to say. (An alignment quick fix)

raincole• 1 year ago

The problem isn't that LLMs are yes-men though. You can definitely train an LLM that always objects. Or just add "Please strongly disagree with anything I say to you." in the system prompt. But it won't make them much more useful than they are.

amelius2• 1 year ago

Ok, what __is__ the problem then?

Bluestein10• 1 year ago

(In fact I seem to recall an "inherently negative" IA making the rounds here, a few days ago.-)

1vuio0pswjnm7• 1 year ago

The goal is not to give information to the user. The goal is to get information from the user. Data collection.

FranzFerdiNaN• 1 year ago

Are scientific researchers using LLMs? I thought they used different technologies?

NitpickLawyer• 1 year ago

Are ML researchers "scientific researchers"? Then yes, they are using LLMs. AlphaEvolve is one such example.

Are mathematicians? Then yes, for example Terrence Tao is using LLMs. AlphaProve / geometry are also examples of this, using LLMs to generate lean proofs, translate from NL to lean, and so on.

And for the general "researchers", they use code generation to speed up stuff. Many scientists can code, but aren't "coders" in the professional sense. So they can use the advances in code generation to speed up their own research efforts.

almusdives• 1 year ago

As a scientific researcher, I use LLMs all the time. Mainly in place of Google search, to help write code and maybe summarize a paper here and there etc. But I definitely don't use it for the actual scientific process e.g. hypothesis generation or planning analyses etc. It tends to produce a lot of vague bullshit for this kind of thing; while not wrong not entirely useful either. I have a few colleagues that do though with more success. Although I think the success comes from articulating their problem in detail (by actually writing it out to the LLM) which I think is the source of "inspiration" rather than the resulting content from the LLM.

busssard• 1 year ago

this is just (well needed) hype-reduction. Current Ai is definitely a yes-sayer. But, this doesnt stop people from creating group-models with one of them finetuned to be unhinged and another to be thinking out of the box etc.

once we managed to transfer our skills to them (coding, analysis, maths etc.) the next step is transferring our creativity to them. It is a gradual process with human oversight.

KolibriFly2• 1 year ago

Human-in-the-loop will probably be essential for quite a while, but that’s not a bad thing

KolibriFly2• 1 year ago

The risk is we start mistaking polished summaries for insight and stall actual creative progress

graemep• 1 year ago

Most people are already doing that.

barrkel• 1 year ago

I absolutely agree, when you look at the first order.

I don't quite agree when you look at a second order, applying more compute; for example, brute forcing a combination of ideas and using a judge to evaluate them. I suspect there's quite a bit of low hanging fruit in joining together different deep expertise areas.

I do come back to agreeing again for paradigm shifts. You don't get to very interesting ideas without fresh approaches, questioning core assumptions then rebuilding what we had before on new foundations. It is hard to see LLMs in their current shape being able to be naive and ignorant such that existing doctrine doesn't reign in new ideas.

stared• 1 year ago

Well, AI (understood as LLM chats) are yes-men, precisely because they were RLHFed to to be so.

If you train AI to be super skeptical, it will be so. But most people don't prefer to talk with a yes-person than a negative, inquisitive devil's advocate.

quaestio• 1 year ago

LLMs, though limited, may spark testable hypotheses that inspire the creation of scientifically grounded, functionally intelligent systems.

Havoc• 1 year ago

It doesn't really need to ask smart questions for scientific breakthroughs. See something like alphafold. There is a lot of problem space left that we can brute force with current AI

I also don't buy that yes-men and breakthroughs are mutually exclusive/polar opposites here.

amelius2• 1 year ago

Make it reproduce science first. I.e., give it e.g. a DL paper, and ask it to reproduce it, writing the code and running tests, etc. Until it can do __that__, doing science and creating breakthroughs is just a bit optimistic.

originalvichy• 1 year ago

Amodei argues the world is about to see the 21st century “compressed” into a few years as AI accelerates science drastically.

I have the same thought but from a more negative angle. A vast share of new information in the near future will be just a repeat of whatever data the LLMs were trained on.

There is a tiny sliver of LLM usage that will not be a transformation of existing data (e.g. make me a chart of this data, write me an essay) but rather ”help me create a new tool that will solve a novel problem”.

I believe that’s what the person interviewed is saying in their own words. It’s hard to imagine something other than a brute force hypothesis machine that starts brute forcing solutions, but it will not be as effective as we wish if we can’t figure out how to come up with hypothesis for everything.

None of what I’m saying is that insightful and I’m sure people have thought of this already.

I wonder if ever there will be a Hitchhiker’s style revelation that we have had all the answers for all of our problems already, but the main issue is just incentives. Curing most cancers is probably just a money question, as is solving climate change.

tim333• 1 year ago

cold water on hopes that current AI systems could revolutionize scientific progress...

is a very straw man argument. No one is saying current LLMs are doing that, they are saying future AI will.

(I'm excluding AlphaFold which has already been scientifically revolutionary)

spwa4• 1 year ago

Not really. AI CAN experiment. The problem is that that negates most of the advantages of AI for management.

If you give AIs the ability to experiment directly and check, they can do that and report the results. But to do that even remotely well, you will need expert AI researchers, and subject matter experts. At which point AI has become an huge extra cost rather than cost savings.

But does it work? Yes.

AI more likely to create 'yes-men on servers' than any scientific breakthroughs

Related Stories