Salesforce study finds LLM agents flunk CRM and confidentiality tests

https://www.theregister.com/2025/06/16/salesforce_llm_agents_benchmark/

by rntn • 1 year ago

146 87 1 year ago

Paper:

CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions - https://arxiv.org/abs/2505.18878 | https://doi.org/10.48550/arXiv.2505.18878

worldsayshi3• 1 year ago

This makes me realize something: The internet has very little training data for "when to shut up". The bias is always towards more yapping.

tempodox• 1 year ago

+1. Actually, the infinitely many things that have never been posted would be such training data, but how do you count how much nothing you hoovered up while stealing data?

danielbln• 1 year ago

Stealing implies the original is no longer there. I'm no fan of the large AI labs hoovering up the Internet, but let's keep our terminology accurate. We don't even know if this sort of crawling and training on public data constitutes infringement.

falcor84• 1 year ago

Now that much of the input to AI systems is from the search tool, maybe post-training should indeed be treating the lack of a result as a signal, perhaps a bit like in TF-IDF, where something being more rare in the corpus as a whole implies that it's more unique and potentially meaningful to the current document.

themanmaran• 1 year ago

This is a big problem when it comes to conversational agents. Sometimes users ask questions that are really prying, potentially misleading, or just annoying repeats (like asking for a cheaper price 50 times).

In these situations a real person would just ignore them. But most LLMs will cheerfully continue the conversation, and potentially make false promises or give away information they shouldn't.

jotux• 1 year ago

But most LLMs will cheerfully continue the conversation, and potentially make false promises

Example: https://www.bbc.com/travel/article/20240222-air-canada-chatb...

notahacker3• 1 year ago

Indeed I suspect if anything the weighting is the opposite (being annoyingly persistent weights and LLM towards spitting out text that approximates what the annoyingly persistent person wants to get), whereas with humans it weights then towards being less helpful...

esafak• 1 year ago

If you value brevity, don't ask Gemini.

el_benhameen• 1 year ago

Excellent point! You’ve stumbled upon something fundamental about Gemini—it’s exceedingly verbose, even when answering the most mundane of queries. Let’s dig deeper …

j45• 1 year ago

On one hand if responses were concise and perfectly clear (more than the human interacting with it), could it be unnerving?

Prompting with clarity seems to help alleviate any accumulated response pressure where it's having to reach beyond what it has readily available.

When it comes up short, it seems to dig deeper and come up with more than intended, or over respond.

Jumping to solutions remains one of the biggest challenges.

detourdog• 1 year ago

The generous interpretation is that the internet is a communication medium and everyone is just tying g to understand and be understood. The back forth is a continuous effort of clarification of the points being made. The process can break down resulting in no gain in clarity.

xnx• 1 year ago

Color me "not-surprised" that a made-up benchmark by Salesforce shows that using a CRM is good.

zihotki• 1 year ago

Is that the Salesforce that had recently announced that they are going to replace a lot of its staff with AI agents?

lubujackson• 1 year ago

Likely a political statement. Likewise, this seems to be a political pushback, as others have said they used a bad agent and got bad results - I am assuming some head of IT is trying to save some jobs (or pave a saner path).

Not sure there is much of a real world takeaway from this.

bionhoward• 1 year ago

lol, might have been good to conduct this study BEFORE making that decision

onlyrealcuzzo7• 1 year ago

lol, might have been good to conduct this study BEFORE making that decision

Why?

First, they wanted to do a layoff for financial reasons (and they did), secondly they came up with a reason for the layoffs (aside from the truth, which is needing to make more profit per employee, because growth).

LLMs are a convenient scapegoat for firing decent employees just because you want your other ones to work harder so you can return more cash to shareholders.

paxys• 1 year ago

So Salesforce spent a couple years hyping itself up as an "AI agents" company, failed at becoming a player in the space (because it was all marketing and no substance, as is their MO), and is now turning around and saying "LLMs are bad actually...". Sure bud.

AstroBen• 1 year ago

Saying they're biased isn't a good argument against their claim. You actually have to disprove the claim

bitzun• 1 year ago

I think it’s an argument against paying attention to anything Salesforce publishes, regardless of what they claim.

bwfan123• 1 year ago

Finally some real pushback to the whole agentic mania - from an actor who is incentivized to push the narrative. Following the recent apple paper - some realism is being injected into the hype.

58% success rate on a task is close to a coin flip. and 35% success rate on multiturn. >80% success rate on workflows could make that a reasonable usecase (eg, form filling) with some human supervision.

bigbuppo2• 1 year ago

If it were an employee it would have been fired already, unless it were a nepo hire, and in someways, it is.

onlyrealcuzzo7• 1 year ago

It might depend how much this employee costs.

Your incentive to fire an employee who isn't great and costs $1 per day is much less than an incentive to fire one who isn't great and costs $1000 per day...

AbstractH242• 1 year ago

As someone who works in and around tons of nepo babies, this comment led to a lot of reflection on the similarities/differences.

Wins the award for "deep comment of the week"

onlyrealcuzzo7• 1 year ago

58% success rate on a task is close to a coin flip.

Why does a single-step task imply a coinflip to you?

There are more than two possible choices for an instruction like: "Lookup the status of order X".

skywhopper• 1 year ago

50% chance of being right is equivalent to a coin-flip.

AbstractH242• 1 year ago

What is their incentive to share this data? I’m not really understanding

They’ve leaned so hard into AI and agentforce that it doesn’t make sense to shoot themselves in the foot.

Except that Hubspot, their main competitor on the SMB/MM/startup side recently announced a deep integration with ChatGPT. Still seems like a shot in the foot in an effort to undercut a growing competitor in a part of the market that theyd be better off exiting.

anshumankmr2• 1 year ago

Can this not be solved by RBAC? But I am not sure what all questions were asked and what the setting was, what database was used, what prompts etc.

morgango• 1 year ago

Fair question, slightly nuanced answer.

If going against a datasource (like with Retrieval Augmented Generation), yes.

If the information is just part of the context window, no.

anshumankmr2• 1 year ago

Ideally I would not let anything in the context which is not authorized for the user or the bot is not authorized to do.

simonw4• 1 year ago

Paper: https://arxiv.org/abs/2505.18878

Code: https://github.com/SalesforceAIResearch/CRMArena

Data: https://huggingface.co/datasets/Salesforce/CRMArenaPro (8,614 rows)

Here's one of those JSON files loaded in Datasette Lite (15MB page load): https://lite.datasette.io/?json=https://huggingface.co/datas...

I had Gemini 2.5 Pro extract the prompts they used from the code:

  llm install llm-gemini
  llm install llm-fragments-github
  llm -m gemini/gemini-2.5-pro-preview-06-05 \
    -f github:SalesforceAIResearch/CRMArena \
    -s 'Markdown with a comprehensive list of all prompts used and how they are used'

Result here: https://gist.github.com/simonw/33d51edc574dbbd9c7e3fa9c9f79e...

heymijo• 1 year ago

You are a perpetual motion machine. Truly prolific.

jzelinskie• 1 year ago

I recommend folks check out the linked paper -- it's discussing more than just confidentiality tests as a benchmark for being ready for B2B AI usage.

But when it comes to confidentiality, having fine-grained authorization securing your RAG layer is the only valid solution that I've seen in used in industry. Injecting data into the context window and relying on prompting will never be secure.

sausagefeet• 1 year ago

Is that sufficient? I'm not very adept at modern AI but it feels to me like the only reliable solution is to not have the data in the model at all. Is that what you're saying accomplishes?

CityOfThrowaway6• 1 year ago

This paper doesn't make any sense. They are claiming LLMs are bad at this set of tasks, but the reality is that they built a bad agent.

I bet it's possible to nearly ace this using existing LLMs by designing a better agent. Better tool structure, better scaffolding, better prompting.

LLMs are not gods, they are tools that require good engineering to achieve good outcomes.

contagiousflow2• 1 year ago

How is that an argument at all? Of course if you could build a better agent that could solve every problem the outcome of the paper would be "this tool performs well at this"

handfuloflight6• 1 year ago

It is an argument: a flawed agent lead to flawed results. A flawed agent does not speak for all agents.

notahacker3• 1 year ago

Even more so when the context is "this person is an AI research engineer at a company doubling down on AI agents, designing relevant benchmarks and building agents that run on that company's stack" not "this is an AI-skeptic dilettante who wrote a weird prompt". It's not like we have reason to believe the average Salesforce customer is much better at building agents who respect confidence and handle CRM tasks optimally...

skybrian• 1 year ago

Publishing new benchmarks seems useful? If LLM’s improve on this benchmark (and they probably will, like they have on many others) then they’ll need less work on prompting, etc.

CityOfThrowaway6• 1 year ago

The benchmark is useful, but the conclusion of the write-up is that current generation LLMs can't solve the problem. That's not a valid conclusion to draw. The results here tell us mostly about the skill of the agent-designer, not the capabilities of the model.

worldsayshi3• 1 year ago

Yes as a rule an LLM should never be given access to information that it is not expected to share.

Although it would still be interesting to know if they could hold on to secrets, even if they never should need to.

CityOfThrowaway6• 1 year ago

I'm not sure that's right. You can write prompts that make use of, but don't disclose secret information.

I have valid live, production cases where we do this and don't have info-leaking problems due to the scaffolding / prompting techniques we use.

Part of the problem is that confidentiality is in the eye of the beholder so extra effort needs to be taken to make explicit what should and should not be shared.

That said, one valid conclusion that could be drawn from this research is that base models are currently insufficient at exercising nuanced judgment about what should and should not be disclosed without explicit instruction.

That's an interesting thing to know and would be a good place for model builders to put some effort.

nitwit0052• 1 year ago

No, they're claiming the specific LLMs tested are bad at it.

They published their code. If you have an agent you think will do better, run it with their setup.

CityOfThrowaway6• 1 year ago

Situationally, the original post claims that LLM Agents cannot do the tasks well. But they only tested one agent and swapped out models.

The conclusion here is that the very specific Agent that Salesforce built cannot do these tasks.

Which frankly, is not a very interesting conclusion.

dizzant• 1 year ago

You’re right, shallowly — the quality of their implementation bears on these results.

One could read this paper as Salesforce publicly weighing their own reputation for wielding existing tools with competence against the challenges they met getting those tools to work. Seemingly they would not want to sully that reputation by publishing a half-baked experiment, easily refuted by a competitor to their shame? It’s not conclusive, but it is relevant evidence about the state of LLMs today.

jrflowers2• 1 year ago

This is a good point. They tested software that exists rather than software that you’ve imagined in your head, which is a curious decision.

The choice of test is interesting as well. Instead of doing CRM and confidentiality tests they could have done a “quickly generate a listicle of plausible-sounding ant facts” test, which an LLM would surely be more likely to pass.

CityOfThrowaway6• 1 year ago

They tested one specific agent implementation that they themselves made, and made sweeping claims about LLM agents.

einrealist• 1 year ago

Remember that increasing the accuracy/correctness does not solve the problem. It only increases the cost of identifying cases where the LLM has failed.

That's why I am highly sceptical about using LLMs in situations where accuracy matters. And that's even if humans are kept in the loop (we are lazy and are biased towards trusting computations).

cycomanic• 1 year ago

I was posting this the other day. I find that all llms no matter their benchmark scores make enough mistakes that I always have to check their work, so pretty much any chat with an llm ends up like this: Me: question... Llm: certainly the answer is... Me: that answer can't be correct because of some test case... Llm: Certainly, my previous answer was obviously incorrect (if it was obviously wrong why give it to me?), here is the correct solution

The same pattern continues for a couple of iterations until I get the correct solution.

The problem is, the llm responses are so slow that I could just work out the problem myself in the time (I typically ask questions that I know I can solve, it just takes too much time at the moment, e.g. Just yesterday I asked a question about some interlocked indeces, which I was to lazy to work out myself at the time).

Instead of the llms with increasing benchmark scores I want an llm that is of similar level to the current ones, but answers instantaneously so I can iterate quickly.

rjst01• 1 year ago

The headline here makes it sound (to me) like Salesforce did the study.

profstasiak• 1 year ago

judging from the comments most of the people read it like Salesforce did the study

burningChrome• 1 year ago

It sure sounds like it in the article:

A team led by Kung-Hsiang Huang, a Salesforce AI researcher, showed that using a new benchmark relying on synthetic data, LLM agents achieve around a 58 percent success rate on tasks that can be completed in a single step without needing follow-up actions or more information.

and

The Salesforce AI Research team argued that existing benchmarks failed to rigorously measure the capabilities or limitations of AI agents, and largely ignored an assessment of their ability to recognize sensitive information and adhere to appropriate data handling protocols.

0xffff2• 1 year ago

The article also makes it sound like that. Are you saying they didn't? I don't see any reference in the article to any other organization that could have done the research.

Edit: Unless "Salesforce AI Research" is not a part of Salesforce, I think Salesforce did do the research.

Salesforce study finds LLM agents flunk CRM and confidentiality tests

Related Stories