Mayo Clinic's secret weapon against AI hallucinations: Reverse RAG in action

https://venturebeat.com/ai/mayo-clinic-secret-weapon-against-ai-hallucinations-reverse-rag-in-action/

by ohjeez • 1 year ago

187 80 1 year ago

beebaween• 1 year ago

Curious if anyone has attempted this in an open source context? Would be incredibly interested to see an example in the wild that can point back to pages of a PDF etc!

pheeney• 1 year ago

I'd be curious too. It sounds like standard RAG, just in the opposite direction than usual. Summary > Facts > Vector DB > Facts + Source Documents to LLM which gets scored to confirm the facts. The source documents would need to be natural language though to work well with vector search right? Not sure how they would handle that part to ensure something like "Patient X was diagnosed with X in 2001" existed for the vector search to confirm it without using LLMs which could hallucinate at that step.

social_quotient4• 1 year ago

I think you’re spot on!

We’re using a similar trick in our system to keep sensitive info from leaking… specifically, to stop our system prompt from leaking. We take the LLM’s output and run it through a RAG search, similarity search it against our actual system prompt/embedding of it. If the similarity score spikes too high, we toss the response out.

It’s a twist on the reverse RAG idea from the article and maybe directionally what they are doing.

theodorewiles• 1 year ago

If I had to guess it sounds like they are using CURE to cluster the source documents, then map each generated fact back to the best-matching cluster, and finally test whether the best-matching cluster actually provides / supports the fact?

unstatusthequo• 1 year ago

Already exists in legal AI. Merlin.tech being one of those that provides citations to queries to validate the LLM output.

eightysixfour• 1 year ago

Plenty provide citations, I don’t think is exactly what Mayo is saying here. It looks like they also, after the generation, lookup the responses, extract the facts, and score how well they matched.

natnat• 1 year ago

Can someone link to a real source for this? Like, a paper or something? This seems very interesting and important and I'd prefer to look at something less sketchy than venturebeat.com

navigate8310• 1 year ago

What I find even more sketchy is Mayo’s medical director, reported VentureBeat about it than a non-sketchy outlet.

htrp• 1 year ago

at that point it becomes a search problem?

simonw• 1 year ago

Most of implementing RAG is a search problem, the R stands for "retrieval", which is the academic computer science term for "search".

hn_throwaway_99• 1 year ago

Can someone more versed in the field comment on whether this is just an ad or actually something unique or novel.

What they're describing as "reverse RAG" sounds a lot to me like "RAG with citations", which is a common technique. Am I misunderstanding?

binarymax• 1 year ago

The article is too high level to figure out exactly what they are doing.

HenryBemis• 1 year ago

where the model extracts relevant information, then links every data point back to its original source content.

I use ChatGPT. When I ask it something 'real/actual' (non-dev) I ask it to give me references in every prompt. So when I ask it to tell me about "the battle of XYZ" I ask it within the same prompt to give me websites/sources, that I click and check if the quote is actually from there (a quick Ctrl+F will bring up the name/date/etc.)

Since I've done this I get near-zero hallucinations. They did the same.

amelius2• 1 year ago

Well, the title said "secret" after all ...

1oooqooq• 1 year ago

in the publishing industry we call that "cooking a press release". the "news" article was entirely written and mailed by the PR of the subject (mayo clinic here) and the "journalist" just copy and paste. at most they will reword a couple paragraphs not for fear of looking bad, but just to make it fit in their number of words required for the column they are publishing under.

so, yes, an advertisement.

stuaxo• 1 year ago

It does sound like that.

I guess they have data they trust.

If that data ever gets polluted by AI slop then you have an issue.

aqme28• 1 year ago

Reverse RAG sounds like RAG with citations and then also verify the citations (e.g. go in reverse).

lmeyerov• 1 year ago

It sounds like they go further by doing output fact extraction & matching back to the RAG snippets. Presumably this is addition to matching back the citations. I've seen papers write about doing that with knowledge graphs, but at least for our workloads, it's easy to verify directly.

As a team who has done similar things for louie.ai - think real-time reporting, alerting, chatting, & BI on news, social media, threat intel, operational databases etc - I find it interesting less on breaking new ground but confirming the quality benefit when being more broadly used in serious contexts. Likewise, hospitals are quite political internally for this stuff, so seeing which use cases got the greenlight to go all the way through is also interesting.

m3kw9• 1 year ago

Can’t fool the patent inspectors if they don’t name it like that

rvnx• 1 year ago

There's probably a patent for: "Just double-checking before answering to the user".

nojito• 1 year ago

They leverage https://en.wikipedia.org/wiki/CURE_algorithm alongside many subsequent LLMs to do ranking and scoring.

barrenko• 1 year ago

A consultant sold them something with a high margin, they need to justify the bill.

jrpt• 1 year ago

"Mayo’s LLM split the summaries it generated into individual facts, then matched those back to source documents. A second LLM then scored how well the facts aligned with those sources, specifically if there was a causal relationship between the two."

It doesn't sound novel from the article. I built something similar over a year ago. Here's a related example from Langchain "How to get a RAG application to add citations" https://python.langchain.com/docs/how_to/qa_citations/

afro88• 1 year ago

I don't think you're getting it, it's not traditional RAG citations.

They are checking the _generated_ text by trying to find documents containing the facts, then rating how relevant (casually related) those facts are. This is different from looking up documents to generate an answer for a prompt. It's the reverse. Once the answer has been generated they essentially fact check it.

zxexz• 1 year ago

This is just standard practice AFAICT. I’ve done it. Everybody I know who’s built apps for unstructured document retrieval etc. is doing it. It works better than The naïve approach, but there are plenty of issues and tuning with this approach too.

jerpint• 1 year ago

It doesn’t solve the biggest problem with RAG, which is retrieving the correct sources in the first place.

It sounds like they just use a secondary LLM to check if everything that was generated can be grounded in the provided sources. It might help with hallucinations, but it won’t improve overall performance of proper retrieval

Palmik• 1 year ago

What I imagine

1. Use LLM, possibly already grounded by typical RAG results, to generate initial answer.

2. Extract factual claims / statements from (1). E.g. using some LLM.

3. Verify each fact from (2). E.g using separate RAG system where the prompt focuses on this single fact.

4. Rerun the system with the results from (3) and possibly (1).

If so, this (and variations) isn't really anything new. These kind of workflows have been utilized for years (time flies!).

unification_fan• 1 year ago

That's like trying to stop a hemorrhage with a band-aid

Daily reminder that traditional AI expert systems from the 60s have 0 problems with hallucinations by virtue of their own architecture

Why we aren't building LLMs on top of ProbLog is a complete mystery to me (jk; it's because 90% of the people who work in AI right now have never heard of it; because they got into the field through statistics instead of logic, and all they know is how to mash matrices together).

Clearly language by itself doesn't cut it, you need some way to enforce logical rigor and capabilities such as backtracking if you care about getting an explainable answer out of the black box. Like we were doing 60 years ago before we suddenly forgot in favor of throwing teraflops at matrices.

If Prolog is Qt or, hell, even ncurses; then LLMs are basically Electron. They get the job done, but they're horribly inefficient and they're clearly not the best tool for the task. But inexperienced developers think that LLMs are this amazing oracle that solves every problem in the world, and so they throw LLMs at anything that vaguely looks like a problem.

Bluestein• 1 year ago

This is tremendously cogent.-

jdaw0• 1 year ago

people stopped making these systems because they simply didn't work to solve the problem

there's a trillion dollars in it for you if you can prove me wrong and make one that does the job better than modern transformer-based language models

ben_w• 1 year ago

I think it's more that the old expert systems (AKA flow charts) did work, but required you to already be an expert to answer every decision point.

Modern LLMs solve the huge problem of turning natural language from non-experts into the kind of question an expert system can use… 95% of the time.

95% is fantastic if you're e.g. me with GCSE grade C in biology from 25 years ago, asking a medical question. If you're already a domain expert, it sucks.

I suspect that feeding the output of an LLM into an expert system is still useful, for much the same reason that feeding code from an LLM into a compiler is useful.

nickpsecurity• 1 year ago

That assumes it can even be done. It's worth looking into. There have been some projects in those areas.

Mixing probabilistic logic with deep learning:

https://arxiv.org/abs/1808.08485

https://github.com/ML-KULeuven/deepproblog

Combining decision trees with neural nets for interpretability:

https://arxiv.org/abs/2011.07553

https://arxiv.org/pdf/2106.02824v1

https://arxiv.org/pdf/1806.06988

https://www2.eecs.berkeley.edu/Pubs/TechRpts/2020/EECS-2020-...

It looks like model transfer from uninterpretable, pretrained models to interpretable models is the best strategy to keep using. That also justifies work like Ai2's OLMo model where all pretraining data is available to use other techniques, like those in search engines, to help explainable models connect facts back to source material.

amelius2• 1 year ago

Probably because translating natural language into logic form isn't very easy, and also the point where this approach breaks down.

WhitneyLand• 1 year ago

Does your brain really tell you it’s more likely that 90% of people in the field are ignorant, rather than old expert systems were brittle, couldn't learn from data, required extensive manual knowledge editing, and couldn't generalize?

Btw as far as throwing teraflops, the ability to scale with compute is a feature not a bug.

earnestinger• 1 year ago

It can be both. (Ignorant not as “idiots”, but as not experts and proponents of this particular niche)

hhh• 1 year ago

‘expert systems’ are logic machines

npiano• 1 year ago

This assumes that logic is derived from predicting the next step from previous information, which is not accurate.

xpe• 1 year ago

The answer to too much exaggeration about AI from various angles is _not_ more exaggeration. I get the frustration, but exaggerated ranting isn’t intellectually honest nor effective. The AI + software development ecosystem and demographics are broad enough that lots of people agree with many of your points. Sure, there are lots of people on a hype train. So help calm it down.

imoreno2• 1 year ago

Why we aren't building LLMs on top of ProbLog
they got into the field through statistics instead of logic

LLMs by definition are built from neural networks, which indeed work via "mashing matrices" rather than "logic". That's the axiom of the technique. Sounds like you're saying let's throw away half a century of progress and start from scratch with a completely different direction. Maybe it will work, but who's gonna do all of that? I doubt that vague heckling from random comment threads will succeed in convincing researcher to commit to multiple lifetimes of work.

Instead of trying to reinvent LLMs, it would be more practical to focus on preprocessing input (eg. RAG) and postprocessing output (eg. detecting hallucination and telling the model to improve it before returning results to the user). This is something where something like using ProbLog might conceivably produce an advantage. So if you really want to rehabilitate Prolog to the field, why don't you go ahead and develop some LLM program in it and everyone can see for themselves how much better it is?

isoprophlex• 1 year ago

If only we could understand the actual mechanism involved in "reverse RAG"... was anyone able to find anything on this beyond the fuzzy details in tfa?

tekacs• 1 year ago

This is very interesting, but it's so perfect that the Mayo Clinic gets to use an algorithm called CURE, of all things.

shermantanktop• 1 year ago

When they describe CURE, it sounds like vanilla clustering using k-means.

mdp2021• 1 year ago

The article is a week old, it was already submitted a few days ago, and the problem remains about finding some paper to shed more light into the practice.

A blog article* came out yesterday - but it is not immediately clear whether the author wrote what he understood, or whether he knows more.

But much perplexity remains: «summarization, which is what LLMs generally excel at» (original article); «The LLM ... reads the patient’s records ... and produces a summary or list of facts» (blog). This is possibly the beginning, and some of us will already be scared - as the summarization capabilities we experienced from LLMs were neither intelligent nor reliable. (...Or did new studies come up and determine that LLMs have become finally reliable, if not cognitively proficient, in summarization?)

* https://usmanshaheen.wordpress.com/2025/03/14/reverse-rag-re...

rscho• 1 year ago

Doctors and statistical processes are always a bad match. (I'm a MD doing medical stats) No, this is not reliable in the slightest.

ttpphd• 1 year ago

If LLMs were good at summarization, this wouldn't be necessary. Turns out a stochastic model of language is not a summary in the way humans think of summaries. Thus all this extra faff.

sroussey• 1 year ago

What are the good models for summarization? I have found all, particularly local models, to be poor. Is there a leaderboard for summarization somewhere?

rafaelmn• 1 year ago

How do you evaluate quality ? Also I suspect the performance between models would varry between datasets. Heck it would vary on same model/source if you included that your mother was being held hostage and will be killed unless you summarize the source correctly :).

I think you are still stuck with try if it works for you and hope it generalizes beyond your evaluation.

mmooss2• 1 year ago

A second LLM then scored how well the facts aligned with those sources, specifically if there was a causal relationship between the two.

What is 'causal' about it? Maybe I'm reading one word too closely, but an accurate citation or summary isn't a matter of cause and effect?

calciphus• 1 year ago

I read this as "what caused this statement to appear in the summary". Was it the source materials or the LLM base model / hallucination?

mmooss2• 1 year ago

I see what you mean. I guess even LLM errors come from the source materials, just along a path we don't like.

wetherbeei• 1 year ago

"The check grounding API returns an overall support score of 0 to 1, which indicates how much the answer candidate agrees with the given facts. The response also includes citations to the facts supporting each claim in the answer candidate.

Perfect grounding requires that every claim in the answer candidate must be supported by one or more of the given facts. In other words, the claim is wholly entailed by the facts. If the claim is only partially entailed, it is not considered grounded."

There's an example input and grounded output scores that shows how the model splits into claims, decides if the claim needs grounding, and the resulting entailment score for that claim in: https://cloud.google.com/generative-ai-app-builder/docs/chec...

numba888• 1 year ago

That would require keeping a library of 'given facts'. Which would include reproducible copyrighted materials or their derivatives.

gloosx• 1 year ago

Hmm, 90 minutes of bureaucracy for practitioners every day, data can be extremely difficult to find and parse out, how can I augment the abilities and simplify the work of the physician?

Let's smack a hallucinating LLM and try to figure out how to make it hallucinate less... Genius

Mayo Clinic's secret weapon against AI hallucinations: Reverse RAG in action

Related Stories