Pruning RAG context down to what the answer actually needs

https://www.kapa.ai/blog/how-we-prune-rag-context

by emil_sorensen • 16 days ago

146 37 16 days ago

rooftopzen• 16 days ago

Cliche topic - from a few years ago (the "RAG is dead" vs "All You Need Is Advanced RAG" BS - it came in waves and cycles, spread by bots on social media networks).

"Pruning RAG Context" is trying to recycle the old stuff (again), presuming the reader is naive (implies kapa.ai is not going anywhere). The current cycles were "openclaw" (I think that died), now we are on "harnesses" - when that dies the paid social media bots will give you something else. Shell game.

Just declare / define dictionary as a variable in your prompt to carry forward (when you decide to continue using LLMs for certain things). Also either summarize or truncate history. 3-4 year old concept. Not a big thing.

timcobb• 16 days ago

Shallow dismissal

wolvoleo2• 16 days ago

"RAG is dead" was declared because of ever expanding context windows.

However, resources consumed expand so much with context that I think there is a huge practical barrier there. In the beginning AI services were basically free so nobody cared but this is rapidly shrinking.

Personally I see more in a combination of RAG and live querying during the thinking process (e.g. by tools).

Also I don't think dumping any context that might be relevant into the model really helps accuracy. In my experience models just get lost when they get an overload of irrelevant stuff in their context and start overlooking the relevant parts even if it does fit the window.

agentdev001• 16 days ago

Am I wrong to be somewhat peeved by the use of "RAG" in these contexts? I always read things like this, and wonder if instead the author should be saying "Semantic Retrieval" or something something Vector, etc. Retrieval augmented generation captures tool-use, and; semantic search of course is really just a tool under the hood.

To make an anology, in my mind, this is akin to saying "fuel air mixture system" when referring to direct fuel injection specifically, when of course, a carburetor also lives in that category.

tingletech2• 16 days ago

I read “RAG Context” as “the retrieved content injected into the context window”

0x696C69612• 16 days ago

So when an agent does "cat file.txt" that's RAG to you?

red_hare• 16 days ago

I agree. We're seeing more variants of "RAG" that aren't semantic at all (e.g. coding agents or simple memory systems that feed summary indexes directly into context).

I think, over time, it's going to become a SQL / NoSQL sort of divide. There will be the right kind of RAG for the job and lots of forcing the wrong kind because the developer doesn't understand the nuances.

teaearlgraycold• 16 days ago

RAG, as far as I understand, is a term that came about before LLM tool-calling was as prevalent. Your options were to have an LLM hallucinate up a response, or instead do a [document -> chunk -> embedding -> vector db -> query -> context window] pipeline. I haven't heard anyone talk of LLMs + web search or other tool calls as RAG, even though if you pull apart the semantics the term is applicable. In fact I don't hear people talk about RAG much at all. I suppose much of what people were trying to solve with document chunking/embedding pipelines has been solved with bigger models and tool calls. And along with that change in tooling we have left behind the term "RAG", which leaves it attached to the concept of those pipelines.

Folcon• 16 days ago

I do wonder if the term will make a comeback, Retrieval Augmented Generation as a concept is a fairly fundamental idea, or maybe it's considered too generic which is why tool calling is used in favour if it?

The problem with tool calling is it's too generic in my mind, maybe RAG will make a return when we get around to having different flavours of it, digging into the rich vein that is Information Retrieval

Also is it just me that doesn't like this sort of wording?

agent has to hold

I find it very generic, I'd much prefer process or recall or any term that indicates what the agent is doing with tokens in that context

jillesvangurp2• 16 days ago

RAG is a fancy acronym that basically boils down to: let's give ai agents the super power of information retrieval (aka. search) and "augment" the generation with a list of results by adding that to the context.

The narrow interpretation of this is usually some kind of vector search. Which some people naively treat as magic pixie dust that will make search quality amazing without any tuning whatsoever.

This does not actually work all that well beyond really simple use cases. A well tuned traditional search engine can be surprisingly competitive. And I know people that do pretty complicated things with vector search that usually involve training their own models and spending a lot of effort on testing and validating those are any good.

I've been doing stuff with search for a bit over two decades. AI use cases makes information retrieval more relevant than ever. It's a key ingredient to answering questions for complex, proprietary data. And especially when that data is very complex and unstructured, naive approaches tend to have their limitations. In other words, it can pay off to to sit down and do it properly and think about things like data ingestion pipelines, transforming & enriching data, testing search quality, etc. Most of the success of a good search system usually boils down to getting your data right for indexing and optimizing it for how you are going to query your data.

The good news is that with large context windows, precision (best results are at the top) matters a bit less than recall (the search returns what you need when you search for it) these days. You can compensate for imprecise search by just fetching more results. As long as what you needed appears somewhere in the top 500 or so, you'll be fine. The flip side is of course that you end up adding a lot of noise to your context which might throw the LLM off and in general wastes a lot of tokens. That's why precision is still important.

What the article is proposing is post processing imprecise results to filter out the noise with an LLM to compensate for what is basically not a great search implementation. That can work of course (provided your recall doesn't suck). But it's going to add some cost and latency to searches. And usually, agents do multiple searches.

But if your search is so poor, why bother with vector search at all? Especially dense vector search at scale is not cheap. If you are going to fetch lots of results, just use some cheap lexical searches. Sparse vector search might be a good compromise (higher cost to index but similar performance to lexical search).

apwheele• 16 days ago

Processing 500 retrieved chunks here (probably with an additional LLM) will also add a bit of latency (not shown in these graphs!).

So for user facing apps, that scenario is probably not feasible (more like filter 10 chunks). Which as the parent of this comment suggests is fine to add in extra context given the current size of context windows.

dd8601fn• 15 days ago

As a real dummy on the subject, maybe you could help me understand where vector search tends to fall over?

I use it to retrieve tool functions by description and it has worked very well for me, but I expect I'm in the "very simple use cases" category that you mentioned.

petesergeant3• 16 days ago

Am I wrong to be somewhat peeved by the use of "RAG" in these contexts?

Probably? The ship has sailed on this one, like it has on "AI" or "crypto". RAG has almost exclusively been used to describe some variation on vector-search + bm25 pushed into the prompt for the last few years.

seanhunter• 16 days ago

Yes you are wrong. RAG means retrieval-augmented generation. If you’re generating something and that generation is augmented by some retrieval you’re doing RAG. The retrieval doesn’t need to be from a vector db or even based on semantic similarity.

weitendorf2• 16 days ago

To be honest as someone working in this space for the past two years, the problem with the “RAG” and semantic search community is it’s mostly vendors and solutions people selling simple, general stuff to product teams.

If you really are into search you probably implement something bespoke for your use case and integrate it into a product directly, and engage with models/infra tools directly rather than through the products in the space.

If you understand how “semantic retrieval” and other search tools are implemented in practice they feel almost embarrassingly primitive to give such fancy names, or pay for through tools that just implement really basic post-filtering. The entire space had the rug pulled out from under it once “agentic search” took off and most major LLM vendors started integrating web search and tool calling into their products. There is still a lot more interesting stuff you could do with customized rerankers/embedding models, and search algorithms, or small models specialized for agentic search/retrieval, etc but the userbase is big companies that realistically don’t need anything more than a list of tech support document titles that a cheap LLM can select from. So “RAG” is basically a sales shibboleth for that type of stuff now.

esafak2• 16 days ago

tl,dr: They used a rubric to have the LLM grade the chunks on a Likert scale. I think this is a good way to coax numbers out of an LLM.

siquick• 16 days ago

Likert scale just doesn't work in LLM evals. My idea of 3/5 is different from your 3 and definitely different from an non-deterministic system's 3.

esafak2• 16 days ago

That's what the rubric is for; it reduces the problem to NLP, which is the LLM's forte. The more objective you can make your rubric, the better.

petesergeant3• 16 days ago

My dissertation used a Likert scale with ChatGPT-3 era LLMs, and it was both internally consistent over 5-6 runs on a given statement, and consistent with human raters. I don't think you can bat it away as simply "doesn't work"

wolvoleo2• 16 days ago

I think this is really where energy saving comes into play. Context is so incredibly energy and processing time sensitive.

chimpanzee2• 16 days ago

i wouldn't trust the small llm there.

it will be an intellectual bottleneck when it comes to processing the very information your just arduously fished out of the ocean!!

kordlessagain• 15 days ago

I've had good success using a local model for preprocessing (tag extraction) and tuning the search parameters (hybrid search).

grewil2• 16 days ago

"Three knobs matter:"

I can't help reading articles with the radar on for signs of AI-generation nowadays. I have noticed that Claude sometimes uses the word knob for parameter, so here I get suspicious.

petesergeant3• 16 days ago

The contribution here is the technique (which looks like a nice alternative to reranking) rather than the article or it being some kind of think piece, so does it really matter?

I was able to skim it and go from "lol they reinvented re-ranking" to "oh, that's something more interesting, I should try this one day", and the writing didn't get in the way.

daveKoala• 16 days ago

I am being constantly mocked for using there term 'knob fiddling', but I am a British child of the 70's :-)

Avery29• 16 days ago

Bad or only loosely related context can make the final answer worse than having less context.

alansaber• 16 days ago

Would be cool to see a retrieval comparison to IE a claude code agent trace for the same query, even a cherry picked one.

Imanari• 15 days ago

Did you experiment with the Pruner completely replacing the Reranker?

timedude• 15 days ago

Yep seems like the pruner and reranker could be combined. That is what i built.

lbaltensp2• 15 days ago

Yes, the pruner operates on the top 15 chunks after reranking. reply

lbaltensp2• 15 days ago

Replacing the reranker with the pruner completely was not feasable since it would have to run on the 150-200 chunks we retrieve before reranking.

Pruning RAG context down to what the answer actually needs

Related Stories