lbaltensp

Replacing the reranker with the pruner completely was not feasable since it would have to run on the 150-200 chunks we retrieve before reranking.

Yes, the pruner operates on the top 15 chunks after reranking. reply

"relevance-to-the-query is exactly the signal that misses the buried caveat."

You are exactly hitting the nail on the head. The underlaying problem of a pruner is to determine document relevance to a user query. And document relevance comes in different flavours:

1. Direct relevance: a document can be relevant to the user query because it directly answers it. 2. Subquery-Conditional Relevance: a document can be relevant because it answers a decomposed subquery of the user query. 3. Document-Interdependent Relevance: a document is relevant only because another document provides bridge context, domain knowledge, disambiguation, a definition or a constraint.

This means the right question is therefore not "is this document relevant to the original query in isolation?", but:

Given the original user query, the decomposed sub-queries, and all other retrieved documents, does this document contribute to form a complete and sufficient set of documents for answering the original query?

This is exactly why we tuned our pruner at kapa based on recall against this Document Relevance Model.