RAG benchmarks mislead: agentic search needs new metrics
Standard retrieval benchmarks cannot predict which content steers multi-step AI agents, leaving B2B brands blind to their real citation influence.
Key takeaways
- Static RAG utility and counterfactual trajectory utility are nearly uncorrelated across 23,322 document observations.
- In agentic search, a document's value lies in what it lets the model ask next, not whether it answers the current question.
- Major RAG evaluation frameworks (RAGAS, TruLens, LangSmith) are built on the static paradigm and cannot measure this gap.
- Foundational, framework-setting content outperforms point-answer content in multi-step agentic pipelines, even when benchmarks suggest otherwise.
- Brands optimising for featured-snippet retrieval are likely overestimating their influence on agentic AI outputs.
Retrieval-augmented generation benchmarks have a measurement problem. According to a preprint posted to arXiv, the correlation between Static Retrieval Utility (SRU) and what the authors call Counterfactual Trajectory Utility (CTU) is close to zero across 23,322 document observations in a multi-step agentic search setting. The document that scores well in isolation and the document that actually moves the agent toward a correct final answer are, statistically, barely related.
That is not a minor calibration issue. It is a structural failure in how retrieval systems are trained and how AI platforms select content to surface.
The measurement gap that matters
The standard RAG evaluation logic runs like this: take a document, take a question, ask a reader model whether the document helps answer it, record the score. Clean, reproducible, and wrong in the context agents now operate in.
The arXiv paper tests this with a ReAct-style agent running 1,000 HotpotQA development questions. For each document the agent retrieves, the authors delete it and re-run the trajectory from that point forward. The counterfactual comparison produces three deltas: final answer quality, the quality of the agent's next retrieval query, and the number of turns the agent needs to reach a conclusion. Together these form CTU.
The finding is simple and uncomfortable. A document's SRU score tells you almost nothing about its CTU score. In a multi-step search, documents matter for what they let the agent do next, not merely for whether they contain the answer to the current sub-question. A source that directly addresses query one may be useless in a five-step reasoning chain. A source that seems tangential on first contact may supply the conceptual bridge that shapes every subsequent query.
Static scoring cannot see the bridge. It only sees the plank.
What RAG systems are actually optimising for
The practical consequence is that retrieval systems trained on SRU-style signals learn to retrieve documents that look locally relevant. In agentic pipelines, where models like ChatGPT, Perplexity, or Gemini issue sequential queries, reformulate questions mid-session, and accumulate context across turns, local relevance is an unreliable proxy for actual utility.
This matters for any institution whose authority depends on being cited at the right point in a reasoning chain, not just at the first.
Consider how a policy analyst at a multilateral institution, or a credit officer at a large bank, might use an agentic AI tool to assemble a briefing. The agent issues an opening query on, say, systemic climate risk in sovereign debt markets. It retrieves three documents, reformulates its query based on what it has learned, issues four more, and synthesises a final answer. The documents that shaped the second and third queries, the ones that steered the reasoning, may never appear as explicit citations. They structured the output nonetheless.