The Cost of Retrieval Tool Output: New Findings on LLM Agent Accuracy Loss

LLM agents can compute the correct answer and still respond incorrectly. This is not a hypothetical—it is a widespread phenomenon revealed by a recent study. As agent frameworks increasingly move long tool outputs out of the context window and save them as files for on-demand retrieval, a critical question emerges: where exactly does this retrieval mechanism cause accuracy loss?

To answer this question, researchers chose a real scientific tool—PHREEQC, a geochemical simulator widely used by the U.S. Geological Survey, whose reports can reach 846,800 characters in length and whose determinism guarantees that every printed value is traceable. Agents used this tool to answer 158 execution-verified multiple-choice questions, enabling precise measurement of retrieval's impact on accuracy.

Retrieval Causes Accuracy Drops in GPT Models, No Significant Loss for Claude Models

Experimental results show that when agents read simulator reports via section indexing rather than in-context, GPT-5.1 and GPT-5.2 accuracy dropped by 11.3 and 7.9 percentage points respectively, while Claude Opus 4.6 and Claude Sonnet 4.6 showed no detectable cost. This contrast indicates that the impact of retrieval mechanisms varies by model.

Re-running all 11,007 recorded simulator calls revealed that both GPT models computed key values at roughly the same frequency as before; what changed was the evidence on which they based their answers. Under retrieval conditions, they frequently skipped the report and instead answered based on their own higher-precision output files, incorrectly rounding to an adjacent option. The researchers termed this failure mode "evidence substitution."

Context Window vs. External Retrieval: An LLM's context window refers to the total number of tokens the model can directly "see" in a single inference pass; capacity is limited and cost grows with length. When tool outputs (such as PHREEQC reports) run to hundreds of thousands of characters, agent frameworks typically do not stuff them entirely into context, but instead save them as files or vector stores, then have the model generate retrieval queries (e.g., by section heading index) to fetch fragments on demand. This "move out of context, retrieve on demand" approach saves tokens and supports extremely long outputs, but introduces a risk: the model may retrieve incomplete, imprecise, or non-authoritative fragments, or even skip retrieval entirely and rely on its own intermediate products. This mechanism is precisely the experimental variable behind the accuracy loss in this study.

Evidence Substitution: The Root Cause of Accuracy Loss

Evidence substitution explains why correct computation still yields wrong answers. The agent should have consulted the simulator report but instead relied on its own generated output files, which may contain subtle rounding differences that cause the final answer to deviate from the correct option. With simulation held fixed, retrieval still cost GPT-5.2 9.4 percentage points.

For GPT-5.4, more frequent correct simulations masked an evidence-use loss of about ten points, making the net change indistinguishable from zero. This shows that looking only at final accuracy can mask the true cost of retrieval.

Deterministic Simulator and Traceable Output: PHREEQC is a water chemistry equilibrium and reaction path simulator whose calculations are based on a given thermodynamic database and input conditions; identical inputs inevitably produce identical outputs, hence "deterministic." This determinism allows researchers to know precisely which correct option corresponds to each printed value, thereby attributing incorrect answers to evidence use rather than the simulation itself. Compared to tools relying on random sampling or approximate solving, deterministic simulators provide a reproducible ground truth, enabling "correct computation but wrong answer" to be cleanly isolated and making them an ideal experimental platform for measuring the impact of retrieval mechanisms.

Naming the Authoritative Output Improves Accuracy

Researchers found that naming the authoritative output improved all three GPT models by 6.3 to 7.9 percentage points over ordinary retrieval. For GPT-5.1 and GPT-5.2, inlining the appropriate output was equally effective, while requiring correct rounding did not help. This suggests that agent framework design should focus on where accuracy is lost, not just final accuracy.

A Cognitive Analogy for Evidence Substitution: Evidence substitution can be likened to a person reconciling accounts not by checking the bank's official statement but by drawing conclusions from intermediate calculations on their own scratch paper. The scratch paper may be off by only a few cents, but rounding to the option level leads to the wrong choice. In the agent pipeline, the model's generated output files (such as self-compiled intermediate results) often carry floating-point rounding or format-conversion errors, while the authoritative report preserves the tool's original full-precision printed values. Naming the authoritative output is equivalent to explicitly telling the model in the prompt "use the statement as the source of truth," thereby suppressing its tendency to take shortcuts; inlining key results further eliminates the retrieval step, placing the correct evidence directly in context.

Implications for Agent Framework Design

The choice of context management strategy should be judged by where accuracy is lost, not merely by final accuracy. The evidence substitution phenomenon shows that agents may bypass authoritative data sources and rely on their own outputs, leading to subtle but significant errors. Future framework design needs to more finely guide agents to use the correct evidence, for example by naming the authoritative output or inlining key results.

This study provides a rigorous method for evaluating retrieval mechanisms through deterministic simulators and traceable outputs. As agents are increasingly applied in high-stakes domains such as scientific computing, ensuring correct evidence use is critical.

Multi-Source Cross-Validation

According to analysis from Bilibili creators and YouTube channels, the study's results align with community observations: retrieval mechanisms perform very differently across models, and evidence substitution is a key factor driving accuracy loss. Multiple discussions emphasize that agent frameworks need to attend to the authority of evidence sources.

One-line Summary

Retrieval mechanisms cause accuracy drops in GPT models, rooted in evidence substitution rather than computation errors.

Paragraph Summary

Using the deterministic geochemical simulator PHREEQC, the study constructed 158 verifiable multiple-choice questions to examine how agent accuracy changes when long tool outputs are moved out of context and retrieved on demand. Results showed GPT-5.1 and GPT-5.2 accuracy dropped by 11.3 and 7.9 percentage points respectively, while Claude models showed no significant loss. Re-running 11,007 calls revealed that GPT models computed key values at unchanged frequency, but under retrieval conditions often skipped the authoritative report, relied on their own output files, and rounded incorrectly—termed "evidence substitution." Naming the authoritative output improved GPT models by 6.3–7.9 points, inlining outputs helped some models, while requiring correct rounding did not. The study recommends that framework design focus on where accuracy is lost rather than only final accuracy.