How Do Large Models Integrate External Evidence? Distributional Theory Reveals the Mechanism Behind LLM Evidence Integration

Introduction: The Black Box of LLM Evidence Integration

As large language models (LLMs) increasingly rely on external evidence for reasoning — whether from tool calls, Retrieval-Augmented Generation (RAG), other agents, or user-provided information — a critical question remains unresolved: When a model has already begun forming its own preliminary answer, how exactly does it integrate external evidence into its final decision?

Retrieval-Augmented Generation (RAG) is one of the most mainstream architectural paradigms in current LLM applications. Its core idea is to retrieve document fragments relevant to the user's query from external knowledge bases (such as vector databases, search engines, enterprise document repositories, etc.) before the model generates a response, then inject these fragments as context into the model's prompt to help generate more accurate, up-to-date answers. RAG has partially addressed the limitations of LLM training data cutoff dates and hallucination issues, but it also introduces new risks: the retrieved documents themselves may contain erroneous, outdated, or misleading information. How models handle such flawed evidence is precisely the core concern of this research.

A recently published study on arXiv (arXiv:2609.04290v1), Evidence Integration in Large Language Models, systematically answers this question. The researchers proposed a distributional theory and validated it through over ten million experimental trials, revealing the deep-seated patterns governing how LLMs integrate evidence. These findings carry not only theoretical value but also direct practical implications for RAG system design, multi-agent collaboration, and AI reliability engineering.

Core Theory: How Evidence Shifts the Model's Answer Distribution

The Distributional Shift Model

The study's core hypothesis is that external evidence is not simply "accepted" or "rejected" by the model, but rather shifts the probability distribution of the receiver's initial answers. This shift is driven by two key factors:

• Receiver prior weight: The strength of the model's initial inclination toward a particular answer;
• Candidate evidence tilt: The degree to which external evidence pushes toward a specific candidate answer.

In other words, LLM evidence integration is not a scalar decision about "trusting the source," but rather a receiver-specific control policy in which the model adjusts based on its existing distribution.

This distributional shift model has deep intellectual roots in Bayesian inference frameworks. In classical Bayesian reasoning, an agent possesses a prior distribution and, upon observing new evidence, updates the prior into a posterior distribution via Bayes' rule. However, the key difference revealed by this study is that LLMs do not perform rationally weighted updates based on evidence reliability like an ideal Bayesian agent. Instead, they exhibit a behavior pattern closer to "distributional tilting" — external evidence influences output by altering the probability weights of candidate answers, and the relationship between this influence and the actual reliability of the evidence is far weaker than what a Bayesian framework would predict. This means LLM evidence integration more closely resembles a statistical mixture of distributions rather than rational belief updating.

Three Key Predictions and Validation Results

Based on this theoretical framework, the study proposed and validated three counterintuitive predictions:

1. The more likely a model considers a candidate answer, the more persuasive it becomes. The effectiveness of evidence depends on whether it aligns with the model's prior inclinations.
2. Models are more susceptible to integrating "mistakes they themselves would make" rather than "unfamiliar errors" from different sources. That is, models are less resistant to errors consistent with their own reasoning characteristics.
3. The same evidence can improve weaker models' performance while degrading stronger models' performance. This means there is no one-size-fits-all evidence augmentation strategy.

Large-Scale Experimental Validation: Thorough Testing Across Models and Domains

To validate the above theory, the researchers conducted an extraordinarily extensive experiment:

• Over ten million trials;
• Twelve LLMs from four model families;
• Covering eight domains, including four scientific discovery tasks in physical and life sciences: quantum mechanics, physics, genetics, and molecular biology.

This large-scale validation across models and domains gives the research conclusions considerable generalizability. Experiments confirmed the high consistency of distributional shift patterns across different models and tasks.

The Receiver-Relative Reliability Frontier

The study also introduced an important concept — the receiver-relative reliability frontier. Experiments found that receiver-congruent errors (errors aligned with the receiver's tendencies) degrade model performance more severely than random errors with equivalent error rates.

This finding is particularly critical for RAG system optimization: if retrieved erroneous information happens to align with the model's own "error preferences," its harmful impact is significantly amplified. Therefore, when evaluating the reliability of external evidence sources, one cannot simply look at the overall error rate — the correlation between errors and the target model's tendencies must also be considered.

The Most Concerning Finding: Dissociation Between Verification and Integration Behavior

Correct Verification Does Not Equal Rejection of Errors

The most thought-provoking conclusion from the study is: Even after an LLM has internally verified that a candidate answer is invalid, it still integrates it into the final answer.

The specific data are striking:

• In tasks with propositional constraints, the rate at which models integrated candidates already verified as invalid reached 93%–100%;
• On held-out physical and life sciences reasoning tasks, this rate reached as high as 99.4%.

This demonstrates that verification capability and integration behavior are two nearly independent processes. The model "knows" the evidence is wrong, but this "knowing" does not prevent it from adopting that evidence.

Causal Mechanism: Integration Occurs in Later Network Layers

Through causal interventions, the researchers further localized the internal mechanism of evidence integration within the model. Causal intervention is one of the core methodologies in the field of mechanistic interpretability. Its basic approach involves artificially modifying the activation values at specific layers or positions during the model's forward pass (e.g., replacing, ablating, or injecting them), then observing the impact of these modifications on the final output. Common techniques include activation patching, causal tracing, and interchange interventions. Through these methods, researchers can attribute model behavior to specific internal components and computational steps, going beyond purely behavioral observation to deeply understand the model's internal causal mechanisms.

Modern large language models are based on the Transformer architecture, consisting of multiple alternating layers of self-attention and feed-forward networks. Recent interpretability research has revealed that different layers of Transformers serve different functions: shallow layers typically handle lexical and syntactic features, middle layers gradually form semantic representations and factual recall, while deep layers are responsible for task-specific decision-making and output generation. This study's finding that evidence integration occurs in later network layers is consistent with existing research on "deep layers handling decision integration." But the more important finding is that while verification information is already encoded in the middle layers, it fails to effectively propagate to the decision pathways in deeper layers, suggesting a potential structural bottleneck in information flow within the Transformer architecture.

Specifically, the study reached the following key conclusions:

• Candidate integration occurs in later network layers, manifesting as a structured sequence of steps — admitting the external candidate answer → boosting its weight → transmitting it to the answer state;
• Representations of verification, while decodable, have almost no causal impact on the final answer;
• An analysis called J-lens decomposition showed that the internal states supporting "verbalized verification" and those supporting "candidate integration" are fully dissociable.

J-lens decomposition is a technical method for analyzing internal representations in Transformers. Its core idea is to project the hidden states at a given layer into vocabulary space or a task-specific space to interpret what that layer is "thinking about." Specifically, J-lens (in the same lineage as logit lens, tuned lens, and similar techniques) multiplies intermediate-layer activation vectors by the model's output unembedding matrix to obtain the "predicted probability" of each token at that layer. This study used J-lens decomposition to distinguish between internal states supporting verbalized verification (e.g., the model generating statements like "this answer is incorrect") and internal states supporting candidate integration (the model actually selecting that answer), finding that the two are completely separable in representation space — key evidence for the verification-integration dissociation hypothesis.

This explains at the mechanistic level why large language models "knowingly do wrong": verification and integration travel along two different neural pathways.

Implications for AI Engineering Practice

This research poses profound challenges to current mainstream LLM application paradigms:

First, RAG systems need more careful evidence filtering mechanisms. Since models tend to integrate evidence consistent with their own priors (even when it's wrong), relying solely on model "self-verification" to filter erroneous information is unreliable. Independent external validation mechanisms need to be introduced.

Second, multi-agent systems must guard against "homogeneous error amplification." When multiple models from the same family collaborate, their shared "characteristic errors" are more readily accepted by one another, causing errors to be reinforced rather than corrected within the system. Multi-Agent Systems are a rapidly evolving paradigm in current AI engineering, with typical applications including multiple LLMs collaborating on complex tasks (e.g., frameworks like AutoGen, CrewAI). Ideally, multiple agents should reduce errors through mutual verification, similar to the "majority voting" mechanism in ensemble learning. However, the "receiver-specific error amplification" phenomenon revealed by this study poses a serious challenge: if multiple agents come from the same model family (e.g., different instances or parameter variants of GPT-4), they may share similar prior biases and characteristic error patterns. When one agent's erroneous output is passed to another agent with the same bias, the latter not only fails to correct the error but is more likely to accept it because it aligns closely with its own priors. This phenomenon is similar to "group polarization" in social psychology and could lead to catastrophic error cascades in AI systems. Therefore, when designing multi-agent systems, agents from different architectures, different training data, or different model families should be intentionally introduced to achieve genuine "cognitive diversity."

Third, more evidence augmentation is not always better. For more capable models, introducing external evidence may actually degrade performance. This suggests that retrieval augmentation strategies should be differentiated based on model capability. For example, for strong models that already possess high-confidence priors in specific domains, a higher evidence admission threshold may be needed, triggering retrieval augmentation only when the model's own uncertainty is high; for weaker models, external evidence can be more aggressively introduced for assistance.

Conclusion

This research redefines LLM evidence integration from a vague concept of "trust" into a distributional control policy determined by the receiver's own properties. It reveals an unsettling reality: models can verify the authenticity of evidence, yet cannot constrain their own behavior accordingly.

As AI systems increasingly depend on external tools, retrieval-augmented generation, and multi-party collaboration, understanding this mechanism is not merely an academic question — it is foundational to building reliable and trustworthy AI systems. Future alignment and safety work may need to specifically focus on bridging the gap between verification and integration, enabling models not only to "know" what is right but also to "do" the right thing. This challenge resonates with the broader "knowing-doing gap" problem in the field of AI alignment — current alignment techniques (such as RLHF, Constitutional AI, etc.) primarily focus on model values and behavioral boundaries. But the verification-integration dissociation revealed by this study indicates that even when a model possesses correct judgment capabilities, ensuring that this judgment effectively influences its actual behavior remains a fundamental unsolved challenge.