Epistemic Humility: How AI Agents Should Handle Conflicts Between Retrieved Evidence and Prior Knowledge

Most agent evaluations only measure whether a task was successfully completed, overlooking a critical question: what does an agent do when its retrieval layer returns evidence that contradicts its training data? A new paper by Sun et al. (arXiv:2610.12360v1) introduces Epistemic Humility as a novel evaluation dimension — does the agent revise its answer, flag uncertainty, or stubbornly commit to a wrong conclusion?

This question is crucial for financial decision agents, compliance bots, and any system that needs to reconcile stale training data with real-time data streams. If your agent places a trade based on outdated earnings guidance while ignoring a contradictory SEC filing, task accuracy alone is not enough. You need to know whether it can detect the conflict and escalate accordingly.

The Overlooked Evaluation Gap

Most agent benchmarks only score the final answer, without tracking whether the agent noticed conflicting evidence during execution. The paper decomposes this behavior into three dimensions:

• Identify: Does the agent notice the conflict during retrieval or reasoning steps?
• Solve: Does it revise its answer or reasoning chain when evidence contradicts prior knowledge?
• Escalate: Does it communicate unresolved uncertainty in the final output (e.g., "I found contradictory sources")?

The researchers tested four agent configurations under two conflict scenarios: controlled conflicts, where contradictory evidence is deliberately injected into the retrieval context, and naturally occurring conflicts, where sources diverge across multi-step execution or parametric knowledge contradicts retrieved documents. Each conflict scenario includes a conflict-free control group to isolate the true effect of knowledge conflict on agent behavior.

Parametric knowledge refers to the implicit memory a language model forms by compressing knowledge into its weight parameters during pretraining, as opposed to non-parametric knowledge retrieved at inference time. When these two conflict — for example, when a revenue figure stored in model weights differs from a freshly retrieved earnings report — the model faces a fundamental choice: trust its "memory" or trust the "document." This tension is an inherent contradiction in RAG (Retrieval-Augmented Generation) systems and the central motivation for epistemic humility evaluation. Existing research shows that LLMs exhibit a clear "memory preference" in such conflicts, tending to rely on knowledge acquired during training even when retrieved evidence is more recent and authoritative.

Key Finding: High Accuracy ≠ Epistemic Humility

The paper's most important finding is that high task accuracy does not guarantee epistemic humility. Some agents identified conflicts in early steps but failed to maintain or resolve them in later steps; others detected contradictions during execution yet still produced a confident final answer.

A typical failure pattern: the agent logs a conflict in an intermediate tool call and internally revises its reasoning, but the final answer carries no uncertainty marker. The evaluation framework sees a "correct answer" and marks it as a success — even though the agent effectively ignored unresolved evidence.

This is fundamentally a pipeline problem. If your agent uses a ReAct loop or chain-of-thought prompting, the conflict may appear at step 3 of 7, but the final answer generator never inherits that uncertainty signal. You need explicit state management to carry the "I'm not sure" flag all the way from the reasoning trace to the output formatting stage.

ReAct (Reasoning + Acting) is one of the most widely adopted agent execution frameworks, introduced by Yao et al. in 2022. Its core idea is to interleave "reasoning" (generating thinking steps) with "acting" (calling tools or APIs), forming an iterative chain of think → act → observe → re-think. Chain-of-Thought Prompting is a technique that improves answer quality by having the model write out its reasoning step by step. The shared problem with both mechanisms is that state generated in intermediate reasoning steps — including uncertainty flags — is not automatically carried forward to the final answer generation step. The final answer is typically produced by a separate generation call, and without explicit context-passing mechanisms, conflict signals identified in earlier steps silently disappear in the pipeline.

How to Instrument Epistemic Humility Measurement

Measuring epistemic humility requires injecting contradictory evidence and tracking three categories of signals:

1. Tool call logs: Did the agent retrieve two sources? Did it invoke a verification tool or re-query?
2. Reasoning traces: Does the chain-of-thought mention the conflict, or does it silently pick one source?
3. Final output structure: Does the answer include confidence scores, caveats, or escalation flags?

The paper presents a minimal evaluation framework pattern: build an EpistemicHumilityEval class that modifies tasks via a conflict injector, executes the agent with trace logging, and scores across three dimensions — identified_conflict, revised_answer, and communicated_uncertainty.

The key is separating the scoring logic: conflict detection can be done by scanning reasoning text for the keyword "conflict" or for verify_source tool calls; revision detection by comparing whether the initial candidate answer differs from the final answer; uncertainty communication detection by checking whether the output contains markers like "conflicting," "uncertain," or "sources disagree." This pattern fully decouples task success from epistemic behavior — you can score an agent as "correct but not humble" or "humble but incorrect."

Confidence Scores vs. Explicit Uncertainty Flags

There are two paths to surfacing uncertainty:

Logit-based confidence: Extract token probabilities from the final answer and apply a threshold. This works for single-turn completions but breaks down in multi-step agents, where the final answer is assembled from multiple tool outputs.

Explicit uncertainty flags: Add structured output fields (e.g., confidence_level, unresolved_conflicts) and force the agent to populate them. This requires prompt engineering or schema-constrained output parsers.

The paper shows that model-level interventions — such as adding "acknowledge uncertainty when sources conflict" to the system prompt — improve epistemic humility but often reduce task accuracy. The agent becomes more cautious, which is desirable in high-stakes domains, but hurts performance on benchmarks that penalize "I don't know" answers.

Trade-offs in Agent Design

Different conflict-handling strategies make different trade-offs across accuracy, humility, latency, and complexity:

Approach	Task Accuracy	Epistemic Humility	Latency	Complexity
No conflict handling	High	Low	Low	Low
Logit-based confidence	Medium	Medium	Low	Medium
Explicit uncertainty flags	Medium	High	Medium	High
Multi-step verification loop	Low	High	High	High

Multi-step verification loops — where the agent re-queries or invokes a fact-checker upon detecting a conflict — achieve the highest epistemic humility but perform worst on task accuracy in standard benchmarks. This is because most benchmarks assume a single correct answer and don't reward "acknowledging uncertainty."

How This Looks in Production Financial Agents

If you're building a financial decision agent that consumes real-time market data, regulatory filings, and analyst reports, you need to handle knowledge conflicts at the orchestration layer:

1. Source deduplication: Tag sources with timestamps and authority rankings; if two sources conflict, present both to the reasoning step.
2. Conflict detection middleware: Before the agent finalizes its answer, run a conflict checker that compares retrieved evidence against parametric knowledge or prior tool outputs.
3. Escalation routing: If the conflict checker flags unresolved uncertainty, route the task to human review or a higher-tier model rather than auto-executing the trade.

This adds latency, but it prevents catastrophic failures. A trading agent that ignores a contradictory SEC filing and buys based on stale earnings data is far more dangerous than one that detects the conflict and escalates to a human.

Failure Modes to Watch For

The paper also highlights several characteristic failure types:

• Early detection, late forgetting: The agent detects a conflict at step 2 but forgets it by step 6, ultimately outputting a confident answer.
• Silent preference: The agent retrieves two sources but silently selects the one that aligns with its prior, with no explanation in the reasoning trace.
• Hedging for score optimization: The agent outputs "sources conflict" regardless of whether an actual conflict exists, to maximize its epistemic humility score.

The third is a classic Goodhart's Law problem. If you optimize purely for uncertainty acknowledgment, the agent will hedge on every answer. You must pair the epistemic humility metric with task accuracy and penalize false positives (claiming conflicts where none exist).

Goodhart's Law originates from British economist Charles Goodhart, and its core formulation is: "When a measure becomes a target, it ceases to be a good measure." In machine learning, this law recurs in many forms: models find shortcuts to maximize evaluation scores rather than genuinely learning the target capability. For the epistemic humility scenario specifically, if the system only rewards the act of "claiming uncertainty," the agent will learn to output uncertainty statements indiscriminately — which in practice causes even well-grounded answers to be unnecessarily flagged, drastically reducing system utility. The standard solution is to construct a dual-objective evaluation: simultaneously tracking the epistemic humility rate and the uncertainty precision rate (i.e., the proportion of claimed conflicts that are genuine), replacing single-dimension optimization with an F1-style composite score.

Observability Requirements

To debug epistemic humility failures, you need: structured trace logs (capturing every tool call, reasoning step, and intermediate answer); conflict annotations (metadata tagging retrieved outputs with timestamps, authority levels, and contradiction flags); and diff tracking (recording when and why the agent revised its answer). Without these, you cannot distinguish an agent that genuinely detected a conflict from one that simply got lucky.

Technical Conclusion: Accuracy and Humility Are Orthogonal

When to use epistemic humility evaluation: the agent operates on real-time data that may contradict its training cutoff (financial markets, regulatory updates, breaking news); the cost of a confident wrong answer exceeds the cost of escalation (medical diagnosis, legal advice, trading); you need to audit agent behavior for compliance or safety review.

When to avoid this evaluation dimension: tasks have a single ground truth and no conflicting sources (closed-domain QA, deterministic code generation); latency is critical and multi-step verification loops are unaffordable; your benchmark penalizes "I don't know" answers and you're optimizing for leaderboard rankings.

The core insight of this research is that task accuracy and epistemic humility are orthogonal dimensions. You can build agents that are accurate but not humble, or humble but not accurate. For high-stakes agent systems, you need both — and you need the evaluation infrastructure to measure them independently.