[KongchangAI]
· 2 min read· 1,007 words

LLM Verifier Reliability Across Abstraction Boundaries: Abstraction Causes 31% of Judgments to Fail

LLM Verifier Reliability Across Abstraction Boundaries: Abstraction Causes 31% of Judgments to Fail

LLM verifiers have up to 31% judgment transport failure under abstraction boundaries, and restoring provenance repairs 85.5% of failures.

The study proposes the Verification Transport framework to systematically measure LLM verifier judgment consistency when execution evidence crosses abstraction boundaries. Experiments construct native, abstracted, and restored views of the same frozen execution process across four benchmarks, seven verifier backbones, and four protocols. Results show transport failure rates up to about 31% under macro abstraction and delegation boundaries, meaning nearly one-third of judgments change. Failure modes differ by protocol: direct verification more often yields erroneous judgments, while evidence-grounded and deliberative protocols tend to abstain, so lower failure rates do not necessarily mean higher reliability. Restoring execution provenance repairs 85.5% of transport failures, but recovery drops when provenance is lossy. The study recommends testing verifiers under multiple abstracted views and incorporating transport failure rates into reliability metrics.

LLM-driven verifiers are becoming an important tool for assessing the execution quality of agents, but their reliability is typically measured only under a single trajectory representation. In real-world deployments, the same execution process is often presented through composed tools, delegation boundaries, or compressed histories, which changes the evidence directly observable by the verifier. So, can the verifier's judgment remain consistent when execution evidence crosses abstraction boundaries? A study called "Verification Transport" is the first to systematically measure this question.

LLM Verifier Reliability Across Abstraction Boundaries

How Abstraction Boundaries Undermine Verification Reliability

The research team constructed three views of the same frozen execution process: native, abstracted, and restored. Keeping the task, trajectory, outcome, ground-truth labels, evaluation criteria, and verifier fixed, they measured the stability of judgments under abstraction-induced observational changes.

The experiments covered four agent evaluation benchmarks, seven verifier backbones, four verification protocols, and three types of abstraction boundaries. The results show that under macro abstraction and delegation boundaries, the transport failure rate reaches as high as about 31%. This means that even if a verifier performs well under the native view, nearly one-third of its judgments change once execution evidence is abstracted.

This finding poses a direct challenge to agent evaluation practice. Most current evaluation pipelines assume that verifiers maintain consistent judgments under compressed or composed views, but the data show that this assumption does not hold.

Macro abstraction and delegation boundary are the two main types of abstraction boundaries in this study's experiments. Macro abstraction refers to merging multiple low-level tool calls or steps into a single high-level semantic unit (for example, compressing "read file—parse content—extract field" into "extract field"), thereby reducing the intermediate states visible to the verifier. Delegation boundaries arise in multi-agent systems: the main agent delegates subtasks to sub-agents, and if the verifier observes only the main agent's trajectory, it cannot see the decisions and evidence inside the sub-agents. Both types of boundaries alter the set of evidence directly observable by the verifier and are the main sources of transport failures. Understanding their differences helps determine which system architectures are most prone to verifier failure.

Failure Modes Differ Significantly Across Verification Protocols

The study reveals different failure modes of verification protocols under abstraction. Direct verification more often produces erroneous judgments—the verifier still forces a conclusion when information is incomplete. In contrast, evidence-grounded and deliberative protocols tend to abstain when observability is limited, rather than risk giving a wrong answer.

This difference leads to a counterintuitive conclusion: a lower transport failure rate does not necessarily mean higher reliability. Direct verification may appear to have a lower failure rate because it "always gives an answer," but it contains more erroneous judgments; deliberative protocols abstain more often but avoid incorrect conclusions. When evaluating verifiers, it is essential to distinguish between two failure modes: "error" and "abstention."

Abstain means that when evidence is insufficient or observability is limited, the verifier chooses not to give a binary "pass/fail" judgment but instead outputs "cannot determine" or requests more information. In this article's context, abstention is an explicitly permitted protocol behavior, as opposed to an erroneous judgment. Direct verification protocols are typically designed to always produce a conclusion, so even when faced with abstracted and compressed trajectories they force a judgment, leading to a higher error rate; evidence-grounded protocols require judgments to be supported by traceable evidence, and deliberative protocols check evidence sufficiency through multiple rounds of reasoning, so both are more likely to abstain when observability is limited. If evaluation counts only the "judgment error rate" while ignoring the abstention rate, it will systematically overestimate the reliability of direct verification protocols.

Restoring Execution Provenance Recovers Most Failures

The study further tested the effect of restoring execution provenance. When the corresponding provenance information was restored, 85.5% of transport failures were repaired. However, when provenance information was lossy or unavailable, the recovery rate dropped significantly.

This result points to an engineering practice: when designing abstraction mechanisms, agent systems should deliberately preserve recoverable provenance cues. Composed tool calls, delegation chains, and compressed histories should not discard the mapping to the original execution evidence; otherwise, the verifier will permanently lose the information needed to make correct judgments.

Implications for Agent Evaluation

The core message of this study is that accuracy under the native view is insufficient to characterize verifier reliability. When execution evidence crosses abstraction boundaries, verifiers may systematically fail. For teams building agent evaluation systems, this means testing verifiers under multiple abstracted views and incorporating the transport failure rate into reliability metrics.

At the same time, the choice of verification protocol needs to match the degree of abstraction. In scenarios with high abstraction and low observability, protocols that force a judgment may introduce more errors; protocols that allow abstention and preserve provenance may appear "conservative" but are more likely to maintain judgment quality.

As agent systems become increasingly complex, abstraction boundaries are inevitable. Whether verifiers can maintain consistent judgments across these boundaries will directly affect the credibility of evaluation results. Verification Transport provides an actionable testing framework and is worth incorporating into evaluation processes before deploying verifiers.

One-line Summary

LLM verifiers have a transport failure rate of up to 31% under abstraction boundaries, and restoring provenance can repair 85.5% of failures.

Paragraph Summary

The study proposes the "Verification Transport" framework to systematically measure the judgment consistency of LLM verifiers when execution evidence crosses abstraction boundaries. The experiments construct native, abstracted, and restored views of the same frozen execution process, covering four benchmarks, seven verifier backbones, and four protocols. Results show that under macro abstraction and delegation boundaries, the transport failure rate reaches about 31%, meaning nearly one-third of judgments change. Failure modes differ by protocol: direct verification more often produces erroneous judgments, while evidence-grounded and deliberative protocols tend to abstain, so a lower failure rate does not necessarily indicate higher reliability. Restoring execution provenance repairs 85.5% of transport failures, but the recovery rate drops when provenance is lossy. The study recommends testing verifiers under multiple abstracted views and incorporating the transport failure rate into reliability metrics.

#LLM verifier#abstraction boundaries#Verification Transport#agent evaluation#execution provenance
Share:

Related articles