When Can Multi-Agent Code Review Be Trusted? Label-Free Detection Teaches AI to "Refuse to Guess"

When a language model judges whether code generated by another model is correct, it never tells you "I actually have no basis for this." It always delivers a confident conclusion with seemingly sound reasoning — and that "well-grounded judgment" is indistinguishable in appearance from a "pure guess." This is the core trust crisis facing multi-agent code judging today.

A newly published arXiv paper (arXiv:2609.30328) offers a sharp analysis of this problem: multi-agent verification works well for retrieval-based tasks, but when transplanted into code judging, it suffers systematic failure. The researchers not only diagnose the root cause of this failure, but also propose two label-free measurement methods that teach review systems to actively "abstain" when they lack sufficient evidence.

The Two Hidden Assumptions Behind Multi-Agent Verification

The core idea of multi-agent verification is to decompose a broad judgment into a set of verifiable specific claims, then validate each one against evidence. This "divide and conquer" approach works well when evidence comes from retrieved documents — for example, checking whether a factual answer is correct can be done by searching a retrieval corpus sentence by sentence.

The paper argues that such methods implicitly require two things from the evidence:

First, the evidence must be independent of the answer being reviewed. That is, evidence cannot be derived from the answer itself — otherwise it becomes circular self-justification.

Second, the evidence must be discriminative between the two candidate answers being compared. Verification can only deliver a meaningful verdict when the evidence associated with two candidate responses actually differs.

The critical distinction is this: in retrieval document settings, the second condition holds naturally — different answers surface different document snippets. But in code judging, this condition breaks down. Two code solutions may be functionally equivalent with similar logic, so the "evidence" extracted from them heavily overlaps, leaving the verification system unable to differentiate the two.

Multi-Agent Verification traces its roots to the "claim decomposition + retrieval verification" paradigm in NLP, whose canonical implementation breaks a long text into atomic claims and then uses a retrieval-augmented generation (RAG) system to find supporting or contradicting evidence for each claim from an external corpus. This approach excels at fact-checking because external documents naturally serve as independent arbiters. However, when the same pipeline is applied to code judging, the fundamental question of where "external evidence" comes from becomes critical: code correctness doesn't depend on external documents but on execution semantics and logical equivalence — evidence that is hard to retrieve and must typically be inferred from the code itself, which violates the "evidence independent of the answer" premise.

MARCH Framework's Measured Failure

To validate this argument, the researchers ran the published MARCH framework without any modifications on two code judging benchmarks, covering 80 condition-by-cell measurements.

The results were striking: MARCH deemed two solutions "equally good" in 78% to 95% of comparisons, failing to make any effective distinction. In these cases, its accuracy was suppressed to just 4.4% — while the same model, when directly asked "which is better," achieved an accuracy of 43.7%.

In other words, decomposing the judgment into a multi-agent verification pipeline not only failed to improve judgment quality — it caused accuracy to plummet nearly tenfold. Even more concerning, the problem cannot be mitigated by "lowering task difficulty" or "using a larger judge model" — the researchers tried both common optimization paths and neither changed the failure pattern.

This indicates that the issue is not insufficient model capability, but a structural flaw at the methodological level: when evidence cannot differentiate candidates, even the most capable model has nothing to work with.

MARCH (Multi-Agent Reasoning and Claim Hierarchy) is a published multi-agent verification framework designed to hierarchically decompose complex judgments: a top-level agent breaks the review task into sub-claims, while lower-level agents each verify individual claims, with results aggregated into an overall verdict. The theoretical advantage of this hierarchical division is reducing the cognitive load on any single agent and improving reliability through parallel verification. However, hierarchical architecture cannot create discriminative evidence that doesn't exist in the first place — when two code solutions behave very similarly, every lower-level agent faces the same "nothing to work with" predicament, causing most claims to be flagged as "undeterminable" and ultimately forcing the top-level aggregation to output a "tie." This demonstrates that architectural optimization cannot compensate for the fundamental absence of discriminative input evidence.

Two Label-Free Measurements: Diagnosing Problems from the Logs Themselves

The most valuable contribution of the paper is the proposal of two measurement metrics extracted directly from the pipeline's own logs — no ground-truth annotation required — that can explain the failure described above.

This has significant engineering implications. In real-world settings, we often lack ground-truth labels to evaluate whether a review system is reliable — if we had ground truth, we wouldn't need the reviewer in the first place. Label-free measurement means: using only the logs produced during system operation, we can determine whether any given judgment has a reliable evidentiary basis.

The researchers used these two metrics to successfully explain why MARCH delivers "tie" verdicts in the vast majority of comparisons — not because the two code solutions are genuinely equally good, but because the verification process simply never obtained evidence capable of distinguishing them.

Teaching Review Systems to "Refuse to Guess"

Building on one of these measurement metrics, the researchers introduced a gating mechanism: when the system detects that it lacks the evidence needed to make a distinction, it proactively abstains from the comparison rather than forcing out a confident but groundless conclusion.

This "know what you know" strategy produced tangible improvements: accuracy rose from 20.7% to 36.9%, while the system still answered half of all comparisons. In other words, it gave up on the questions it couldn't answer well anyway, focusing its output on judgments that had evidentiary support.

The researchers are particularly emphatic about one point: the contribution of this paper is not building a more accurate judge, but providing a label-free method to determine when a judge has no evidentiary basis for its answer. This is a shift in perspective — from pursuing "answer more correctly" to identifying "when not to answer."

Gating is a "selective activation" design pattern widely used in machine learning systems: before outputting a conclusion, the system first evaluates its own confidence or evidence sufficiency, and only produces an answer when a threshold condition is met; otherwise it refuses to answer or flags the sample for human review. This shares conceptual lineage with the router gates in Mixture of Experts (MoE) models and the uncertainty sampling approach in Active Learning. The key challenge of introducing gating in review settings is that it cannot rely on ground-truth labels to calibrate thresholds (otherwise the problem degenerates into supervised classification). The innovation here lies in using label-free measurement metrics — signals extracted from runtime logs — as the gating criterion, making the mechanism deployable in fully unlabeled production environments without needing to collect new annotation data for each task.

Implications for AI Review Systems

This research strikes at a soft underbelly of the current "LLM as judge" paradigm: the confidence of language models is often decoupled from the reliability of their judgments. A model can produce elaborate reasoning for a verdict that has no evidentiary basis whatsoever, and this "hallucinatory confidence" poses a latent risk in automated review, code auditing, model alignment, and similar settings.

For teams building automated code review or multi-agent verification systems, here are several practical takeaways:

• Multi-agent verification is not a universal solution — it can backfire in tasks where evidence cannot differentiate candidates;
• Rather than simply pursuing more powerful models, invest in mechanisms to identify "evidence-insufficient" scenarios;
• Giving a system the ability to proactively abstain may be more valuable than forcing it to always output a conclusion.

This paper doesn't promise a perfect judge, but it proposes a more honest framework — teaching AI to say "I'm not sure" when it has no solid ground to stand on. That may well be the first step toward trustworthy automated code review.