A Multi-Agent Review Panel Was Confidently Wrong: Lessons from a Million Tokens

How a "Confident Consensus" Turned into a Disaster

A developer had been using a multi-agent pipeline for code review for some time. His approach: have dozens of agents each take a different perspective, produce structured output, and run through adversarial verification rounds, yielding review results that looked extremely rigorous.

Last week, this pipeline delivered a "confident, unanimous, but completely wrong" answer. Because he kept the full transcript, he was able to pinpoint the cost of this failure and how to fix it.

The data came from one week of operation: 18 workflows, 158 agents, 139 MB of transcripts, and roughly 28 million tokens. About a third of that was avoidable waste.

How the Failure Happened

A review panel of three agents was tasked with examining a technical claim to determine whether it was true and worth acting on. All three said "no." Each wrote detailed, specific, well-argued reasons for rejection.

Their reasoning centered on a public standards document—the one that governed the decision. The problem: none of the three agents could open that document. It sat behind a login wall, and the agents correctly declined to authenticate (the behavior the author wanted to see). So they reasoned from the title, from search snippets, and from "what a document with that name probably says."

As it turned out, the actual content of the document was almost the opposite of what they'd guessed.

The author read that document himself through an authenticated browser session in a single tool call. The document's actual text—in its own worked examples—directly answered the two specific counterpoints the three agents had built their rejections on.

That read cost about 2,000 tokens. It overturned roughly 1.2 million tokens' worth of conclusions.

That's a 600:1 asymmetry, and it was the single most valuable thing the author did that week.

Multi-agent pipeline refers to splitting a task across multiple independently running LLM agents, each with its own prompt, role, or perspective, whose outputs are then aggregated, pitted against each other, or cross-validated. Common patterns include fan-out (distributing the same question to multiple agents for diverse answers), review panels (multiple agents independently ruling on the same claim), and adversarial verification (agents finding faults in each other).

Its core assumption is that majority opinion from independent perspectives cancels out single-point errors. But the hole this article exposes is that independence may be only skin-deep. If all agents share the same biased briefing, the same inaccessible source document, and the same ask-without-answers failure channel, they are effectively forking from the same false premise—and consensus amplifies rather than corrects the error. This is why the author emphasizes reading your own prompts, requiring a disagreements field, and adding provenance labels to outputs: all of these turn "pseudo-independence" back into "verifiable independence."

The Truly Unsettling Part

The author re-ran the review panel with the real document content. This time all three agents flipped and said "yes."

The result was good, but one thing unsettled him: both briefings were written by him. The first told the agents to be hostile and assume the claim was wrong. The second quoted the standard text and pointed out that the previous panel had misread it.

Same question, same model, same evidence. Opposite framing, opposite verdict.

So the second run wasn't a correction—it was the author getting back the answer he'd already decided on, with three agents returning it to him complete with citations. He caught it only because he happened to reread his own prompt and noticed the line "read the last paragraph twice" sitting directly above the passage he wanted the agents to focus on.

He killed that run and rewrote it: cite sources without commentary, strip helps and hurts labels before passing along evidence, and require every agent to fill in a mandatory field explaining where the briefing went wrong.

This version also said "yes." But this time two of the three agents told him: his briefing had exaggerated a constraint and buried a counterargument that should have surfaced. One pointed out that he'd handed the agents a favorable argument while omitting an unfavorable clause from the same document.

That's what a review panel is for. Consensus is cheap. The value is in disagreement, and you only get it when there's room for disagreement.

Framing effect originated as a behavioral economics concept: the same set of facts, presented differently (emphasizing loss vs. gain, assuming true vs. false), can lead people to opposite judgments. In the LLM context, it manifests as tone, ordering, labels, and implicit expectations in a prompt systematically biasing the model's output.

In this article, the author personally wrote the briefing in both versions: the first instructed the agents to "be hostile and assume the claim is wrong," and the second quoted the standard text and noted that the previous panel had misread it. Model, question, and evidence all unchanged—yet the verdict flipped completely. This shows that a model's "independent review" largely reflects the conclusion already baked into the prompt—a variant of what's called sycophancy: the model organizes its argument in the direction the user hints at, complete with citations that look solid.

So when a review panel delivers a unanimous verdict, what's actually being measured is often the bias of the briefing, not the strength of the evidence. To get informative disagreement, the briefing must be neutral: give sources without commentary, strip verdict-shaped labels like helps/hurts, and require agents to point out where the briefing might be wrong.

The Failure Mode You're Not Watching For

Midway through that week, the author hit a usage limit. Four agents died mid-run.

This caused no damage at all. The runtime printed failure blocks. An independent check of the logs with a one-liner confirmed it: count the number of started and result events in each directory, and if the numbers don't match, something died. Loud failure is cheap because it's visible.

The truly expensive failure was the agent that succeeded. It completed normally, returned a well-structured result, and reported that it couldn't access the needed document. Then it reasoned from the abstract anyway, and downstream agents treated its output as settled fact. Nothing raised an alarm, because nothing had gone wrong in any sense a harness could detect.

A dead agent tells you it's dead. An agent that guesses tells you nothing.

Provenance means labeling how each key claim was actually obtained, rather than only checking whether the conclusion is formatted correctly. In this article's schema, four values distinguish the cases: READ (actually read the source file), EXECUTED (actually ran the code or command), INFERRED (inferred from abstracts, titles, or indirect clues), and BLOCKED (needed a piece of material but couldn't access it).

Its value lies in replacing "does the result look right" with "can the result be traced." An agent that completes normally, is well-structured, and yet can't say what it read is a completely different creature from a dead agent—the latter trips a failure block, while the former quietly passes its guess downstream, where it's treated as established fact. Adding a mandatory provenance field costs almost nothing, yet it lets a reviewer immediately see which premises are INFERRED and which numbers were trusted from other agents without verification. Both models the author tested followed the field well, and the outputs shifted from "confident" to "verifiable."

The Nearly Free Fix

The author added a mandatory field to every judging agent's output schema:

provenance: for each claim carrying a key argument, label READ (what was read),
            EXECUTED (what was run), INFERRED, or BLOCKED.
            Verdicts whose key premises are INFERRED must say so.

Both models he tested followed this field well, at almost zero cost. It turned returned results from "confident" into "verifiable."

The best example came from one run: an agent flagged a login-restricted page as BLOCKED, precisely listed the specific numbers it had trusted from earlier agents and not re-run, and separated those from facts it had verified itself from source files. That agent's verdict was more valuable than three confident verdicts, because the author could see where to dig deeper.

The Most Thorough Run Returned a Clean "Nothing"

After seeing the framing effect, the author wanted a harder measurement, so he set up a blind test. The method was the same one he uses to score evaluation suites: take material whose answer is already known, hide the answer, and see what the measurement tool actually detects. A detection rate you've measured beats a detection rate you claim.

He locked six agents inside a source tree, moved every file containing prior analysis out of reach, and gave all six agents an identical prompt that never said what to look for.

Three of the six surfaced the underlying mechanism. What really mattered was two independent observations—which only became meaningful when put together.

His first pass scored all six as failing to make the connection. That was wrong, and wrong for an embarrassing reason: he'd scored their outputs using keyword search instead of actually reading. When he actually read, he found that two of the three had explicitly made the connection. One wrote that a step in its third finding was "exactly the precondition that finding 1 needs." Another wrote that the same step was "precisely what finding 1 and point (a) above jointly exploit."

Then each of them filed that observation as a sub-clause of a minor finding and moved on.

So the failure wasn't in detection, or even in composition. It was in re-scoring. Both agents found the connection and wrote it down in plain language, then let the combination inherit the low severity of its parts, without asking what the combination itself was worth. The interesting result isn't that agents miss connections—it's that they can find connections and still fail to update their judgment accordingly.

The author's scoring error had the same shape: a check returned a confident answer for the wrong reason, and only reading the underlying material caught it. Three times in one week. That's probably the whole point of this article.

An Agent That "Found Nothing" Was Actually the Most Careful

The run the author keeps thinking about returned nothing.

That agent did the most meticulous work of the six. It built the project, ran the existing 391 tests, extracted every reference document in the repository, and checked each candidate finding against all of them. It correctly matched the fragments it found to known, documented, accepted issues. Then it reasonably concluded there was nothing new.

One line in its notes described the exact lines of code where the real problem lived, and recorded them as safe.

It wasn't careless. It was following the author's instruction to "treat the known-issue list as a hard filter"—and it followed that instruction more faithfully than the agents that came closer. Each documented issue describes a mechanism. Each carries an exception for a case that its own mechanism never touches. The combination touched it.

So the author's filter was wrong, not the agent. "Does this match something known" discards the combination that escapes the known scope. The question has to be: "Does this match, and does any combination of these candidates touch the known items' own declared exceptions?"

The generalized version: when you give an agent a filter, you're also telling it what to throw away—and a careful agent will throw away more.

Combinatorial emergence is the class of finding most easily missed in this kind of pipeline: individual candidate problems may each be trivial, harmless, and covered by the known list, but two or more stacked together trigger the exception case of some already-documented rule, producing a genuinely new problem.

The agent in this article that "found nothing" was judged safe because it was told to "treat the known-issue list as a hard filter"—each known issue describes a mechanism and carries an exception its own mechanism never touches. Checking candidates one at a time, none fell into the exception; combined, it hit exactly that. The problem was the filter, not the agent.

The generalized lesson: when you give an agent a filter, you're also telling it what to throw away, and a careful agent will throw away more. Hence the need for a dedicated "combination round"—a single agent (not a fan-out) reviewing the whole candidate list, asking whether any combination sums to something none of them equal alone, and re-scoring that combination from scratch rather than letting it inherit the low severity of its components.

What the Author Changed

Four rules, in execution order:

One: Read the governing document yourself before any agent starts. Whatever the decision actually hinges on—specs, standards, policies, configs—read all of it. Hand a single file of facts to every agent, so nobody re-derives and nobody guesses. If it's behind a login wall, that's human work, and it takes one tool call.

Two: Put cheap checks before expensive ones. The author's pipeline built full reproductions first, then discovered far too late that two candidates died on questions that each needed one query. Now the cheap elimination questions run on all candidates first, and only then do expensive checks run on anything.

Three: Never brief a review panel toward an answer. Cite sources without commentary. Strip verdict-shaped labels before passing along evidence. Require a disagreements field.

Four: Bound exploratory work by time or error rate, not by completion. One probe ran 99 minutes and accumulated 465 errors before dying to a rate limit, chasing something an earlier probe had already structurally ruled out. Nothing stopped it, because nothing was watching except whether it finished.

Five: Add a combination round, and re-score what it finds from scratch. One agent, not a fan-out, asks of the whole candidate list: is there any combination here that sums to something none of them equal alone, and what is that combination worth—on its own terms, not inherited from its parts. That last clause is exactly where the failure happens.

Six: Audit your filters. For every filter you hand an agent, write down what it throws away, then go take a look in the trash.

Seven: Don't score agents with regex. The author scored a detection test with keyword search and got the headline result backwards. Read the output.

Honest Limitations

One week. One pipeline. One domain. The author measured transcript volume and token counts, not a controlled experiment, and the framing-effect result is n=3, a single question.

What he can be confident about is narrower than it sounds: when a panel of agents agrees, and every briefing was written by the same person, that agreement is evidence about the briefing. Everything else above is a corollary of that fact.

The deeper version of this problem isn't new. A check that can't fail isn't a check. We know this about CI gates—a required task satisfied by a skipped step can pass review for years, because nobody ever asked it to prove it can go red. We know it about test suites. The same holds for review panels, and it's harder to see, because the output is fluent, structured, and arrives in triplicate.

If you're running agent fan-outs, go read your own prompts. Find the sentence—the one where you told them what you wanted to hear.

The author found his.