Phantom Consensus in Multi-Agent LLMs: How Social Reasoning Leads to Collective Decision Failures

Multi-agent LLM systems are widely used to aggregate judgments from multiple model instances, but a new study reveals a hidden failure mode: after observing ambiguous social signals, models change their public decisions and misestimate other agents' private judgments—even when the group's recorded private judgments do not support this apparent consensus. The researchers call this phenomenon "Phantom Consensus" and introduce a dedicated benchmark, PHANTOM-Bench, to quantify it.

What Is Phantom Consensus?

Phantom consensus refers to a collective illusion in multi-agent systems: agents outwardly display converging decisions, but this convergence is not based on genuine agreement in private judgment—it is induced by ambiguous social signals. The study draws an analogy to "pluralistic ignorance" in human social psychology—individuals privately disagree but, observing others' ambiguous behavior, mistakenly assume everyone agrees, and adjust their public expression accordingly.

In controlled LLM collectives, the researchers found that models not only change their decisions during the public phase but also give severely distorted estimates when reporting the "group's private judgment." This means the "consensus" the system ultimately presents may be entirely fictitious, matching no agent's internal true judgment.

Pluralistic Ignorance is a classic concept in social psychology: most members of a group privately disagree with a certain view or behavioral norm but mistakenly believe that most others agree, and therefore publicly go along with it. Typical examples include the classroom situation where "no student asks a question because everyone assumes the others understand." Its formation depends on two conditions: the unobservability of others' private attitudes and the ambiguity of public behavior. In LLM multi-agent systems, both conditions hold—agents can only see each other's public outputs and cannot directly access each other's internal judgments, so ambiguous public signals are sufficient to induce similar cognitive biases. Understanding pluralistic ignorance helps grasp the theoretical root of phantom consensus: it is not simple conformity but a metacognitive error about the distribution of group beliefs.

How Does PHANTOM-Bench Measure This Phenomenon?

To systematically capture phantom consensus, the research team designed the PHANTOM-Bench benchmark, decomposing the decision process into multiple independently observable stages:

• Isolated private judgment: the independent judgment an agent makes without social-signal interference
• Public decision after ambiguous social signals: the public choice an agent makes after observing ambiguous group behavior
• Estimation of the group's private judgment: the agent's inference of what other members privately think
• Transparency recovery: whether failures can be repaired after publicly revealing the agents' recorded private decisions
• Control conditions: neutral public text and explicit follow-the-vote instructions, used to distinguish the effects of ambiguous signals from explicit directives

The key value of this staged design is that it separates "whether there is ultimately agreement" from "whether the agreement is genuine." Traditional evaluations often only look at whether final outputs converge, whereas PHANTOM-Bench asks: behind this convergence, have the agents' private judgments actually changed?

Collective Rationality in such research generally refers to whether the quality of a group's aggregated judgment is better than, or at least no worse than, the aggregation of individuals' independent judgments. Metrics include the accuracy of group decisions, consistency with ground-truth labels, and the calibration of group-belief estimates. When agents deviate from their own private judgments due to social signals, the group output may be systematically worse than simple majority voting—a "collective rationality failure." The "collective rationality failure rate" in PHANTOM-Bench refers to the proportion of cases where the group's public decision disagrees with the majority opinion of independent private judgments and is of worse quality. This metric shifts the evaluation focus from "whether there is agreement" to "whether the agreement is wise," and is a key quantitative tool for distinguishing genuine consensus from phantom consensus.

Experimental Results: Significant Decline in Decision Quality

The study covered nine closed-source and open-source models, and the results showed a highly consistent pattern. Ambiguous public signals consistently degraded decision quality and could turn originally correct group private judgments into incorrect public collective outcomes.

In the most susceptible settings, public accuracy dropped by as much as 47.8 percentage points, and the collective rationality failure rate reached 47.3%. Meanwhile, models also made numerous errors when reporting the group's private beliefs—failures in the public phase coexisted with distorted belief estimation, indicating that the problem is not only "going along with the crowd" but also incorrect modeling of others' mental states.

Control conditions revealed differences among models: different models vary in their sensitivity to ambiguous signals versus explicit peer signals. Some models are more easily swayed by ambiguous signals, while others are more sensitive to explicit voting instructions. This suggests that phantom consensus is not a single mechanism but a superposition of multiple social-reasoning deficiencies.

Public accuracy refers to the proportion of an agent's final publicly expressed decisions, after observing social signals, that match the correct answer. In contrast, private accuracy is the accuracy of an agent's independent judgment without social-signal interference. The gap between the two directly reflects the degree to which social signals distort decision quality. The "drop of 47.8 percentage points" in the study refers to the decline in public accuracy relative to isolated private judgment accuracy in the most susceptible settings. The design significance of this metric is that it does not rely on a subjective definition of "consensus" but uses verifiable correct answers as an anchor, thereby objectively quantifying the performance loss caused by phantom consensus.

How Much Can Transparency Repair?

A notable finding is that revealing agents' recorded private decisions can repair many failures without exposing the ground-truth answers. In other words, as long as group members can see each other's true private judgments, the illusion of surface consensus largely dissipates.

However, the repair effect varies by model, and not all models recover to the same extent. This result has direct implications for the design of multi-agent systems: when aggregating judgments, making each member's independent judgment public may ensure collective decision quality better than having them repeatedly communicate.

Transparency recovery: the core operation of the experiment is that, after agents make public decisions and estimate the group's private judgments, each member's previously recorded private judgment (without the ground-truth answer) is disclosed to all agents, and then their subsequent decisions are observed to see whether they return to independent judgment. This design simulates a social intervention of "revealing true preferences." Its effectiveness rests on the assumption that phantom consensus stems from misestimating others' private attitudes, and once true information is visible, the misestimation is corrected. But the degree of recovery varies by model, suggesting that some models may have formed strong public-commitment biases under social signals and cannot fully revert even after seeing true private judgments. For system design, this means transparency interventions need to be paired with model selection or training to reliably deliver repair effects.

Implications for Evaluating Multi-Agent Systems

The core argument of this research is a paradigm shift in evaluation. Many current evaluations of multi-agent LLM systems only look at whether final outputs reach agreement, but phantom consensus shows that agreement itself may be a disguise for failure. What truly needs to be measured is whether this agreement reflects independently measured private judgments or is merely surface consensus.

For developers building multi-agent decision systems, this means preserving the recording and visibility of private judgments in the architecture and incorporating the accuracy of group-belief estimation into evaluation metrics. Otherwise, the system may ostensibly "reach consensus" while actually failing collectively in rationality.

The introduction of PHANTOM-Bench provides a reproducible measurement tool for this direction and makes "social reasoning" a capability dimension that needs to be examined separately in multi-agent LLM research.