Multi-Agent Debate
A multi-agent collaborative reasoning paradigm in which multiple large language models mutually critique and debate one another through iterative interactions to improve reasoning accuracy and factuality. Proposed by Du et al. around 2023, it has generated broad research interest and scholarly discussion.
Timeline (last 90 days)
In budget-constrained scenarios, sampling a single model multiple times and then voting is more cost- and time-efficient than building a complex multi-agent debate framework.
During the debate process, conversation logs overflow the server-side context window, and fixing this overflow issue can pull the debate-vs-sampling comparison result from −1.8 points back to parity.
The paper proposes that any future debate mechanism claiming to be effective must clear the baseline threshold of budget-matching and contamination checking.
An arXiv paper questions the assumption that the gains from multi-agent debate (MAD) stem from cognitive diversity, arguing that its essence is an ensemble sampling effect rather than a unique advantage brought by cognitive diversity.
The experiment covers 23 models from 11 vendor families, 5 tasks, and accumulated over 5,500 debate and control runs in total.
The study systematically introduces diversity along three dimensions: personas, sampling temperature, and model identity.
On tasks with room for improvement, debate brings a 3 to 7 percentage point gain over single-agent reasoning.
After budget alignment, debate can only match or even underperform self-consistency sampling, at the cost of 1.6× wall-clock time and 3.4× token cost.
Mixed teams composed of different models lost to directly applying majority voting across the same set of models, with accuracy depending more on individual member capability than team heterogeneity.
Almost all of debate's gains come from the first round of answer exchange, with subsequent rounds of debate yielding no substantial improvement.
2 more timeline events
All Facts (12)
In budget-constrained scenarios, sampling a single model multiple times and then voting is more cost- and time-efficient than building a complex multi-agent debate framework.
50%UnverifiedDuring the debate process, conversation logs overflow the server-side context window, and fixing this overflow issue can pull the debate-vs-sampling comparison result from −1.8 points back to parity.
50%UnverifiedThe paper proposes that any future debate mechanism claiming to be effective must clear the baseline threshold of budget-matching and contamination checking.
50%UnverifiedAn arXiv paper questions the assumption that the gains from multi-agent debate (MAD) stem from cognitive diversity, arguing that its essence is an ensemble sampling effect rather than a unique advantage brought by cognitive diversity.
50%UnverifiedThe experiment covers 23 models from 11 vendor families, 5 tasks, and accumulated over 5,500 debate and control runs in total.
50%UnverifiedThe study systematically introduces diversity along three dimensions: personas, sampling temperature, and model identity.
50%UnverifiedOn tasks with room for improvement, debate brings a 3 to 7 percentage point gain over single-agent reasoning.
50%UnverifiedAfter budget alignment, debate can only match or even underperform self-consistency sampling, at the cost of 1.6× wall-clock time and 3.4× token cost.
50%UnverifiedMixed teams composed of different models lost to directly applying majority voting across the same set of models, with accuracy depending more on individual member capability than team heterogeneity.
50%UnverifiedAlmost all of debate's gains come from the first round of answer exchange, with subsequent rounds of debate yielding no substantial improvement.
50%UnverifiedThe Proposer/Critic dual-agent architecture originates from the Multi-Agent Debate research paradigm and is used to mitigate self-confirmation bias in a single LLM.
50%UnverifiedThe Multi-Agent Debate (MAD) paradigm attracted widespread attention around 2023, but multiple follow-up studies found its advantages difficult to reproduce consistently.
50%