[KongchangAI]
· 1 min read· 744 words

TandemBench: A Benchmark for Multi-Agent Collaborative Decision-Making Models

TandemBench: A Benchmark for Multi-Agent Collaborative Decision-Making Models

TandemBench reveals that behind 62.5% team plan selection accuracy, decision models are near random on optimality verification with systematic bias.

TandemBench is a multi-agent collaborative decision benchmark with 1,200 cooking scenarios, requiring models to select the optimal team plan and judge whether given options already contain the optimal solution. It evaluated 40 native decision models and 10 general-purpose language models; the best native model reached 62.50% decision accuracy but was near random on optimality verification. Paired testing reveals systematic bias: some models claim a better alternative exists regardless of whether the optimal plan is provided, showing decision accuracy alone can mask errors in judging option completeness.

How do decision models choose team plans in multi-agent collaboration? The TandemBench benchmark uses 1,200 cooking-scenario questions to reveal a significant gap in current decision models' performance on team plan selection and optimality verification.

What is TandemBench?

TandemBench is a benchmark specifically designed to evaluate multi-agent collaborative decision-making models, containing 1,200 questions based on cooking scenarios. In these scenarios, coordination choices directly affect whether the goal can be completed. Models must select the optimal plan from candidate team plans and, in paired variants (with or without the optimal plan retained), determine whether a better arrangement exists.

The benchmark's core innovation is that it not only tests a model's ability to choose among given options, but also examines its metacognitive ability to judge "whether an optimal solution already exists." Reference answers are established through execution and exact search, ensuring evaluation reliability.

TandemBench Benchmark Overview

Native decision models refer to models specifically designed for sequential decision-making or planning tasks (such as planners based on reinforcement learning or search), as distinct from general-purpose language models (such as the GPT series). They are typically trained directly on the state-action space and output executable plans or action sequences rather than natural language. TandemBench compares native decision models with general-purpose language models to assess performance differences between architectures on collaborative planning tasks. Native decision models achieve 62.50% accuracy on the benchmark, but are near random on optimality verification, suggesting they may overfit to selection rather than metacognitive judgment.

Evaluation Design and Findings

The study evaluated 40 native decision models and compared them with 10 general-purpose language models. Results show that the best native decision model achieves 62.50% decision accuracy, which appears good, but deeper analysis reveals problems.

On the optimality verification task, native decision models perform near random level. More concerning, paired testing reveals a systematic bias: some models claim a better alternative exists regardless of whether the optimal plan was already provided. This indicates that team plan decision accuracy alone may mask systematic errors in judging "whether the given options contain the optimal arrangement."

According to data from the original paper, these findings point to a key conclusion: the reliability of decision models in complex collaborative scenarios still requires substantial improvement, especially on tasks requiring judgment of option completeness.

Paired testing is an experimental design: two variants are constructed for the same scenario, one containing the optimal plan and the other removing it, and the model is asked to judge "whether a better plan exists." By comparing the model's answers across the two variants, it is possible to detect whether the model truly evaluates the completeness of options rather than relying only on surface features. TandemBench uses paired testing to find that some models claim a better alternative exists regardless of whether the optimal plan is present, revealing a systematic bias in optimality verification—they may tend toward "over-optimization" or fail to recognize already-optimal states.

Implications for Multi-Agent Collaboration

TandemBench's evaluation results have cautionary significance for the practical deployment of multi-agent systems. In tasks such as routing and action selection, a decision model's selection accuracy is certainly important, but if the model cannot correctly judge whether the given options already contain the optimal solution, the system may fall into unnecessary "optimization loops" or miss the truly optimal plan.

The benchmark's dataset and evaluation code are publicly available on GitHub (https://github.com/lililibaohang/TandemBench), providing a reproducible testing platform for future research. For researchers and developers focused on multi-agent collaboration and decision model evaluation, TandemBench offers a more comprehensive perspective: it is not enough to see whether the model chooses correctly; we must also see whether it knows that it chose correctly.

One-line Summary

TandemBench reveals that behind 62.5% team plan selection accuracy, decision models are near random on optimality verification and exhibit systematic bias.

Paragraph Summary

TandemBench is a multi-agent collaborative decision benchmark containing 1,200 cooking scenarios, requiring models to select the optimal plan from candidate team plans and judge whether the given options already contain the optimal solution. The study evaluated 40 native decision models and 10 general-purpose language models; the best native model achieved 62.50% decision accuracy but was near random on optimality verification tasks. Paired testing further reveals systematic bias: some models claim a better alternative exists regardless of whether the optimal plan is provided. This shows that decision accuracy alone may mask errors in judging option completeness. The benchmark dataset and code are publicly available, offering a more comprehensive perspective for multi-agent collaboration evaluation.

#TandemBench#multi-agent collaboration#decision models#optimality verification#benchmark#team plan selection
Share:

Related articles