TandemBench
TandemBench is a benchmark for evaluating multi-agent collaborative decision-making models. Through cooking-scenario problems, it tests a model's ability to select the optimal team plan and to judge the completeness of options.
Timeline (last 90 days)
TandemBench is a benchmark specifically designed to evaluate multi-agent collaborative decision-making models, containing 1,200 problems based on cooking scenarios.
TandemBench's model task is to select the optimal plan from candidate team plans, and in paired variants (retaining or removing the optimal plan) to judge whether a better arrangement exists.
TandemBench's core innovation lies in testing not only the model's ability to choose among given options, but also its metacognitive ability to judge 'whether an optimal solution already exists'.
TandemBench's reference answers are established through execution and exact search, ensuring the reliability of the evaluation.
Native decision models achieve 62.50% accuracy on the TandemBench benchmark, but are close to random on optimality verification.
The study evaluated 40 native decision models and compared them with 10 general-purpose language models.
In the optimality verification task, native decision models perform close to the random level.
The paired test revealed a systematic bias: some models claim that a better alternative exists regardless of whether the optimal plan has already been provided.
Decision accuracy on team plans alone may mask systematic errors in the model's judgment of 'whether the given options contain the optimal arrangement'.
TandemBench's dataset and evaluation code are publicly available on GitHub (https://github.com/lililibaohang/TandemBench).
All Facts (10)
TandemBench is a benchmark specifically designed to evaluate multi-agent collaborative decision-making models, containing 1,200 problems based on cooking scenarios.
50%UnverifiedTandemBench's model task is to select the optimal plan from candidate team plans, and in paired variants (retaining or removing the optimal plan) to judge whether a better arrangement exists.
50%UnverifiedTandemBench's core innovation lies in testing not only the model's ability to choose among given options, but also its metacognitive ability to judge 'whether an optimal solution already exists'.
50%UnverifiedTandemBench's reference answers are established through execution and exact search, ensuring the reliability of the evaluation.
50%UnverifiedNative decision models achieve 62.50% accuracy on the TandemBench benchmark, but are close to random on optimality verification.
50%UnverifiedThe study evaluated 40 native decision models and compared them with 10 general-purpose language models.
50%UnverifiedIn the optimality verification task, native decision models perform close to the random level.
50%UnverifiedThe paired test revealed a systematic bias: some models claim that a better alternative exists regardless of whether the optimal plan has already been provided.
50%UnverifiedDecision accuracy on team plans alone may mask systematic errors in the model's judgment of 'whether the given options contain the optimal arrangement'.
50%UnverifiedTandemBench's dataset and evaluation code are publicly available on GitHub (https://github.com/lililibaohang/TandemBench).
50%