XiangqiBench
An executable benchmark for LLM agents using Chinese chess (Xiangqi) as the evaluation domain. It assesses closed-loop decision-making ability through checkmate endgame positions and measures both coverage and reliability as primary metrics.
Timeline (last 90 days)
XiangqiBench is an executable benchmark using Chinese chess as its vehicle, published on arXiv, and argues that agent evaluation should measure closed-loop outcomes and report both coverage and reliability.
The XiangqiBench research team constructed 119 tactical endgame positions with forced checkmates, verified through an engine or pure check search.
XiangqiBench designed an interactive REPL interface that distinguishes among three types of operations: actual moves, state queries, and forward simulation.
Researchers recorded 8,568 multi-turn interaction trajectories from 12 frontier LLMs under two observation protocols.
In the Sighted (visible board) trials, models made a first move consistent with the reference answer in 26.1% of cases, but ultimately won the game in only 13.9% of cases.
The best-performing model achieved 38.7% on pass@3, but only 5.9% on pass^3.
Of the 46 positions that models had previously won, only 7 positions achieved three consecutive wins.
32.3% of accepted simulation calls ended on an illegal move.
In comparable cases, the actual defensive response differed from the line the agent had previously simulated in 49.3% of instances.
The agent's own simulated reasoning can check whether a move is legal but cannot predict the opponent's actual response, resulting in a simulation gap.
2 more timeline events
All Facts (12)
XiangqiBench is an executable benchmark using Chinese chess as its vehicle, published on arXiv, and argues that agent evaluation should measure closed-loop outcomes and report both coverage and reliability.
50%UnverifiedThe XiangqiBench research team constructed 119 tactical endgame positions with forced checkmates, verified through an engine or pure check search.
50%UnverifiedXiangqiBench designed an interactive REPL interface that distinguishes among three types of operations: actual moves, state queries, and forward simulation.
50%UnverifiedResearchers recorded 8,568 multi-turn interaction trajectories from 12 frontier LLMs under two observation protocols.
50%UnverifiedIn the Sighted (visible board) trials, models made a first move consistent with the reference answer in 26.1% of cases, but ultimately won the game in only 13.9% of cases.
50%UnverifiedThe best-performing model achieved 38.7% on pass@3, but only 5.9% on pass^3.
50%UnverifiedOf the 46 positions that models had previously won, only 7 positions achieved three consecutive wins.
50%Unverified32.3% of accepted simulation calls ended on an illegal move.
50%UnverifiedIn comparable cases, the actual defensive response differed from the line the agent had previously simulated in 49.3% of instances.
50%UnverifiedThe agent's own simulated reasoning can check whether a move is legal but cannot predict the opponent's actual response, resulting in a simulation gap.
50%UnverifiedThe core conclusion of the study is that agent evaluation should score closed-loop outcomes and report both reliability and coverage.
50%UnverifiedEvaluation methodologies that emphasize closed-loop, adversarial, and reproducibility properties—such as XiangqiBench—are likely to become an important direction for the next stage of agent evaluation frameworks.
50%