XiangqiBench: Chinese Chess Exposes the Closed-Loop Gaps in LLM Agents

Finding the Right Move Isn't the Same as Winning the Game

The way we evaluate large language model (LLM) agents is due for a serious rethink. Traditional static benchmarks award points the moment a model identifies the "correct next move," but real-world agents must carry a complete plan through to a verifiable outcome — all while an opponent actively fights back. In other words, spotting a brilliant move is just the beginning; executing an entire game to victory is the real test of capability.

A new paper on arXiv introduces XiangqiBench — an executable benchmark built around Chinese Chess (Xiangqi). Its core argument is straightforward: agent evaluation should measure closed-loop outcomes and report reliability alongside coverage, rather than focusing solely on single-step accuracy.

How XiangqiBench Is Designed

The research team constructed 119 tactical endgame positions, all featuring forced mates verified either by an engine or through checks-only search. The LLM agent's task is to actually deliver checkmate against an engine opponent that actively defends and counterattacks.

To keep the evaluation process controllable and observable, the benchmark uses an interactive REPL interface that clearly separates three types of operations:

• Real moves: actions committed to the board that are irreversible;
• State queries: reading current board information;
• Forward simulation: the agent's own lookahead reasoning about future moves.

Using this interface, the researchers recorded 8,568 multi-turn interaction trajectories from 12 frontier LLMs under two observation protocols. This scale of trajectory data allows a fine-grained breakdown of exactly where models fail in closed-loop gameplay.

REPL (Read-Eval-Print Loop) is an interactive programming environment originally used for immediate execution and debugging of code. XiangqiBench borrows this concept to describe the multi-turn interaction loop between an agent and the board environment: the agent submits a command (a move, a query, or a simulation), the environment immediately executes it and returns a result, and the agent decides its next action accordingly. The key feature is state persistence — the board changes irreversibly with each real move, and the agent cannot undo or reset. Unlike static, one-shot input-output evaluations, the REPL model inherently demands context tracking and dynamic adaptation, making it a much more faithful simulation of closed-loop decision-making in real deployments.

Three Illusions of Capability

The most valuable contribution of this research is identifying three categories of signals that look like capability but systematically overestimate closed-loop success. Each corresponds to a distinct crack in an agent's real-world performance.

Conversion Gap

In Sighted (visible board state) trials, models played the same first move as the reference solution 26.1% of the time — which sounds reasonably impressive. But in those same trials, only 13.9% ended in an actual win. The model can "recognize" the right opening move, yet is far from converting that advantage into victory. Knowing the first step is not the same as playing the whole game.

Consistency Gap

The best-performing model achieved 38.7% on pass@3 (winning at least once in three attempts), but only 5.9% on pass^3 (winning all three). Of the 46 positions it had ever won, only 7 could be won three times in a row. This exposes severe instability: the same position leads to a win sometimes and a loss other times, suggesting victories owe more to luck than to reproducible skill. For agents that need to be reliably deployed, this inconsistency is a critical flaw.

pass@k and pass^k are two different metrics for evaluating the reliability of stochastic systems, and the distinction directly affects how we judge model capability. pass@k represents the probability of succeeding at least once in k independent attempts — it reflects the model's potential ceiling or coverage. pass^k requires all k attempts to succeed, measuring consistency and reproducibility. For agents deployed in production (e.g., automated code generation, robotic control, financial decision-making), a high pass@k paired with a low pass^k means the system's behavior depends on luck rather than skill — users cannot predict whether the next call will succeed. XiangqiBench reports both metrics precisely to expose the vast gulf between "occasionally winning" and "winning every time."

Simulation Gap

This is the most direct test of self-reasoning ability. The study found that 32.3% of accepted simulation calls terminated on an illegal move, and in comparable cases, 49.3% of the time the actual defensive response from the opponent differed from the line the agent had simulated.

The root cause: an agent's self-authored rollouts may verify move legality, but they cannot anticipate the opponent's true responses. The model mentally narrates a winning line, but the real opponent plays nothing like what was imagined. This is precisely the most fundamental difference between static, single-player evaluation and genuine adversarial environments.

Forced Mate is a chess term referring to a position where, even under optimal defense, the attacking side can guarantee checkmate within a fixed number of moves. The defining feature of such endgames is that the outcome is unique and engine-verifiable, regardless of how the defender responds. This makes forced-mate positions ideal for testing an agent's planning ability: there is a ground-truth answer (the engine's optimal line), yet execution requires responding dynamically to the opponent's counterplay — eliminating the possibility of scoring through guessing or probability, and directly testing whether a model can translate correct tactical understanding into an end-to-end action sequence.

Why This Research Matters

XiangqiBench's significance goes beyond Chinese Chess. It offers a paradigm for closed-loop evaluation. A large number of current agent benchmarks remain stuck at static Q&A or single-step decisions, which makes models appear highly capable. But the moment they are placed in an environment with an opponent, evolving state, and irreversible actions, many surface-level abilities quickly collapse.

The paper's central conclusion is clear and compelling:

Finding the move is not the same as winning the game — agent evaluation should score closed-loop outcomes and report reliability alongside coverage.

Chinese Chess is a particularly effective test platform: the rules are unambiguous, victory conditions are deterministic, forced-mate endgames have verified ground-truth solutions, and yet the game still involves dynamic adversarial play. This makes it an ideal arena for measuring both planning ability and execution reliability.

Implications for Agent Development

For teams building LLM agents, this research offers several practical takeaways:

• Don't be misled by single-step accuracy: high coverage can mask low conversion rates; end-to-end success is the real metric.
• Take reliability evaluation seriously: pass@k can be inflated; pass^k (winning consistently across multiple attempts) better reflects deployability.
• Be wary of blind spots in self-simulation: if an agent's internal lookahead fails to model opponent behavior, its planning will diverge from reality.

As agents move from "answering questions" to "executing long-horizon tasks," evaluation approaches that emphasize closed-loop completion, adversarial play, and reproducibility — like XiangqiBench — are likely to become an increasingly important direction for the next generation of benchmarking.