Six AI Models Play Chess via CLI, No Checkmates Reveal the Truth About Automation Reliability
Six large models played 12 CLI chess games with zero checkmates, actually measuring call stability under hard time limits rather than chess strength.
The developer organized a four-round Swiss-system chess tournament on SpaceApp, connecting six mainstream large models via Claude Code, Codex, and Gemini CLI, with a 120-second limit per move and a three-strikes loss rule. 11 of 12 games ended early by strikes, and the only full 60-move game was a material draw—no checkmates. Opus 5.5 and Gemini 4 Argon tied for first with 3.5 points and zero strikes. The author notes the sample is too small and outcomes were decided by strikeouts, so the experiment truly measured reliability of legal, timely CLI responses under hard deadlines—the core metric for long-running unattended agents.
A Chess Experiment with No Checkmates
When six mainstream large models battled it out on the board through their respective command-line interfaces (CLIs), the result turned out to be a game with no winners. The developer organized a 4-round Swiss-system chess tournament on their self-hosted platform SpaceApp. Of the 12 games, 11 ended early due to the "three-strikes" rule, and the only game that went the full 60 moves ended in a draw by material adjudication. What this experiment truly tested was not chess strength, but the reliability of CLI calls under strict time limits.
The Lineup and Rule Design
The six "players" covered current frontier model families: Opus 5.5 and Fable 5.1 accessed via Claude Code, GPT-6-Astra and GPT-6.1-Sol via Codex, and Gemini 4 Argon (High) and Gemini 3.8 Flash (High) via Gemini CLI. All models ran at the xhigh reasoning tier, with a 120-second limit per move, returning one move per single CLI call.
The rules introduced a strike-like penalty mechanism: timeouts, provider errors, or a second illegal response would trigger a random legal move and record one "strike," with three cumulative strikes resulting in a loss. If no winner emerged after 60 moves, the result was adjudicated by material.
Swiss system is a tournament format commonly used in chess and other board game competitions: rather than playing out every possible pairing as in a round-robin, players with similar scores are paired against each other each round, typically over far fewer rounds than there are participants, thereby approximating a ranking within a limited number of games. Its characteristics are that every player competes each round and stronger-vs-stronger matchups increase as rounds progress, but with few games the ranking is easily affected by pairing randomness. Part of the reason the author considers the 12-game sample of limited reference value is precisely that the Swiss system's pairing and ranking stability is inherently weak with small samples.
The "three-strikes" rule borrows from baseball terminology; here it means that three cumulative violations (timeout, provider error, or a second illegal response) result in a loss. The mechanism's intent is to convert "sporadic failures" into quantifiable match results, making stability an explicit scoring dimension; but the side effect is that many games are terminated early, with wins and losses reflecting the robustness of the call chain rather than chess strength—which is why the author emphasizes that this tournament cannot be used to evaluate chess ability.
Results: Stability Is More Striking Than Chess Strength
The final rankings showed clear stratification. Opus 5.5 and Gemini 4 Argon tied for first with 3.5 points each, with zero strikes throughout and average times per move of 13 seconds and 23 seconds respectively. GPT-6-Astra scored 2 points but recorded 2 strikes, while Fable 5.1, Gemini 3.8 Flash, and GPT-6.1-Sol each scored 1 point and accumulated 3 strikes each.
Interestingly, only Opus 5.5 and Gemini 4 Argon never needed the system to substitute a "safe move." Although GPT-6-Astra won two games, it was also twice pushed into random moves due to errors. The 11 games ending in three strikes shows that most models struggle to maintain stable output under high-frequency, continuous calls.
What the Experiment Revealed and What It Hid
The author candidly admits that the 12-game sample is extremely small and that a tournament decided by strikeouts has almost no reference value for evaluating chess strength. What was actually measured was: with a hard deadline on every move, can the model continuously produce legal, timely responses without timeouts, provider errors, or illegal moves?
This metric maps directly to long-running autonomous inference scenarios in production. When an agent needs to run unattended for hours, a high score on a single-prompt leaderboard does not guarantee reliability at every step. The CLI call chain adds extra failure points such as network latency, authentication, and streaming parsing, so model-level stability gets amplified and tested.
The CLI call chain refers to the complete path from a program initiating a command-line call to the model returning a result, typically including process startup and argument parsing, network requests and retries, API authentication and quota checks, chunked parsing and assembly of streaming responses, and timeout and error-code handling. Compared with calling an HTTP API directly, the CLI wrapper adds a layer of process and standard input/output, and any jitter in any link can cause a timeout or parsing failure. This is the technical meaning of "model-level stability gets amplified and tested" in the article: even if the model's own output quality is stable, chain failures will be counted as strikes in the form of illegal responses or timeouts.
SpaceApp: A Workbench for Running Multiple CLIs Side by Side
SpaceApp is a self-hosted workspace built by the author, supporting Node.js 20.11 or higher on Linux, macOS, and Windows 11. The install command is npx --yes run-spaceapp@latest install, and the code is released as source-available under the Business Source License 1.1.
The author extends an invitation to the community: if you have done similar tests, they would like to know which models perform more stably over long runs, and how the rules should be adjusted for a fairer second round.
Business Source License 1.1 (BSL) is a "source-available" license rather than a strictly open-source license. It allows users to view, modify, and use the source code within certain bounds, but restricts providing the software as a commercial hosted service to third parties; each BSL license specifies a "change date," after which it automatically converts to a designated open-source license (commonly Apache 2.0 or GPL). Therefore, the wording "released as source-available" in the article is accurate: SpaceApp's code is obtainable and self-hostable, but commercial redistribution must comply with its additional restrictions.
One-line Summary
Six large models played 12 CLI chess games with zero checkmates, actually measuring call stability under hard time limits rather than chess strength.
Paragraph Summary
The developer organized a four-round Swiss-system chess tournament on the self-hosted platform SpaceApp. Six mainstream large models were connected via Claude Code, Codex, and Gemini CLI, with a 120-second limit per move and a "three-strikes" rule where three cumulative violations result in a loss. Of the 12 games, 11 ended early due to three strikes, and the only game that went the full 60 moves ended in a draw by material adjudication—with no checkmate throughout. Opus 5.5 and Gemini 4 Argon tied for first with 3.5 points each and zero strikes, averaging 13 and 23 seconds per move; all other models recorded strikes. The author notes that the sample is too small and outcomes were decided by strikeouts, so the experiment truly measured the reliability of models in continuously producing legal, timely CLI responses under hard deadlines—precisely the core metric for long-running unattended agent scenarios.
Related articles
Kimi K3.1 Open-Weight Model Hands-On: 3 Trillion Parameters Take On Closed-Source Giants
Kimi K3.1 open-weight model hands-on: 3T-parameter MoE, 1M-token context, stunning 3D rendering but weak spatial reasoning and factual accuracy. Benchmarks near Opus 5.5 at $2.11/task.
Anthropic's New Guide: Most AI Agents Don't Need Subagents — a Single Main Loop Is the Right Answer
Anthropic's new Agent architecture guide argues most use cases should return to a single-model, single-context open-ended tool loop, splitting only on measurable degradation. Single loops are ~2x faster, tool selection degrades past 30-50 definitions, and Claude Code's system prompt was cut by over 80%.
Reddit User Reverse-Engineers OpenAI's System Prompt: Full Exposure of AI Behavioral Guidelines
Reddit user reverse-engineers OpenAI's system prompt via multi-model collaboration, exposing 6.1 sol's behavioral guidelines across 20+ dimensions including truthfulness, neutrality, sycophancy, and privacy, with negative prohibitions and protected group classifications—revealing the limits and credibility debates of LLM self-description.