Reshaping Reward Signals with Replay: Breaking the Scoreboard Trap in Long-Horizon Strategy Game AI

Training language model agents to play long-horizon strategy games has always been a thorny problem: deterministic feedback is sparse and often only appears when the entire game ends. In-game real-time scores may seem to provide dense signals, but they don't necessarily point to ultimate victory—in eight strategy games, in as many as 8% to 68% of position pairs, the higher-scoring position actually led to a worse outcome. The researchers call this phenomenon "scoreboard traps."

Scoreboard Traps: When a High Score Means a Bad Ending

In open-source turn-based empire-building games like Freeciv, players must manage units, cities, and diplomacy, with a single game often lasting hundreds of turns. Such environments are naturally suited to testing long-horizon decision-making, but they also expose the contradictions of reward design more thoroughly: if real-time scores are used as rewards, agents may learn to "farm points" rather than "win games."

The quantitative evidence presented in the paper is quite convincing—in tests across eight strategy games, a considerable proportion of high-scoring configurations were not the optimal path. This means that treating the scoreboard directly as a learning signal will push agents to optimize in the wrong direction.

Freeciv is an open-source derivative of the Civilization series, featuring turn-based 4X gameplay (eXplore, eXpand, eXploit, eXterminate). Players start from a single settlement and expand their influence through city production, technological research, diplomacy, and war. Victory conditions include conquering all opponents and being the first to complete a spaceship to Alpha Centauri. A standard match typically involves hundreds of turns, and the number of units and cities to manage each turn grows as the map expands, creating an enormous decision space with clearly delayed rewards. For this reason, Freeciv is often used as a testbed for long-horizon planning and multi-agent competition; its open-source nature also makes it easy for researchers to modify rules, integrate interfaces, and reproduce experiments.

CivHarness: Letting Games "Reload and Replay"

To obtain reliable long-term feedback, the authors built CivHarness—an open system that can save the current game state, branch at different decision points, and replay into future turns, thereby obtaining measurable long-horizon feedback from saved positions.

Its value lies in turning "hindsight" into a training resource: rather than waiting until a game ends to know whether a decision was right or wrong, it is better to actively branch and replay, using future outcomes to evaluate current decisions in reverse. This replay-based signal is closer to the true outcome than real-time scores.

CivTelescope: Predicting Which Position Will Win

Using CivHarness, the researchers trained CivTelescope—an LLM Judge that determines which of two game positions will lead to a better final outcome.

On scoreboard-trap pairs, CivTelescope's accuracy is 1.3 to 2.0 times that of off-the-shelf language models, and it generalizes to new starting positions, later turns, and unseen rules. This shows that the model has learned something about the long-term value of positions rather than rote memorization.

An LLM Judge refers to using a pretrained language model to perform pairwise comparisons or scoring of candidate solutions, positions, or generated content, rather than directly outputting actions. A common approach is to present two options to the model and ask which one is better, thereby converting open-ended evaluation into a binary classification judgment. Compared with human annotation, LLM Judges are low-cost and can be called in batches; however, they may also be affected by biases such as position preference and sensitivity to wording, so they need to be calibrated against real outcomes. CivTelescope is an instance of this paradigm applied to game position evaluation: it does not play the game directly but compares the probability that two states will lead to victory in the future, and its output can serve as a training signal for a reward model.

CivMarshmallow: Turning the Predictor into a Dense Reward

With a reliable judge in hand, the next step is to use it as a dense reward to train a gameplay agent. This recipe is named CivMarshmallow.

In the midgame phase, where scoreboard traps occur frequently, CivMarshmallow achieved an average training-phase gain of 8.94 points, 77% higher than the best baseline (terminal reward). In complete 120-turn games, it uses a hybrid reward: at the start, it relies on the scoreboard or terminal reward, and as the game progresses it gradually switches to CivTelescope. Among the various rewards compared, this hybrid reward scored the highest.

Why Hybrid Rewards Make More Sense

Early positions contain little information, and although terminal signals are reliable, they are too sparse; the midgame has many traps and requires dense guidance from the judge; by the late game, the outcome becomes clearer, so it returns to real results. This "coarse first, fine later" scheduling precisely matches the rhythm in which information gradually unfolds in long-horizon decision-making.

Sparse rewards and dense rewards are a fundamental trade-off in reinforcement learning. Sparse rewards are given only at episode termination (such as the final win/loss outcome); the signal is reliable, but credit assignment is difficult, and the agent can hardly tell which step led to victory. Dense rewards provide feedback at each step or each intermediate state, enabling faster learning, but if the proxy metric is inconsistent with the true objective, they can induce reward hacking—the agent optimizes the metric rather than the objective itself. Scoreboard traps are essentially a typical case of distorted dense rewards. The idea behind hybrid rewards is staged scheduling: tolerate sparsity when information is insufficient, introduce calibrated dense signals in trap-prone areas, and return to real feedback as the outcome approaches, thereby balancing learning efficiency and signal credibility.

Implications for Long-Horizon Agent Training

The core contribution of this work is not just a particular model, but a methodology: when the built-in signals provided by the environment are unreliable, use replayable simulation to construct more credible rewards.

For teams working on game AI or multi-step decision-making, CivHarness-style branch-and-replay infrastructure is worth learning from—it upgrades evaluation from "looking at scores" to "looking at the future." And the existence of scoreboard traps reminds us that any proxy metric needs to be calibrated against real outcomes; otherwise, the harder we optimize, the further we drift.

One-line Summary

Researchers reveal scoreboard traps in strategy games and use replayable simulation to construct credible rewards for training long-horizon agents.

Paragraph Summary

The article introduces a study on training agents for long-horizon strategy games. The authors point out that real-time game scores are unreliable as dense rewards: in eight strategy games, in 8% to 68% of positions, high scores instead led to worse outcomes—the "scoreboard trap." To obtain credible long-term feedback, the researchers built CivHarness, which supports saving states, branching decisions, and replaying future turns; with it, they trained the LLM Judge model CivTelescope to determine which of two positions is more likely to win. Its accuracy on trap pairs is 1.3 to 2.0 times that of off-the-shelf models, and it generalizes to new starting positions and unseen rules. Finally, CivMarshmallow uses CivTelescope as a dense reward, achieving an average gain of 8.94 points in the midgame, 77% higher than the terminal-reward baseline; in complete games, it adopts a hybrid reward that switches with progress and achieves the highest overall score.