Why Does Self-Play AI Always Get Crushed by Cheap Units? An RTS Developer's Account of the Dilemma

An independent developer is building a 4v4 game that blends RTS and MOBA, where each team consists of a commander responsible for economy, buildings, and army, along with three heroes handling lane combat. To test game balance without human players, he adopted a pure self-play approach similar to AlphaGo Zero: using the PPO algorithm combined with league-style self-play to train the commander AI. However, the training results revealed a fascinating phenomenon—the AI achieved extremely high win rates against its past selves, yet was repeatedly crushed by human players using cheap tier-1 units.

The Brilliance and Fragility of Self-Play

According to the developer's description on Reddit, the trained commander AI "overwhelmingly won" against historical versions, but once facing real human players, it exposed numerous vulnerabilities. The most typical failure mode was: when human players climbed to tier-2 tech, the AI had almost no defense. Even more puzzling to the developer was that although he expected different training runs to produce diverse builds, each run ultimately converged to the same strategy—mass producing cheap tier-1 units.

This is not an isolated case. In the field of self-play reinforcement learning, this "strategy collapse" phenomenon has long been documented. When AI only fights against its own historical versions, it can get trapped in a local optimum: one strategy is sufficient to beat its past selves, but it is never forced to learn how to respond to completely different tactical styles. The developer's question strikes at the core: is this a known weakness of self-play—the population never discovers counter-strategies, and therefore never learns to defend against them?

Why Self-Play Develops "Specializations"

The core appeal of self-play is that it requires no human data—agents can evolve from scratch through self-competition. The successes of AlphaGo Zero and AlphaStar have made this paradigm highly regarded. But successful cases often rely on enormous computational resources and carefully designed training. In small-scale projects, the flaws of self-play get amplified.

A key issue is insufficient opponent diversity. Although league-style self-play retains historical versions as opponents, if all historical versions share similar weaknesses, then the current strategy only needs a slight improvement in one dimension to win, without needing to develop comprehensive offensive and defensive capabilities. The developer's observation that "each run converges to mass tier-1 units" is exactly this dynamic: tier-1 unit swarm tactics are effective enough in internal competition, so the population stops exploring other possibilities.

Another factor lies in reward design. If the reward signal primarily reflects win rate without incentives for tactical diversity, the AI will naturally choose the most direct path to victory. The developer asked in the post whether AlphaStar-style exploiters, diversity rewards, or action and reward design itself is the problem—this precisely touches on the three key levers of self-play training.

PPO (Proximal Policy Optimization) is one of the most commonly used policy gradient algorithms in current reinforcement learning. It avoids drastic policy changes in a single update by limiting the magnitude of each policy update (implemented with a clipped surrogate objective function), thereby achieving a balance between sample efficiency and training stability. In self-play scenarios, PPO is often used to train agents against a constantly updating opponent pool. However, PPO itself does not contain mechanisms for exploring diversity—it optimizes expected return under a given reward. Therefore, when the reward signal is singular (e.g., based only on win rate), PPO will naturally converge to the most effective strategy in the current opponent pool, without actively exploring other tactical branches. This is one of the algorithmic-level reasons for the developer's observation that "each run converges to the same build order": it's not that PPO is flawed, but that it faithfully executes the objective defined by the reward.

Action masks are a common technique in reinforcement learning for handling illegal actions. In game environments, not all actions are legal at all times (e.g., you can't build when resources are insufficient, or cast skills when cooldowns aren't finished). Action masks set the probabilities of illegal actions to zero and renormalize when the policy network outputs action probabilities, ensuring the agent only samples from legal actions. This mechanism can significantly accelerate training and prevent the agent from wasting exploration on invalid actions. But it can also introduce implicit biases: if the masking logic or the ordering of the action space inadvertently makes certain legal actions more likely to be sampled early on (e.g., cheap units are always buildable, while advanced units require prerequisites), the agent may develop path dependency and favor those "always available" actions over the long term. The developer's commander AI has 23 commands and uses action masks, so it's necessary to check whether the mask design inadvertently reduces the early occurrence probability of actions other than tier-1 unit production.

Exploiters and Diversity: Which Approaches Actually Work

In the AlphaStar project, the DeepMind team introduced "exploiters": in addition to the main agents, they also trained specialized agents to find weaknesses in the main agents. These exploiters serve as "sparring partners," forcing the main agents to patch vulnerabilities. For small-scale projects, a simplified version of this idea may be worth trying: periodically train an opponent specifically targeting the current strategy and add it to the league.

Diversity rewards are another path. By rewarding behavioral differences between strategies and historical strategies (e.g., diversity of build orders, entropy of unit composition), you can encourage the population to explore different tactical spaces. But diversity rewards need to be designed carefully, otherwise they may reward behaviors unrelated to winning.

The developer's action and reward design also deserves scrutiny. The commander AI has 23 commands, the observation space includes fog of war, and action masks are used to block illegal actions. If the action space or reward function inadvertently favors tier-1 unit production, then strategy collapse may stem from design rather than algorithm.

AlphaStar's league-style self-play is the training framework DeepMind designed for StarCraft II, whose core innovation is maintaining an opponent pool containing main agents, past players, and exploiters. Main agents learn by competing against the entire league; past players provide stability anchors to prevent strategy forgetting; exploiters are trained specifically against the main agents' current weaknesses and periodically feed discovered vulnerabilities back to the main agents. In addition, AlphaStar uses "Prioritized Fictitious Self-Play" (PFSP) to sample opponents, prioritizing opponents with a win rate close to 50% against the current agent to maximize learning signals. The key to this entire design is: it artificially creates opponent diversity and actively searches for weak areas in the strategy space. For small-scale projects, fully reproducing this framework is unrealistic, but "periodically training an exploiter targeting the current strategy and adding it to the league" is a simplified version that can be implemented on single-machine resources.

Open Bot Kit: Making Small-Scale Self-Play Reproducible

To explore solutions together with the community, the developer turned the commander seat into an open bot kit and plans to organize a small AI league. According to its GitHub repository, the kit includes: a local practice server that runs real match rules at approximately 100x real-time speed without requiring the game client; a Python library using only the Python standard library; a Gymnasium-style environment with action masks, plus a MaskablePPO example with Stable-Baselines3; rule-based bots and learned-policy opponents for testing. Bots receive the same 23 commands and the same fog-of-war vision as human commanders, and the README lists all observations and actions.

This initiative lowers the barrier to community participation and provides a benchmark for comparing different approaches. The developer stated that if an observation is missing, it can be raised via issue or Discord, and he will add it. This openness is particularly valuable in self-play research—many failure cases cannot be deeply analyzed due to the lack of reproducible environments.

Feasible Paths for Small-Scale Self-Play

For developers with limited resources, fully reproducing AlphaStar-level training is unrealistic. But the experience in the post suggests several pragmatic directions. First, diagnosis takes priority over tuning: before introducing complex mechanisms, analyze the AI's losing matches to confirm whether strategy collapse stems from insufficient opponent diversity, biased reward signals, or action space limitations. Second, introduce external pressure: even without human data, rule-based bots or scripted opponents can serve as fixed sparring partners, providing challenges different from historical versions. Finally, iterate on league design: not only retain historical versions but also "add noise" to or "weaken" historical versions, forcing the current strategy to adapt to different conditions.

Self-play is not a silver bullet, but its potential in small-scale projects is still worth exploring. This developer's experience reminds us that behind AlphaGo Zero-style success lies massive computation and carefully designed curricula. For independent developers, understanding the failure modes of self-play may be more valuable than blindly pursuing "pure self-play."

More technical details and replays can be found in the developer's Reddit post and GitHub repository.