Training a 4-Player Squad in a 2D Battle Royale with MAPPO: 40 Seconds of Gameplay and Lessons Learned

When Battle Royale Becomes a Multi-Agent Coordination Experiment

A developer on Reddit shared his experiment training a 4-player squad to play survev using MAPPO (Multi-Agent PPO). survev is an open-source surviv.io clone where every character in the match is an AI-controlled bot. He released a 40-second gameplay clip and candidly listed the pitfalls that "bit" him during training.

What's interesting here isn't that "AI can play games," but that it compresses some of the hardest problems in multi-agent reinforcement learning—partial observability, credit assignment, and hierarchical action spaces—into a lightweight, reproducible experimental setup.

Architecture Breakdown: The Actor Can't See the God's-Eye View

The most critical constraint in this system is information isolation. All four squad members share the same actor network, with agent id as input to distinguish identities. The actor's available information is strictly equivalent to what a real game client sees: roughly 1.4k-dimensional features, two 24x24 local grids, and a minimap, processed through a small CNN; set encoding over up to 12 visible enemies, then fed into a 256-dimensional GRU for temporal memory.

This design directly determines whether the model can be deployed. Because the actor's input matches the real client, the same network can be packaged as a browser extension and run directly, without needing a server backdoor. Privileged state—such as global enemy positions and full health information—is only fed to the centralized critic to reduce variance during training. The critic is the training-time "cheat," the actor is not.

Hierarchical Action Space: Think Intent First, Then Choose Target

Actions aren't structured as one flat list but split into three layers: the policy first selects an intent (loot, explore, find cover, peek, retreat, regroup, push, enter zone, etc.), then selects a target, and finally selects what to use (gun, melee, grenade, smoke, heal).

The benefit is that the action space is structured, exploration is more efficient, and the policy is more interpretable—you know what the bot "wants to do," not just "what key it pressed." The cost is that each layer needs its own learning signal, and hierarchical credit assignment is harder to tune than a single layer.

MAPPO (Multi-Agent PPO) is an extension of the PPO algorithm to multi-agent settings. PPO itself belongs to the actor-critic family and uses a clipped surrogate objective to prevent overly large policy updates. MAPPO's key modification is adopting the CTDE (Centralized Training, Decentralized Execution) paradigm: each agent has its own actor, but during training they share a centralized critic that can observe the global state. This gives the critic more information when estimating value, thereby reducing gradient variance, while the actor still relies only on local observations at deployment. MAPPO achieves performance close to or exceeding off-policy methods with a relatively simple implementation on benchmarks like StarCraft micromanagement (SMAC), making it a common baseline choice for multi-agent coordination tasks.

Partial observability means a single agent cannot know the full state of the environment and can only see local information. In multi-agent systems, each agent must deal not only with environmental uncertainty but also with uncertainty about other agents' behavior and intent, which invalidates the single-agent MDP assumption and requires modeling as a Dec-POMDP. Recurrent networks like GRUs are often used to maintain temporal memory under partial observability, compensating for the lack of information in a single frame.

Credit assignment refers to determining how much each agent's action at each step contributed to the final outcome when the team shares a single reward signal. It is one of the core challenges in multi-agent reinforcement learning: the more shared the reward, the harder it is to distinguish individual contributions, and the more prone it is to free-riding. Common countermeasures include centralized critics, counterfactual baselines (COMA), and explicit reward decomposition.

Hierarchical reinforcement learning splits the policy into multiple layers, where higher layers output abstract intents or subgoals and lower layers handle concrete execution. It is mainly used to alleviate the problems of large action spaces and sparse exploration in long-horizon tasks, and also facilitates injecting human priors. Hierarchical structures are typically trained with the options framework or hierarchical policy gradients, and layers can share or independently update parameters. The cost is that each layer needs its own credit assignment signal, and the time span between a high-level intent and its eventual consequence is longer, making training less stable than single-layer policies.

How Rewards Are Set: One Shared Team Signal

The squad shares a team reward rather than each agent receiving individual scores. Reward terms include damage dealt and taken, knockdowns and kills, final ranking, plus several small team coordination terms: focus fire, crossfire, and revive.

Shared rewards are a direct approach to solving multi-agent credit assignment and typically work well with a centralized critic. But they also bring the classic free-rider risk—if one member does nothing, they still get a share of the team reward. The author uses terms like focus fire and crossfire to guide cooperative behavior, essentially explicitly injecting coordination priors into the reward rather than hoping they emerge on their own.

Reward shaping refers to designing additional auxiliary reward terms beyond the sparse original reward to guide the policy toward desired behavior more quickly. Coordination terms like focus fire, crossfire, and revive are typical shaping signals—they translate the abstract goal of "cooperation" into accumulable numeric values. Shaping can significantly accelerate convergence but also introduces risk: if auxiliary rewards are misaligned with the ultimate objective, the policy may learn to farm points without actually improving win rate. In practice, shaping terms are often multiplied by a small weight and gradually decayed in later stages of training.

What Bit Him

The author did not release the code or full details, for a very practical reason: if such a bot were released, it could be used as a cheating tool on public servers. This concern itself is worth noting—a research project trained on a local, unmodified server is only one step away from a usable cheat.

He was willing to answer high-level questions but explicitly withheld implementation details. From what was shared, several implicit challenges can be inferred: the hierarchical action space makes training more complex; the combination of GRU, CNN, and set encoder has a non-trivial parameter count and high training cost; cooperative behavior under shared rewards requires manually designed reward terms to guide it.

What This Experiment Means

survev provides a clean testbed: open-source, controllable, and with complete rules. Multi-agent reinforcement learning has seen substantial work on esports-level games (like Dota and StarCraft), but those environments are heavy and have high barriers to entry. Using a 2D battle royale clone for 4-player squad coordination makes both training and analysis far cheaper, suitable for rapid idea iteration.

More noteworthy is that information isolation design. Having the actor see only what a real client can see is not just an engineering trick but a necessary condition for making research results practically deployable. Much multi-agent research stops at simulation precisely because agents rely on information unavailable in reality.

The author has invited questions, and if the community pushes, he may reveal more details about how the hierarchical actions are trained and how team reward weights are tuned. For those working on multi-agent coordination, the value of this case lies not in the results but in the fact that it lays out an entire set of engineering trade-offs on the table.

Information isolation in reinforcement learning corresponds to the problem of designing the observation space: the observations available to the agent must match the information obtainable at deployment. Otherwise, the trained policy will depend on "privileged information" unavailable in reality, causing performance collapse when transferred to real environments. Aligning the actor with the real client interface and reserving privileged state only for the critic is a common practice that combines the sample-efficiency advantages of training with deployment feasibility, and is also a key prerequisite for this experiment to actually run in practice.

One-line Summary

A Reddit developer trained a 4-player squad to play the open-source battle royale survev with MAPPO, showcasing a multi-agent coordination experiment.

Paragraph Summary

A Reddit developer trained a 4-player AI squad to play the open-source battle royale game survev using MAPPO, releasing a 40-second gameplay clip and several training pitfalls. The experiment's highlight is compressing several core challenges of multi-agent reinforcement learning—partial observability, credit assignment, and hierarchical action spaces—into a lightweight, reproducible scenario. The system uses information isolation: all four members share one actor network, agent id serves as input to distinguish identities, the actor can only see information consistent with the real client, and privileged state is fed only to the centralized critic. Actions are divided into three layers—intent, target, and use—to improve exploration efficiency and interpretability. The squad shares a team reward and uses coordination terms like focus fire, crossfire, and revive to guide behavior. The author does not release code or details, concerned it could be used as a cheating tool on public servers.