Getting Started with MARL (Multi-Agent Reinforcement Learning): A Complete Path from Theory to Code

A Common Learning Dilemma

Recently, a graduate student's help post on the Reddit reinforcement learning community resonated with many. This student was working on a graduation research project focused on Multi-Agent Reinforcement Learning (MARL) while also taking an RL course. However, they ran into a classic problem that many RL learners face: I understand the theory, but how do I write the code?

They candidly admitted: "I know the concepts of rewards, policies, value functions, the difference between value-based and policy-based methods, bias, etc., but I really struggle to understand how it works in practice and how to implement it from scratch. I'm not good at absorbing pure theory—I need to see code to understand how the system's pipeline connects together."

reddit original post

This post reflects a real phenomenon: the MARL field is overflowing with theoretical papers, but tutorials that truly hold your hand through "writing code, getting it running, and understanding every line" are extremely scarce. This article attempts to systematically address this problem by outlining a practical path from single-agent RL to multi-agent reinforcement learning.

Why Are MARL Code Tutorials So Rare?

The Inherent Complexity of the Field

There are objective reasons why MARL lacks "code-first" introductory content. Compared to single-agent RL, multi-agent environments introduce several thorny problems:

Non-stationarity: While each agent is learning, other agents are also changing, making the environment unstable from any single agent's perspective. In single-agent settings, the environment's transition probability P(s'|s,a) is fixed, satisfying the basic assumptions of Markov Decision Processes (MDPs). But in multi-agent scenarios, the "environment" each agent faces actually includes the policies of all other agents. When Agent A is learning to improve its policy, Agent B is simultaneously updating, meaning from A's perspective, the environmental dynamics are constantly drifting. This breaks convergence guarantees—the convergence proofs of most single-agent RL algorithms rely on the stationarity assumption. In practice, non-stationarity manifests as wildly oscillating training curves, policy cycling, and other phenomena, and is one of the main causes of MARL training instability.
Credit Assignment: When a team receives a reward, it's difficult to determine which agent contributed. Suppose a 5-agent team completes a task and receives a +100 reward—how do you determine each agent's actual contribution? If you simply divide the total reward equally, free-riding agents and agents making critical contributions receive the same incentive, severely affecting learning efficiency. This problem is also known as the "Shapley value" allocation problem in game theory. In MARL algorithm design, methods like VDN and QMIX are essentially attempting to solve credit assignment—by decomposing the team value function into some combination of individual contributions, allowing each agent to receive gradient signals related to its behavior.
Scalability: As the number of agents increases, the joint action space expands exponentially. If each agent has 5 available actions, the joint action space for 2 agents is 25, for 3 it's 125, and for 10 it reaches approximately 10 million. This combinatorial explosion makes learning directly in the joint space infeasible, which is why virtually all practical MARL algorithms employ some form of decomposition or independent learning strategy.

These problems make MARL implementation far more complex than "running a CartPole," and tutorial authors often need extensive space to explain background context, causing much content to lean toward theory rather than hands-on practice.

Fragmented Tool Ecosystem

Single-agent RL has mature and user-friendly libraries like Stable-Baselines3 and CleanRL, while the MARL tool ecosystem is relatively scattered with low standardization, raising the barrier to entry for newcomers. This fragmentation has multiple causes: MARL covers an extremely diverse range of scenario types (cooperative, competitive, mixed-motive games, communication, heterogeneous agents, etc.), making it difficult for a single unified API to cover all cases. Additionally, many MARL codebases are byproducts of academic papers, primarily aimed at reproducing experimental results, and lack ongoing maintenance and documentation support.

Recommended "Code-First" MARL Learning Path

Step 1: Build a Solid Single-Agent RL Code Foundation

Before jumping into multi-agent reinforcement learning, it's strongly recommended to first be able to implement single-agent algorithms from scratch. The top recommendation here is CleanRL, an open-source project. Its core philosophy is "single-file implementation"—each algorithm (DQN, PPO, SAC, etc.) is contained in a single independent Python file with no layers of abstract wrappers, making it ideal for reading line by line to understand "how the pipeline connects together."

CleanRL was initiated by Costa Huang in 2021, and its design philosophy stands in stark contrast to Stable-Baselines3. Stable-Baselines3 pursues modularity and reusability, splitting algorithms into multiple abstraction layers like policies, callbacks, and buffers—suitable for engineering deployment but not conducive to learning and understanding. CleanRL deliberately "anti-patterns," with each algorithm implementation completely self-contained in a single Python file of approximately 300-500 lines, with all logic arranged linearly. The project also integrates Weights & Biases experiment tracking, with each algorithm having public training curves and hyperparameter configurations to ensure reproducibility. As of 2024, CleanRL covers over 20 algorithm variants, including multi-GPU training versions and classic benchmarks like Atari/MuJoCo.

For learners who "need to see code to understand," CleanRL is practically tailor-made. You can open ppo.py and trace a complete data flow from environment interaction, experience collection, and advantage estimation to gradient updates.

Step 2: Use Dedicated MARL Frameworks

Once you've built a solid foundation, you can move to MARL-specific tools:

PettingZoo: Think of it as the multi-agent version of Gymnasium. It provides numerous standardized multi-agent environments (cooperative, competitive, mixed scenarios) and serves as the de facto standard for MARL experiments. PettingZoo is maintained by the Farama Foundation (which also maintains Gymnasium, the successor to OpenAI Gym). It defines two core APIs for multi-agent environments: AEC (Agent Environment Cycle) and Parallel. In AEC mode, agents act sequentially in turn, suitable for turn-based games (like board games); in Parallel mode, all agents act simultaneously, suitable for real-time interaction scenarios. PettingZoo includes dozens of built-in environments including classic games (like Prisoner's Dilemma, Rock-Paper-Scissors), Atari multiplayer games, and MPE (Multi-Particle Environment). Its core value lies in providing a unified interface standard that enables fair comparison of different MARL algorithms on the same environments.
PyMARL / EPyMARL: Maintained by Oxford University's WhiRL lab, implementing classic algorithms like QMIX, VDN, and MADDPG. Paired with the StarCraft Multi-Agent Challenge (SMAC) environment, it's a common starting point for academic research. SMAC was proposed in 2019 and is the most widely used benchmark environment for cooperative MARL research. Built on Blizzard's StarCraft II game engine, it features a series of micromanagement combat scenarios—for example, controlling 3 Marines against 3 enemy units, or controlling 8 Zerglings against stronger enemy formations. SMAC's characteristics include partial observability (each unit can only see a limited field of view), heterogeneous agents (different unit types have different abilities), and the need for fine-grained coordination (like focus fire, positioning, and ability combos). In 2023, the team released SMACv2, adding randomness and difficulty to address the "overfitting" problem in earlier versions.
MARLlib: Built on Ray RLlib with broad algorithm coverage, suitable for scenarios requiring distributed training.

Step 3: Start Reading from the Simplest MARL Algorithms

Recommended algorithm learning order:

Independent Q-Learning (IQL): The most straightforward approach—each agent independently trains a DQN, ignoring the existence of other agents. From each agent's perspective, it's solving an ordinary single-agent RL problem—except that the "environment" contains other agents that are also learning, thus violating the MDP stationarity assumption. Although there's no theoretical convergence guarantee, IQL performs surprisingly well in many practical scenarios. It's an excellent starting point for understanding MARL and serves as the baseline comparison for all more complex methods.
VDN (Value Decomposition Networks): Introduces the value decomposition concept, breaking down the team value into the sum of individual agent values, i.e., Q_tot = Q_1 + Q_2 + ... + Q_n. While this simple additive decomposition assumption limits expressiveness (it implies that each agent's contribution is additive and independent), it enables each agent to contribute to team optimality by independently maximizing its own Q value. VDN is the best entry point for understanding "value decomposition"—a core concept in MARL.
QMIX: An advanced version of VDN that uses a mixing network to non-linearly combine individual agent values. It's the benchmark algorithm for cooperative MARL. QMIX was published by Rashid et al. at ICML 2018, with its core innovation being a monotonicity constraint—∂Q_tot/∂Q_i ≥ 0—ensuring "Individual-Global-Max" (IGM), meaning individual greedy actions are equivalent to globally optimal actions. The mixing network's weights are generated from the global state through a Hypernetwork, which is a typical embodiment of the CTDE paradigm: during training, global information can be used to generate mixing weights, but during execution, each agent only needs to compute its own Q value based on local observations and take the argmax.
MADDPG: Designed for continuous action spaces, adopting the "Centralized Training with Decentralized Execution" (CTDE) paradigm. CTDE is the most mainstream design philosophy in current MARL research, solving a core contradiction: during training we can usually access all agents' information (e.g., in a simulator), but during deployment each agent can only make decisions based on its own local observations. MADDPG's specific approach maintains a centralized Critic (taking all agents' observations and actions as input) and a decentralized Actor (taking only its own observations as input) for each agent. This paradigm was first systematically articulated in Lowe et al.'s 2017 paper and has since become the design foundation for virtually all mainstream MARL algorithms.

Three Core Tips for Bridging Theory to Practice

Start by "Modifying" Rather Than "Writing from Scratch"

Many beginners fall into the trap of thinking they must start from a blank file to "truly understand." In reality, a more efficient path is to first get someone else's code running, then gradually modify it. For example, change the number of agents in QMIX, swap out the reward function, or adjust the network architecture, and build intuition by observing how results change. This approach is known in education as "Scaffolding"—gradually building your own understanding with the support of existing structures, rather than starting from zero without any reference.

Validate Understanding with Minimal Environments

Don't jump straight into StarCraft II. PettingZoo has many toy-level environments, such as simple predator-prey scenarios or MPE's simple_spread (where multiple agents need to separately cover multiple landmarks). These can produce training results in minutes and are perfect for quickly verifying whether your understanding of an algorithm is correct. Another benefit of using small environments is debugging efficiency—when training doesn't converge, you can determine within minutes whether it's a code bug or a hyperparameter issue, rather than waiting hours only to discover something is wrong.

Map Theoretical Concepts to Lines of Code

For the pain point of "understanding theory but not implementation," a practical technique is: take your theory notes and find the corresponding location for each concept in the code. For example, "what does the policy network output" (typically a probability distribution over actions or action means), "which line computes the advantage function" (typically using GAE—Generalized Advantage Estimation, obtained through an exponentially weighted sum of TD residuals), "how does the experience replay buffer sample" (uniform random sampling or priority-based sampling)—when you can map every mathematical symbol to specific code, the gap between theory and practice naturally closes.

Final Thoughts

This Reddit user's confusion fundamentally reflects the long-standing disconnect between theory and engineering practice in reinforcement learning education. The good news is that with the maturation of open-source projects like CleanRL and PettingZoo, the barrier to getting started with MARL code is rapidly decreasing.

For students working on research projects, the best way to learn is often not passively consuming more tutorials, but rather choosing a specific environment and a specific algorithm, getting it running, reading through the code, and then modifying it hands-on. Theory becomes clear through practice, not the other way around.