The Nonstationarity Problem in Multi-Agent Reinforcement Learning: A Deep Dive into PACT

When "Individually Stable" Meets "Collectively Drifting"

Multi-Agent Reinforcement Learning (MARL) has long grappled with a core dilemma: each agent's policy appears stable in isolation, yet when multiple agents interact within the same environment, their behaviors continuously reshape the distribution of the environment each agent faces — causing the overall system to exhibit nonstationarity. The paper Stationary Alone, Drifting Together is built around exactly this problem. Its title captures the paradox in a nutshell: stable alone, drifting together.

In traditional single-agent reinforcement learning, the environment is assumed to be a relatively fixed Markov Decision Process. But in multi-agent settings, when other agents are simultaneously learning and adjusting their policies, the environment itself — from any single agent's perspective — becomes a constantly moving target. This interaction-mediated nonstationarity severely undermines the convergence and stability of learning.

The Markov Decision Process (MDP) is the standard formal framework for single-agent reinforcement learning. It describes agent-environment interaction via state spaces, action spaces, transition probabilities, and reward functions — and crucially assumes that transition probabilities remain fixed throughout training. In multi-agent settings, researchers typically extend this to Stochastic Games (or Markov Games), where each agent's transitions and rewards depend on the joint actions of all agents. The root of the problem is this: when every agent is updating its policy — synchronously or asynchronously — the "current policies of other agents" constitute implicit environment parameters from any single agent's perspective. These parameters shift continuously during training, violating the stationarity assumption of MDPs and invalidating all convergence guarantees derived from it.

Why Interaction-Mediated Nonstationarity Is So Difficult

The fundamental issue is that the state transitions and reward signals an agent observes during training do not come from a fixed environment — they are modulated by the current policies of other agents. When teammates or opponents change their behavior, a previously learned optimal response can become instantly obsolete.

This "drift" has several characteristic features: it is coupled, with multiple agents' policy updates intertwined and difficult to analyze in isolation; it is time-varying, with the direction and magnitude of drift changing as training progresses; and it is resistant to static modeling, since the challenge arises precisely from the dynamics of interaction itself.

This is why many algorithms that perform brilliantly on single-agent tasks fall apart in multi-agent cooperative or competitive settings — training becomes unstable, policies oscillate, or convergence fails entirely. Enabling agents to stay adaptive amid collective drift is one of the field's central challenges.

The research community has proposed several mainstream strategies for mitigating nonstationarity, which provide useful context for understanding where PACT fits. The Centralized Training with Decentralized Execution (CTDE) paradigm shares global information during training (as in QMIX, MAPPO) to reduce mutual interference from policy updates. Opponent Modeling methods attempt to infer the policy intentions of other agents so as to anticipate environmental drift. Meta-Learning approaches aim to train agents that can rapidly adapt to shifts in policy distributions. Each approach has its limitations, however: CTDE still suffers from drift during decentralized execution; opponent modeling scales exponentially in computational cost with the number of agents; and the generalization bounds of meta-learning methods remain unclear. PACT's motivation is to find a solution that targets the interaction structure itself more directly.

PACT's Approach: Interaction-Aware Adaptation

The method proposed in this work — named PACT in the demo — aims to directly model and adapt to interaction-mediated nonstationarity. Based on the naming and demo positioning, its core insight is to stop treating the environment as a fixed black box, and instead explicitly identify and adapt to the distributional shifts caused by other agents' interactions.

Typical entry points for such approaches include: modeling or predicting the policies of other agents to anticipate environmental drift ahead of time; incorporating adaptive mechanisms into the learning process so that policies can respond quickly to interaction-induced changes; and applying consistency or coordination constraints to reduce the instability caused by simultaneous multi-agent updates.

The authors have made an interactive demo available on Hugging Face Spaces (pact-demo), allowing readers to observe the method's behavior in multi-agent scenarios directly — rather than relying solely on theoretical descriptions in the paper. A runnable demo is especially valuable for validating an algorithm's adaptive capabilities in real interactions.

Implications for Research and Practice

MARL is a critical pathway toward more complex collaborative systems. Whether it's coordinating autonomous vehicle fleets, controlling distributed robot formations, or simulating large-scale games and economic systems — all of these require effective modeling of group interaction dynamics. The ability to handle interaction-mediated nonstationarity is one of the prerequisites for moving these applications from the lab to the real world.

For researchers, this type of work offers a fresh lens through which to examine the sources of nonstationarity — not as noise to be avoided, but as a phenomenon determined by interaction structure that can be modeled and adapted to. That shift in perspective is itself valuable.

It should be noted that, since the currently available materials are primarily the paper title and the online demo link, the specific algorithmic details, experimental benchmarks, and comparative results of PACT still require consulting the full paper. Readers interested in this direction are encouraged to visit the Hugging Face demo page for a hands-on experience, and to consult the original paper for complete methodological and evaluation details.

Autonomous vehicle fleet coordination and multi-robot formation control are among the most representative real-world applications of MARL — and both have an extremely low tolerance for nonstationarity. In vehicle fleets, each car's policy change instantly reshapes the traffic distribution perceived by other vehicles; policy oscillation could directly cause safety incidents. In robot formations, a single robot's path adjustment affects the stability of the entire formation, and convergence failure from nonstationarity means mission failure. Because the cost of errors in these applications is so high, algorithms must be robust to interaction-induced distributional shifts both during training and deployment — not just on isolated simulation benchmarks. This is precisely the critical threshold that must be crossed as research in this direction moves from pure theory toward engineering practice.