Adversarial RL: Why Are Critic Attacks Stronger in Multi-Agent Environments?

Introduction: A Puzzle from Practice

In the study of reinforcement learning (RL) safety, adversarial attacks have become an increasingly critical topic. When a trained agent policy is deployed in the real world, how to evaluate its robustness against malicious perturbations directly determines the system's reliability and safety.

Recently, a researcher shared an intriguing experimental phenomenon on Reddit: when performing adversarial attacks on multi-agent PPO policies, the results obtained were exactly the opposite of the conclusions in the field's classic paper, Zhang et al. (2020). This seemingly contradictory finding actually reveals a subtle tension between theoretical assumptions and real-world scenarios in adversarial reinforcement learning.

reddit source: On Adversarial RL

The SA-MDP Framework and Two Attack Pathways

What Is SA-MDP

The SA-MDP (State Adversarial Markov Decision Process) framework proposed by Zhang et al. is an important theoretical foundation for studying observation-perturbation attacks. Its core idea is: the attacker does not change the true state of the environment, but instead perturbs the observation received by the agent, inducing the agent to make suboptimal decisions.

The SA-MDP framework is built upon the classic Markov Decision Process (MDP), characterizing security threats in real deployment scenarios by introducing adversarial observation perturbations. In a standard MDP, the state observed by the agent is consistent with the true state of the environment; in SA-MDP, however, the attacker is modeled as a constrained perturbation function whose output is subject to L-p norm ball constraints (usually the L∞ norm), ensuring that the perturbation is perceptually imperceptible. This framework has deep connections to adversarial example research in computer vision—the FGSM method proposed by Goodfellow et al. in 2015 and the PGD method proposed by Madry et al. in 2018 have both been directly transferred to RL observation attack scenarios, allowing the field to advance rapidly on the rich toolchain already established in image adversarial attacks.

This setup is close to reality—in many scenarios, attackers cannot directly tamper with the physical world, but they can interfere with sensor readings or communication channels, thereby affecting the agent's perception of the environment.

Critic Attacks vs. Actor Attacks

Policy-gradient-based algorithms (such as PPO) typically contain two core networks. Proximal Policy Optimization (PPO), proposed by OpenAI in 2017, is one of the most widely used deep reinforcement learning algorithms today. Its Actor-Critic dual-network architecture provides attackers with two distinct sources of gradients:

Actor network π(s): Directly determines the action the agent takes in a given state. For continuous control tasks, the Actor typically outputs the mean and variance of a Gaussian distribution.
Critic network V(s): Evaluates the value of the current state, used to guide policy updates. PPO's core innovation lies in introducing a clipped objective function that limits the magnitude of each policy update, while the Critic provides the advantage estimation baseline for this process.

Based on these two networks, two types of adversarial attacks can be designed:

Critic-based attacks: Use the gradient of the value function V(s) to generate perturbations, aiming to push the agent into states evaluated as "low value."
Actor-based attacks: Directly use the gradient of the policy network π(s) to generate perturbations, aiming to maximally change the action output.

The core assertion of Zhang et al. is: attacks generated based on the Critic network are expected to be weaker than attacks generated directly using the Actor network. This conclusion was empirically supported across the multiple single-agent simulation environments they used.

An Anomalous Phenomenon in Practice

Opposite Results in Multi-Agent Environments

The experiments by the aforementioned researcher observed exactly the opposite conclusion. The specific experimental setup was as follows:

Environment: Multiple multi-agent scenarios based on the VMAS (Vectorized Multi-Agent Simulator) library. VMAS is a high-performance multi-agent simulation platform developed by Bettini et al., designed for large-scale parallel training. It supports GPU-accelerated vectorized environments and provides various continuous control benchmark tasks such as navigation, pursuit, and object transport, with native support for cooperative, competitive, and mixed game scenarios.
Policy types: IPPO (Independent PPO) and GPPO (Graph Independent PPO, see Bettini et al. 2023), including heterogeneous versions.
Attack method: Adapting the PGD (Projected Gradient Descent) attack to continuous-action policies, using the closed-form solution of KL divergence to construct the perturbation objective. PGD was systematically proposed by Madry et al. in 2018 and is regarded as the "strongest baseline" among first-order adversarial attack methods. Its essence is to perform multi-step gradient ascent on the attack objective function under L-p norm constraints, projecting the perturbation back into the valid constraint set after each step. For continuous Gaussian policies, the KL divergence has an analytical closed-form solution that can be directly used for backpropagation without sampling approximation, offering significantly higher computational efficiency than discrete-action scenarios.

Under the above setup, Critic-based attacks were instead stronger than Actor-based attacks—running directly counter to the expectation of Zhang et al.

Experimental Error, or Scenario Difference?

This researcher raised a key question: Is this due to a flaw in the experimental design, or does the result, owing to scenario differences, not actually contradict the findings of Zhang et al. at all?

This touches on the question of the boundaries of applicability of empirical conclusions, which is worth exploring in depth.

In-Depth Analysis: Why the Two Scenarios Yield Different Results

The Essential Difference Between Single-Agent and Multi-Agent

The empirical basis of Zhang et al. is single-agent environments, whereas multi-agent systems introduce a fundamental change in complexity. The core challenge of multi-agent reinforcement learning (MARL) lies in the problem of non-stationarity: from an individual agent's perspective, other agents' policies continuously change throughout training, making the environment effectively a non-stationary dynamic system. This fundamentally changes the semantics of the value function and the nature of the gradient signal:

Non-stationarity: Each agent's optimal policy depends on the behavior of other agents, and the environment is non-stationary from an individual agent's perspective. In a single-agent MDP, V(s) only needs to encode the interaction dynamics between a single agent and the environment; in multi-agent systems, however, V(s) implicitly compresses the complex interaction information of the entire joint system.
Coupling of value functions: In multi-agent scenarios, the information encoded by V(s) may be tightly coupled with the joint states of other agents, and the "sensitivity" information carried by its gradient is far richer than in the single-agent case.

This means that, in a multi-agent setting, the Critic network may capture more sensitive perturbation directions for the policy, thereby significantly amplifying the effect of Critic-based attacks.

Behavioral Differences Between Continuous Action Spaces and KL-Divergence Attacks

Another key difference lies in the continuity of the action space. Some experiments in the original paper involved discrete actions, whereas VMAS scenarios are typically continuous control tasks. When constructing PGD attacks using the closed-form solution of KL divergence, the sources of the "target signal" for the two types of attacks are essentially different:

Actor-based KL attacks: Directly disturb the action distribution, aiming to maximize KL(π(·|s) ∥ π(·|s+δ)). But in continuous Gaussian policies, a small perturbation in the observation space, after nonlinear mapping through a deep network, may have its impact on the action mean diluted by the network structure, and may not substantially change the action output.
Critic-based attacks: Search for more "lethal" observation perturbations along the direction of the value gradient, potentially pushing the agent toward genuinely bad state regions. Their destructive power acts directly on the expectation of cumulative return, rather than merely on the distributional shift of the current step's action.

The Additional Impact of Graph-Structured Policies (GPPO)

GPPO introduces graph neural networks to model relationships between agents. The message-passing mechanism of GNNs enables each agent's representation to adaptively integrate information from neighboring agents, thereby allowing each agent's policy to perceive the global context. Once the graph structure is introduced, gradients no longer propagate solely within a single agent's local network but instead diffuse across the entire agent graph through edge connections. This globally coupled gradient flow characteristic may cause the Critic network to implicitly encode the joint dynamics of the multi-agent system when computing state values, providing more destructive perturbation directions for Critic-based attacks—a property that single-agent policies with traditional MLP architectures do not possess, further intensifying the divergence between the experimental results and the original paper's conclusions.

Conclusions and Insights

Based on comprehensive analysis, the "anomalous" phenomenon observed by this researcher most likely does not truly contradict the conclusions of Zhang et al., but rather reveals the boundaries of applicability of the original conclusion:

Theoretical conclusions are context-dependent: Zhang et al.'s assertion is built on single-agent settings, specific environments, and specific attack constructions. Directly extrapolating it to multi-agent continuous control scenarios requires caution. The theoretical derivation of the SA-MDP framework itself does not explicitly account for multi-agent non-stationarity, which is its core limitation.
Multi-agent value functions are more "information-dense": In coupled systems, the Critic network implicitly compresses the global interaction dynamics of the joint system, potentially encoding a stronger attack signal that gives Critic attacks an advantage in such scenarios.
Attack construction is crucial: Continuous-policy attacks based on the closed-form solution of KL divergence differ essentially from discrete-action attacks—the former acts directly on the analytical form of continuous distributions, while the latter relies on discrete probability vectors. The propagation characteristics of gradient signals are entirely different and cannot be conflated.

For practitioners engaged in adversarial RL research, this case provides a valuable reminder: do not blindly copy existing conclusions; any robustness evaluation should be re-validated in the target deployment scenario. Systematically comparing the relative strength of the two types of attacks under single-agent versus multi-agent settings is itself a valuable contribution to the field, worth further organizing into a paper or technical report.