RobustRLlib: A Unified Benchmark Library for 17 Robust Reinforcement Learning Algorithms

When reinforcement learning is deployed in the real world, policies often fail due to environmental shifts and disturbances, making Robust RL a key research direction. However, despite the many existing algorithms, two prominent obstacles remain: poor algorithm reusability and unclear empirical capability boundaries. The recently released RobustRLlib attempts to solve both at once, providing the robust RL community with an algorithm-centric library and benchmark.

Unified interface integrates 17 algorithms to improve reusability

RobustRLlib's core design philosophy is "algorithm-centric." It integrates 17 representative robust online, offline, and safe reinforcement learning algorithms under a unified interface, significantly improving code reusability. Previously, researchers comparing different robust RL algorithms often had to set up environments and adapt interfaces separately, resulting in extremely high migration costs. RobustRLlib standardizes the interface, making algorithm switching and combination simple and lowering the barrier to applying existing methods to new problems.

This design also accounts for extensibility. As an open-source, user-friendly platform, it provides a reproducible, extensible codebase that makes it easy for researchers to develop new robust RL algorithms on top of it without building infrastructure from scratch.

Robust RL focuses on maintaining acceptable performance when environment dynamics, observations, or reward functions shift relative to training. Its formulation is typically modeled as worst-case optimization over a perturbation set, for example maximizing a lower bound on expected return within an uncertainty set of environment parameters. Related is safe reinforcement learning, which constrains the probability or cumulative cost of violating safety constraints rather than solely pursuing robustness to disturbances; offline reinforcement learning refers to learning policies only from a fixed dataset without interacting with the environment, often requiring handling of distribution shift. The three have different goals but are often conflated; RobustRLlib unifies online, offline, and safe algorithms precisely so researchers can distinguish these settings within the same framework.

Enhanced evaluation platform covers real-world shifts and advanced tasks

Beyond the algorithm library, RobustRLlib also builds an enhanced evaluation platform. The platform covers diverse real-world shift scenarios and incorporates advanced humanoid robot and vision-language-action (VLA) tasks. This means the benchmark is no longer limited to simple simulated environments but moves toward complexity closer to actual deployment.

For robust RL, the design of evaluation environments is crucial. If test scenarios are too uniform, algorithms may simply overfit to specific perturbations. By introducing multiple shifts and high-end tasks, RobustRLlib attempts to examine algorithm robustness boundaries more comprehensively.

Vision-language-action (VLA) models are a class of policy architectures that jointly model visual perception, natural language instructions, and action generation, typically using pretrained multimodal large models as backbones to output control commands for robot end effectors or joints. VLA tasks demand higher robustness: changes in lighting, occlusion, and background in visual inputs, as well as rephrasing of language instructions, all cause distribution shift. Humanoid robots bring challenges such as high-dimensional continuous control, underactuation, and rich contact, with dynamics uncertainty far greater than that of desktop robotic arms. Including these two task types in the benchmark means evaluation no longer only examines parameter perturbations under low-dimensional states but covers shifts across the entire perception-language-control chain, closer to failure modes in real deployment.

273 perturbation configurations map capability boundaries

RobustRLlib systematically evaluates current robust RL algorithms across 273 perturbation configurations, producing an up-to-date capability map. The evaluation results reveal two key insights: which scenarios existing algorithms remain effective in, and where significant robustness gaps exist.

The value of this capability map lies in providing researchers with clear reference coordinates. Previously, the empirical boundaries of robust RL algorithms lacked systematic organization, and different papers used varying experimental setups, making horizontal comparison difficult. RobustRLlib's unified evaluation fills this gap, helping the community identify directions that truly need tackling rather than repeatedly validating known-effective scenarios.

Perturbation configurations typically refer to combinations of various disturbances applied to observations, dynamics parameters, action execution, or rewards, such as observation noise intensity, mass and friction coefficient shifts, action delays, and external pushes. 273 configurations imply grid-based or combinatorial sampling across multiple perturbation dimensions, forming a relatively broad test matrix. The capability map approach aggregates algorithm performance across different perturbation types and intensities into structured results to identify which dimensions algorithms are sensitive to and which they are robust to. This evaluation differs from a single average score, exposing hidden risks of "good average performance but collapse under certain perturbations" and providing targeted directions for future algorithm design.

Open ecosystem drives robust RL deployment

RobustRLlib positions itself not only as a benchmarking tool but as the starting point for an open ecosystem. Its open-source nature means researchers can reproduce experimental results, extend the algorithm collection, and contribute new evaluation tasks. For engineers hoping to apply robust RL to real systems, the library also provides directly usable algorithm implementations and realistic test environments.

For robust reinforcement learning to move from papers to deployment, a unified algorithm library and a trustworthy benchmark are both indispensable. RobustRLlib works on both dimensions simultaneously, promising to accelerate the iteration and deployment validation of robust RL algorithms.

One-line Summary

RobustRLlib integrates 17 robust RL algorithms under a unified interface and maps capability boundaries with 273 perturbation configurations.

Paragraph Summary

RobustRLlib addresses two major obstacles in robust reinforcement learning—poor algorithm reusability and unclear empirical capability boundaries—with an algorithm-centric open-source library and benchmark. It integrates 17 robust online, offline, and safe reinforcement learning algorithms under a unified interface, reducing algorithm switching and migration costs while remaining extensible. The evaluation platform covers diverse real-world shift scenarios and incorporates humanoid robot and vision-language-action tasks, breaking beyond simple simulated environments. Through systematic evaluation across 273 perturbation configurations, the work produces a capability map of current algorithms, clarifying which scenarios remain effective and where significant robustness gaps exist, providing the community with comparable reference coordinates and driving reproduction, extension, and deployment validation of robust RL through an open ecosystem.