REVERSAL-BENCH: A Reversibility Benchmark for Reset-Free Reinforcement Learning

The Hidden Assumption in Reset-Free Reinforcement Learning

A core goal of Autonomous RL is to enable agents to continuously train policies without external intervention — no human repositioning a robotic arm, no manually rearranging scattered objects. This "reset-free" capability is widely seen as a critical step toward deploying reinforcement learning in the real world.

However, REVERSAL-BENCH exposes a long-overlooked premise baked into the dominant paradigms: they rely heavily on the reversibility of the underlying environment. In other words, mistakes made during exploration can mostly be undone or naturally recovered. But real-world manipulation tasks don't work that way — pushing an object off a table or spilling granular material (like sand or grain) cannot be undone once it happens. When environments are irreversible, reset-free training hits a steep cliff.

The fundamental difference between Autonomous RL and traditional RL lies in the closed-loop nature of the training process. Traditional RL relies on human or scripted resets at the end of each episode, while Autonomous RL requires the agent to explore and recover on its own, forming a continuously running training loop. Representative works in this paradigm include algorithms like MEDAL and EARL, which typically train agents to simultaneously learn a "forward task policy" and a "reset policy" that alternate to sustain continuous training. This design has shown promise on tasks like table tidying and door manipulation — but its implicit assumption is that environment states can be physically reversed, i.e., the reset policy can find a path back to the initial distribution. Once that assumption breaks down, the reset policy loses any feasible solution, and the entire training loop deadlocks in an irreversible state.

Quantifying "Reversibility" with a Single Continuous Parameter

Prior discussions of reversibility have largely remained qualitative, lacking quantifiable and comparable tools. One of REVERSAL-BENCH's core contributions is the introduction of a continuous parameter ρ ∈ [0, 1] to control the degree of environmental reversibility.

The value of this design is that it transforms the binary opposition of "reversible vs. irreversible" into a smoothly adjustable axis. Researchers can now systematically observe how reset-free algorithm performance gradually degrades as ρ changes, and where performance drops sharply near a critical threshold — what the authors call the reset-free RL cliff. This continuous experimental design is far more informative than simply comparing the two extremes of "fully reversible" and "fully irreversible."

Reset Oracle: Ground-Truth Verification of State Recoverability

Beyond the reversibility axis, REVERSAL-BENCH also provides a reset oracle — a ground-truth-based verification mechanism for determining whether a given state can truly be recovered.

This matters considerably. In many tasks, an agent or algorithm may "believe" a mistake is still recoverable, when in fact the state has already entered an irreversible region. The reset oracle acts as an objective referee, providing definitive judgments on state recoverability so that evaluation results aren't distorted by the algorithm's own optimistic self-assessment. With such a ground-truth baseline, researchers can accurately measure an algorithm's actual robustness to irreversible situations, rather than being misled by overly optimistic self-reporting.

An "oracle" is a common design pattern in algorithmic benchmarking, referring to an external judgment mechanism with perfect information that provides definitive answers. In RL evaluation, an oracle typically does not participate in the agent's decision-making — it serves only as an independent evaluation tool. REVERSAL-BENCH's reset oracle is concretely implemented using the simulator's underlying state information — such as precise 3D object coordinates, velocity vectors, and contact constraints — to determine whether the current state is still within the basin of attraction of the reachable initial distribution. This type of information is often unavailable in real robotic systems, so the reset oracle provides an idealized upper-bound baseline: if an algorithm performs poorly even with perfect recoverability information, the root cause lies in the policy itself rather than information deficiency.

Spanning Multiple Physics Engines and Manipulation Scenarios

To ensure breadth and credibility, REVERSAL-BENCH spans five physics engines and covers eight manipulation settings.

Multiple physics engines mean that conclusions aren't tied to the idiosyncrasies or numerical behavior of a single simulator, reducing the risk of results that hold in one engine but fail in another. Multiple manipulation scenarios allow the benchmark to capture different types of irreversible events — from objects falling to materials being spilled. This cross-engine, cross-task combination makes the benchmark a test of an algorithm's "generalizable reliability" rather than an overfitted evaluation on a narrow task.

A physics engine is a core component of robot simulation, responsible for computing rigid body collisions, friction, joint constraints, and other dynamic behaviors. Different engines differ significantly in numerical integration methods, contact solvers, and collision detection precision — directly affecting the trigger probability and trajectory of critical irreversible events like granular material spills or small object tipping. Mainstream engines include MuJoCo (known for high-precision contact dynamics), PyBullet (open-source and lightweight), Isaac Gym (GPU-accelerated parallel simulation), and Gazebo (ROS ecosystem integration), each with distinct use cases and community preferences. REVERSAL-BENCH's design spanning five engines ensures that experimental conclusions are not systematically biased by any single engine's particular handling of irreversible events, significantly improving the benchmark's external validity.

What This Work Means

The significance of REVERSAL-BENCH lies not just in introducing a new set of evaluation tasks, but in providing a concrete measurement tool for a problem that has long been handled vaguely. The reversibility axis makes "difficulty" adjustable and comparable; the reset oracle makes "whether recovery is actually possible" decidable.

For teams working on reset-free and autonomous reinforcement learning, this benchmark serves as an honest mirror: many algorithms that perform well in reversible environments may experience a cliff-like performance collapse as they approach irreversible thresholds. Confronting this cliff is a necessary step toward taking reinforcement learning out of simulation and into real-world robotic manipulation. Future algorithm designs may need to explicitly incorporate "avoiding irreversible states" as an objective, rather than defaulting to the assumption that mistakes can always be undone.