AtomicRL: Atomic Reinforcement Learning Solves Long-Horizon Robotic Manipulation Challenges

Vision-Language-Action (VLA) models show promise in robotic manipulation, but imitation learning (IL) alone often struggles to master complex long-horizon skills. Reinforcement learning (RL) offers an effective path to improving VLA policies, yet existing VLA-RL methods often suffer from delayed and biased credit assignment due to trajectory-level sparse rewards, while uneven mastery of subtasks creates bottlenecks that limit overall performance. To address this, researchers propose AtomicRL—an RL post-training framework that leverages atomic task structure to achieve reliable credit assignment and adaptive exploration.

Core Challenges: Credit Assignment and Bottlenecks

Long-horizon robotic manipulation tasks typically consist of multiple atomic subtasks (e.g., "grasp the cup" followed by "place it on the table"). Traditional RL uses trajectory-level sparse rewards, meaning a reward is only given when the entire task succeeds, making it difficult for the model to determine which subtask contributed to success or failure. This delayed and biased credit assignment makes VLA policy learning inefficient.

More challenging still, mastery levels across subtasks are often uneven. For example, a robotic arm may easily learn to grasp but repeatedly fail at placement. This imbalance creates a learning bottleneck: the model wastes extensive sampling on already-mastered subtasks while the truly weak links receive insufficient attention. AtomicRL is designed precisely to address these two pain points.

TD Residuals and Credit Assignment: The temporal difference (TD) residual is a core signal in reinforcement learning, defined as the immediate reward plus the discounted value estimate of the next state, minus the value estimate of the current state. It measures "how much better the actual outcome is than expected," and policy gradients use this signal to update value functions and policies. Under trajectory-level sparse rewards, a non-zero reward is only obtained when the entire trajectory ends, at which point the TD residual propagates backward along the whole trajectory, causing the credit of early actions to be diluted or even distorted by later outcomes—this is the "delayed and biased" credit assignment mentioned in the paper. The role of atomic task structure is to provide finer-grained boundaries so that the TD residuals of failed subtasks can be locally constrained, thereby preventing error contamination across subtask boundaries.

PPO (Proximal Policy Optimization): PPO is one of the most commonly used online policy gradient algorithms in reinforcement learning, balancing sample efficiency and training stability by clipping the importance sampling ratio to limit the magnitude of each policy update. The paper uses PPO as a baseline because it is a typical online RL method in VLA-RL post-training. AtomicRL's improvement does not replace PPO itself but introduces semantic-aware gating and bottleneck-adaptive sampling on top of it, so the 3.6%–4.0% absolute gains reflect improvements from structural credit assignment rather than a change in algorithm family.

Semantic-Aware Policy Optimization (SAPO)

AtomicRL introduces Semantic-Aware Policy Optimization (SAPO), whose core is a semantic-aware gating mechanism. This mechanism constrains TD residuals from failed atomic subtasks, preventing them from propagating across task boundaries. Simply put, if a subtask fails, its error signal does not "contaminate" the learning of other subtasks, thereby mitigating credit contamination while preserving intra-subtask credit assignment and value bootstrapping.

This design allows each atomic subtask to independently evaluate its own performance rather than being obscured by the success or failure of the overall trajectory. For long-horizon tasks, this fine-grained credit assignment significantly improves learning stability.

Bottleneck-Adaptive Targeted Exploration (BATE)

To address uneven subtask mastery, AtomicRL proposes Bottleneck-Adaptive Targeted Exploration (BATE). This method dynamically coordinates task sampling and local exploration incentives based on each atomic subtask's mastery level. Poorly mastered subtasks receive more sampling opportunities and exploration rewards, directing learning resources toward remaining bottlenecks.

BATE is essentially an adaptive curriculum learning approach: the model no longer samples all subtasks uniformly but focuses on the currently weakest links. This avoids redundant training on already-mastered skills and accelerates overall task convergence.

Curriculum Learning and Adaptive Sampling: Curriculum learning is a training strategy that organizes samples or tasks in order from easy to hard, enabling the model to learn stably as its capabilities gradually improve. Traditional curriculum learning typically relies on manually designed difficulty sequences, whereas BATE implements a data-driven adaptive curriculum: it dynamically allocates sampling probabilities and local exploration incentives based on each atomic subtask's current mastery level (e.g., success rate or value estimate). Poorly mastered subtasks are assigned higher sampling weights and exploration rewards, effectively automatically focusing the training curriculum on the "zone of proximal development." This approach is conceptually related to Prioritized Experience Replay but operates at the subtask level rather than the individual experience level, making it better suited to identifying and breaking through structural bottlenecks in long-horizon robotic manipulation.

Real2Sim2Real: From Simulation to Reality

To enable policies trained with online RL in simulation to transfer to real robots, AtomicRL develops a Real2Sim2Real pipeline. This pipeline bridges online RL in simulated environments with real-world deployment, facilitating transfer to actual physical robots.

This engineering design addresses a key obstacle to deploying RL in robotics—the gap between simulation and reality. Through a systematic transfer process, AtomicRL goes beyond simulation benchmarks and possesses practical deployment potential.

Real2Sim2Real and the Sim-to-Real Gap: Sim-to-real (Sim2Real) transfer is a classic challenge in robot learning, primarily stemming from mismatches between simulation and the real world in physical parameters (friction, mass, latency), visual appearance, and sensor noise. Real2Sim2Real typically refers to a closed-loop process in which real data is first used to calibrate or construct a simulation environment (Real2Sim), and then a policy is trained in simulation and deployed back to the real world (Sim2Real). AtomicRL's pipeline operates within this framework, enabling policies trained with online RL to bridge simulation and physical deployment. It should be noted that the 21.3% average real-world improvement depends on the effectiveness of this transfer pipeline, while the gains on simulation benchmarks mainly come from SAPO and BATE's improvements to credit assignment and exploration—the two have different sources of contribution.

Experimental Results: Significant Gains on Long-Horizon Tasks

On the LIBERO-LONG benchmark, AtomicRL achieves absolute performance improvements of 3.6% and 4.0% over PPO on two VLA architectures. On CALVIN ABC-D, full-task performance improves over PPO by up to 9.5%. Combined with the Real2Sim2Real pipeline, real-world task performance improves by an average of 21.3%.

These results validate AtomicRL's effectiveness in improving VLA policy learning efficiency and manipulation performance on long-horizon tasks. In particular, the 21.3% average improvement in real environments demonstrates the framework's practical value from simulation to reality.

Significance and Outlook

AtomicRL's value lies in combining RL post-training with atomic task structure, using semantic-aware gating to address credit contamination and adaptive exploration to focus on bottlenecks. For the robotic manipulation field, this provides a path to continuously optimize VLA policies through online RL without relying on large amounts of expert demonstrations. In the future, how to automatically discover atomic subtask structure and how to extend to more open-ended task scenarios will be directions worth watching.

One-line Summary

AtomicRL leverages atomic task structure to improve credit assignment and exploration efficiency in RL post-training for VLA models.

Paragraph Summary

AtomicRL is a reinforcement learning post-training framework for Vision-Language-Action (VLA) models, targeting two major problems in long-horizon robotic manipulation: delayed and biased credit assignment caused by trajectory-level sparse rewards, and bottlenecks formed by uneven subtask mastery. It proposes Semantic-Aware Policy Optimization (SAPO), which uses semantic-aware gating to constrain TD residuals of failed atomic subtasks from propagating across boundaries, mitigating credit contamination; and Bottleneck-Adaptive Targeted Exploration (BATE), which dynamically adjusts sampling and exploration incentives based on subtask mastery to direct learning resources toward weak links. Combined with the Real2Sim2Real pipeline, AtomicRL improves over PPO by 3.6%–4.0% on LIBERO-LONG, up to 9.5% on CALVIN ABC-D full tasks, and an average of 21.3% on real-world tasks.