[KongchangAI]
· 2 min read· 1,156 words

Off-Policy RL + Online Inference Tuning: My Solution to the CoRL 2026 RoboEval Challenge

Off-Policy RL + Online Inference Tuning: My Solution to the CoRL 2026 RoboEval Challenge

Combining off-policy RL with online inference tuning to achieve 0.99 RDS, near-perfect, across eight dual-arm tasks in the RoboEval Challenge.

This article presents the author's solution to the CoRL 2026 RoboEval Challenge. The challenge features eight dual-arm tasks scored against human demonstrations across safety, efficiency, and coordination. The author proposes an off-policy RL framework based on the π0.5 policy, using AWR and RECAP to optimize official scoring advantages with Thompson sampling for online inference tuning. Generated demonstrations and recovery segments further enhance performance. The final policy achieves 0.990 RDS on held-out seeds and 0.987 RDS on official seeds, near-perfect. The general solution reached 0.85 RDS within two days and was far from convergence, showing strong potential.

In the field of robotic manipulation, a core challenge has always been enabling policies not only to complete tasks but to do so safely, efficiently, and with coordinated dual-arm motion. The CoRL 2026 RoboEval Challenge features eight dual-arm tasks, where scoring focuses not only on success rate but also on whether each stage reaches the level of human demonstrations. The author of this article proposes a solution combining off-policy reinforcement learning with online inference tuning, achieving a near-perfect score in the official evaluation.

rss source: Off-Policy RL with Online Inference Tuning: My Solution to the CoRL 2026 RoboEval Challenge

Challenge Background and Scoring Mechanism

The core of the CoRL 2026 RoboEval Challenge lies in fine-grained evaluation of policies. The eight dual-arm tasks require the robot to demonstrate safety, efficiency, and coordination at every stage, with scoring referenced against human demonstrations rather than simple binary success/failure outcomes. This design forces participants not only to pursue task completion but also to optimize the quality and coordination of actions. The author's goal was to develop a general solution capable of adapting to this multi-dimensional scoring standard.

RDS (RoboEval Score) is the comprehensive scoring metric proposed by the challenge to measure policy performance at each task stage. Unlike traditional success rates, RDS references human demonstrations and conducts fine-grained evaluation of action sequences across multiple dimensions including safety, efficiency, and dual-arm coordination. Specifically, the score considers whether the robot completes subtasks in a manner close to humans, whether it avoids dangerous actions, and whether coordination between the two arms is smooth. RDS typically ranges from 0 to 1, with 1 indicating performance at or beyond the level of human demonstrations. Thus, 0.99 RDS means the policy is near-perfect across almost all evaluation dimensions.

General Solution: Off-Policy RL Based on the π0.5 Policy

The author's foundational solution builds on prior work from the LeHome challenge: a policy based on π0.5 that simultaneously serves as a value function. Training employs AWR (Advantage-Weighted Regression) and the RECAP algorithm, directly optimizing the advantage values from the official scoring. Throughout the training loop, each rollout simultaneously serves as a Thompson sampling trial over approximately 20 inference parameters. This design tightly integrates policy optimization with inference tuning, enabling the model to continuously adjust inference parameters during exploration and thereby approach the optimal policy more efficiently.

According to the author, this general solution reached 0.85 RDS (RoboEval Score) on held-out seeds within two days, and was far from convergence. This demonstrates that the off-policy RL framework itself has strong potential to rapidly improve policy performance.

AWR (Advantage-Weighted Regression) is an off-policy reinforcement learning algorithm whose core idea is to weight behavior cloning using the advantage function: actions with high advantage values are assigned higher weights, biasing the policy toward actions better than average. AWR avoids explicit policy gradient estimation, trains stably, and is easy to implement, making it particularly suitable for policy improvement using historical data (offline datasets).

RECAP in this context refers to a recursive or replay-based advantage-weighted policy optimization procedure that iteratively collects data, estimates advantages, and updates the policy, enabling off-policy RL to continuously leverage newly generated rollout data. Combined with AWR, RECAP allows the policy to extract useful signals from historical experience at each iteration, progressively approaching the optimal policy.

Thompson sampling is a probability-based exploration strategy: it samples a set of parameters from the posterior distribution and acts based on that sample. In this article, each rollout simultaneously serves as a Thompson sampling trial over approximately 20 inference parameters, meaning the author treats inference-time hyperparameters (such as action temperature and sampling steps) as random variables, balancing exploration and exploitation through multiple samples. This approach enables online adjustment of inference behavior without additional training, allowing the policy to dynamically adapt to different scenarios at deployment while providing diverse rollout data for off-policy RL.

Competition-Specific Enhancements: Generated Demonstrations and Recovery Segments

To further break through the limitations of local exploration, the author introduced competition-specific data augmentation: generated demonstrations capable of achieving full reward, and recovery segments starting from the policy's own failures. These data supplement regions that local exploration cannot cover, helping the policy learn more comprehensive behaviors. Generated demonstrations provide high-quality positive examples, while recovery segments teach the policy to recover from failure states, enhancing robustness.

The final submitted policy achieved 0.990 RDS on held-out seeds and 0.987 RDS on official seeds. This near-perfect result demonstrates the effectiveness of the solution.

Recovery segments are a data augmentation technique that starts from states where the policy itself failed and records how the policy attempts to recover and ultimately complete the task. Unlike full demonstrations starting from initial states, recovery segments focus on corrective behavior after failure, teaching the policy how to return to the right track when it deviates from the ideal trajectory. In off-policy RL, such data is especially valuable because it covers failure regions difficult for local exploration to reach, enhancing the policy's robustness and generalization. Generated demonstrations provide high-quality positive examples; combined, the two make the training data distribution cover the task's state space more comprehensively.

Technical Details and Implementation Highlights

The key to the entire solution lies in combining off-policy RL with online inference tuning. Off-policy RL enables efficient learning from historical data, while online inference tuning dynamically adjusts inference parameters through Thompson sampling, allowing the policy to adapt to different scenarios at deployment. The AWR and RECAP algorithms ensure the stability of advantage weighting, while generated demonstrations and recovery segments compensate for insufficient exploration. The author notes that the solution achieved significant results within two days and still has room for improvement, providing direction for future research.

Conclusion and Outlook

This work demonstrates the potential of off-policy reinforcement learning in complex robotic manipulation tasks, particularly when combined with online inference tuning and data augmentation, to reach a level close to human demonstrations. In the future, more efficient exploration strategies and finer-grained scoring optimization can be further explored to handle more complex tasks. For participants and researchers, this solution provides a reusable framework and approach.

One-line Summary

Combining off-policy RL with online inference tuning to achieve 0.99 RDS, near-perfect, across eight dual-arm tasks in the RoboEval Challenge.

Paragraph Summary

This article presents the author's solution to the CoRL 2026 RoboEval Challenge. The challenge features eight dual-arm tasks, with scoring referenced against human demonstrations and evaluating policies across multiple dimensions including safety, efficiency, and coordination. The author proposes an off-policy reinforcement learning framework based on the π0.5 policy, using AWR and RECAP algorithms to directly optimize official scoring advantages, with online inference tuning via Thompson sampling within the training loop. To further overcome the limitations of local exploration, the author introduces generated demonstrations and recovery segments as competition-specific data augmentation. The final policy achieves 0.990 RDS on held-out seeds and 0.987 RDS on official seeds, near-perfect. The general solution reached 0.85 RDS within two days and was far from convergence, demonstrating the framework's strong potential.

#off-policy reinforcement learning#online inference tuning#RoboEval Challenge#CoRL 2026#dual-arm manipulation#AWR#RECAP#Thompson sampling#π0.5 policy#robot learning
Share:

Related articles