DexDICE: Dual-Expert Offline Imitation Learning Balances Safety and Performance

In the field of robot learning, safety and performance are often difficult to achieve simultaneously. Traditional imitation learning methods either rely on carefully designed safety cost functions or require expert demonstrations that are both safe and high-performing—yet both are hard to obtain in the real world. DexDICE proposes a novel dual-expert offline imitation learning framework that simultaneously achieves high return and safety constraints by fusing two complementary types of demonstration data: safe but potentially suboptimal demonstrations, and high-performing but potentially safety-violating demonstrations. The method demonstrates significant advantages on the DSRL benchmark and in real-world mobile robot navigation tasks.

Problem Background: The Safety-Performance Trade-off Dilemma

Offline imitation learning aims to learn a policy from pre-collected demonstration data without interacting with the environment. However, in real-world safety-critical scenarios (such as autonomous driving and robot navigation), pursuing only high returns may lead to dangerous behavior, while emphasizing only safety may make the policy overly conservative. Existing methods face two major bottlenecks: first, explicitly specifying cost functions requires domain experts to perform tedious design for complex constraints and makes it difficult to cover all potential risks; second, collecting expert demonstrations that simultaneously satisfy safety and high performance is extremely costly, or even infeasible.

The core insight of DexDICE is that rather than pursuing a perfect single expert, it is better to leverage two types of imperfect but complementary data sources. Safety-compliant demonstrations (such as trajectories of human operators following safety protocols) can provide reliable safety boundaries, while performance-oriented demonstrations (such as aggressive but efficient policies) reveal high-return behavioral patterns. How to effectively fuse the two becomes the key challenge.

Offline Imitation Learning is a reinforcement learning paradigm that trains a policy using only a fixed, pre-collected demonstration dataset without additional interaction with the environment. Unlike online reinforcement learning, it avoids the high costs and safety risks that exploration in real environments may bring, making it particularly suitable for safety-sensitive scenarios such as robotics and autonomous driving. The core challenge of offline imitation learning is distribution shift: the learned policy may visit state-action regions not covered by the demonstration data, leading to error accumulation. To mitigate this issue, researchers often introduce conservative constraints or support constraints to limit the distribution range of the policy. DexDICE combines this idea with safety constraints, using complementary dual-expert data to define the safe feasible region, thereby balancing performance and safety under offline conditions.

DexDICE Method: Dual-Expert Learning Based on Stationary Distribution Correction Estimation

The full name of DexDICE is Dual-expert offline imitation learning via stationary DIstribution Correction Estimation. The method innovatively combines DICE-style offline learning with safe support constraints, rather than adopting traditional distribution matching constraints.

Specifically, DexDICE learns a policy whose stationary distribution is as close as possible to the high-return region of performance-oriented demonstrations, while ensuring through safe support constraints that this distribution does not exceed the region covered by safe demonstrations. This constraint approach allows the agent to fully exploit high-performance behavior within the safe region without explicitly defining a cost function. Theoretical analysis shows that the ideal solution satisfying this constraint can achieve zero unsafe region visits with high probability, thereby theoretically guaranteeing safety.

Compared with existing methods, the advantages of DexDICE are: it does not require any single dataset to simultaneously satisfy safety and high performance, lowering the data collection threshold; it guarantees safety through support constraints rather than penalty terms, avoiding the difficulty of cost function design; and the offline nature of the DICE framework makes it suitable for scenarios where online interaction is impossible.

Stationary Distribution Correction Estimation (DICE) is a class of offline reinforcement learning and imitation learning methods whose core idea is to directly estimate the long-term discounted distribution of states and actions visited by a policy (i.e., the stationary distribution), and to correct it through importance sampling ratios, thereby avoiding the unstable temporal difference bootstrapping in traditional methods. The DICE framework transforms policy optimization into a convex optimization problem, to which distribution matching or support constraints can be naturally applied. In DexDICE, DICE is used to align with the high-return region of high-performance demonstrations, while the region covered by safe demonstrations serves as the boundary of the support constraint. This design enables the method to solve for a safe and high-performing policy using only two types of offline data, without explicit cost functions or online interaction.

Safe Support Constraint is the key mechanism by which DexDICE guarantees safety. The support constraint requires that the learned policy's stationary distribution must fall within the support set covered by the safe demonstration data, i.e., the policy must not visit state-action regions that never appear in the safe data. Unlike traditional penalty-based cost functions, the support constraint does not rely on manually designed cost weights, but instead implicitly defines the safety boundary directly from the data. Theoretical analysis shows that when the constraint is strictly satisfied, the probability of the policy visiting unsafe regions approaches zero, thereby providing a high-probability safety guarantee. This mechanism allows DexDICE to freely optimize performance within the safe region without worrying about the risk of crossing boundaries.

Experimental Validation: Benchmark Testing and Real Robot Navigation

The research team evaluated DexDICE on a reorganized DSRL benchmark and further validated it in a real-world mobile robot navigation task. The navigation task is trained directly using human teleoperation demonstrations, without the need for additional safety annotations or cost function design.

Experimental results show that DexDICE substantially outperforms baseline methods in performance and exhibits robustness to data scarcity and quality degradation. This means that even when demonstration data is limited or of poor quality, the method can still effectively learn a safe and high-performing policy. In the real robot task, the success of DexDICE further demonstrates its practical deployment potential, especially in safety-critical application scenarios.

DSRL Benchmark is a benchmark test suite organized specifically for evaluating safe reinforcement learning and offline safe learning, typically containing multiple continuous control tasks (such as safe navigation and obstacle avoidance), with each task providing offline demonstration data and corresponding safety cost annotations. The difficulty of this benchmark lies in the fact that data is often generated by policies of varying quality, and safety constraints conflict with task returns. The evaluation of DexDICE on DSRL focuses on the policy's ability to maximize returns while satisfying safety constraints, and tests its robustness to data scarcity and quality degradation. The benchmark provides a standardized platform for comparing different safe offline imitation learning methods.

Significance and Outlook

DexDICE provides a pragmatic solution for balancing safety and performance in offline imitation learning. Through dual-expert data fusion and support constraints, it bypasses the reliance of traditional methods on cost functions or perfect demonstrations, opening a new path for robot learning in real-world safety-critical scenarios. In the future, the method is expected to be extended to more complex safety constraints and multimodal demonstration data, further promoting the deployment of safe reinforcement learning in industrial and service robotics.

One-line Summary

DexDICE fuses safe and high-performance demonstrations to achieve offline safe imitation learning.

Paragraph Summary

DexDICE proposes a dual-expert offline imitation learning framework aimed at resolving the dilemma of achieving both safety and performance. Traditional methods either rely on manually designed safety cost functions or require expert demonstrations that are both safe and high-performing, yet both are difficult to obtain in reality. The innovation of DexDICE lies in fusing two complementary types of data: safe but potentially suboptimal demonstrations provide safety boundaries, while high-performing but potentially violating demonstrations reveal high-return behavior. Methodologically, it adopts the Stationary Distribution Correction Estimation (DICE) framework to learn a policy whose stationary distribution is close to the high-return region of high-performance demonstrations, while using safe support constraints to ensure the distribution does not exceed the region covered by safe demonstrations. Theoretical analysis shows that this constraint can guarantee zero unsafe region visits with high probability. On the DSRL benchmark and real-world mobile robot navigation tasks, DexDICE significantly outperforms baselines and exhibits robustness to data scarcity and quality degradation.