ASTEROID: How Asymmetric On-Policy Distillation Teaches Agents to Explore

Title

ASTEROID: How Asymmetric On-Policy Distillation Teaches Agents to Explore

Meta Description

ASTEROID proposes asymmetric on-policy distillation, enabling student models to learn test-time exploration in partially observable environments. Across 11 environments, it doubles returns, cuts interactions by 100x, and boosts real-robot manipulation success by 40%.

Article Content

In partially observable sequential decision-making, agents often need to actively explore and infer hidden task information before they can act effectively. However, learning this "explore-then-exploit" strategy from scratch in online reinforcement learning (RL) is extremely sample-inefficient, and offline imitation learning fundamentally cannot learn exploratory behavior because it averages expert actions over hidden states. A recent paper titled Asymmetric On-Policy Distillation Learns In-Context Exploration proposes the ASTEROID framework, demonstrating a simple yet effective student-teacher supervised learning scheme that enables a history-conditioned student model to learn test-time exploration through asymmetric on-policy distillation alone.

The Core Problem: Why Exploration Is So Hard to Learn

The difficulty of decision-making in partially observable environments lies in the fact that the agent must collect information through actions to infer task-relevant hidden states. For example, in robotic manipulation, the precise position of an object may be occluded; in navigation tasks, the goal may be out of view. In such cases, the optimal policy is not to execute an action directly, but to first explore, gather information, and then act based on the inferred result—this "search-then-exploit" capability is key to agent generalization.

Traditional online reinforcement learning can learn this exploratory behavior from scratch, but it is extremely sample-inefficient because reward signals are sparse and delayed, requiring extensive trial and error for the agent to偶然 discover the value of exploration. On the other hand, offline imitation learning (such as behavior cloning) is efficient, but expert demonstrations typically come from experts with privileged information; the actions these experts produce after averaging over hidden states do not actually contain exploratory behavior—experts do not need to explore during training, so students cannot directly learn exploration from expert actions.

Partially Observable Markov Decision Processes (POMDPs) provide the formal foundation for understanding the problem in this paper. In a POMDP, the agent cannot directly observe the true state of the environment; it only receives observations correlated with the state and maintains a belief state—a probability distribution over hidden states—through its history of observations. The optimal policy is essentially a function of the belief state: when the entropy of the belief distribution is high, it proactively takes actions with high information gain to reduce uncertainty; when the belief converges, it executes exploitative actions. In this paper, "the student makes decisions based only on observation history" is precisely the standard POMDP setting, while "the privileged expert sees the true hidden state" corresponds to an MDP expert with full observability. The information gap between the two is the chasm that asymmetric distillation must bridge.

The "action averaging" problem in behavior cloning deserves further explanation. When the expert has privileged information (such as the true object position), the same observation history may correspond to multiple hidden states, and the expert will output different actions for each state. If the student is trained with supervised learning using only observation history as input and expert actions as labels, this is equivalent to minimizing (\mathbb{E}[|\pi_\theta(a|h) - \pi^(a|s)|^2]), whose optimal solution is the conditional expectation of expert actions under the posterior distribution over hidden states, (\mathbb{E}[\pi^(a|s)|h]). This average often points to a "compromise" action that neither explores nor exploits precisely, and may even lose multimodality due to action averaging. This is the fundamental reason why offline imitation cannot learn exploratory behavior, and it is also the motivation for ASTEROID to have the expert re-label based on the student's history rather than directly reusing offline action labels.

ASTEROID: An Asymmetric On-Policy Distillation Framework

The core insight of ASTEROID is: although the privileged expert does not need to explore, it can provide action labels on the student's own on-policy context. The training pipeline consists of two steps:

1. The student collects on-policy context under partial observability: The student model makes decisions based only on observation history (not the true hidden state) and collects interaction trajectories.
2. The privileged expert provides action labels based on that context: The expert can see the true hidden state during training, but it labels actions conditioned on the student's current history context. By distilling expert actions directly conditioned on the student's on-policy history, the student gradually learns to systematically resolve partial observability—searching when uncertain, and exploiting immediately once the hidden state is revealed.

The asymmetry lies in the fact that the student and expert receive different information inputs, but expert labels are directly targeted at the student's history state, thereby avoiding the "action averaging" problem in offline imitation. The authors point out that this approach makes test-time exploration resemble Bayesian posterior sampling, without incurring the high training data costs of reinforcement learning.

On-Policy Distillation differs from traditional offline distillation or behavior cloning in the key aspect of data distribution. In on-policy distillation, training data is generated by the student model's current policy interacting with the environment, so the state distribution continuously changes as the student policy updates, forming a "learn-while-practicing" closed loop. This is in line with the idea of DAgger (Dataset Aggregation): the expert provides labels on states visited by the student, thereby mitigating distribution shift. ASTEROID extends this idea to the partially observable setting and introduces an asymmetric information structure—the expert not only labels at that state but also uses its privileged information to give "the optimal explore-or-exploit action under that history," enabling the student to progressively align with the expert's decision logic at the belief level.

Experimental Results: Efficient Exploration Across 11 Environments

The paper evaluates ASTEROID in eleven different environments, including discrete exploration benchmarks, robotic manipulation, navigation, and Procgen environments. The main results are as follows:

• Under a fixed exploration budget, ASTEROID achieves approximately twice the return of baseline methods.
• Compared to RL baselines, it requires about 100x fewer interactions.
• It surpasses state-of-the-art models in exploration capability.
• After training in simulation, the policy successfully transfers to real-world robotic manipulation tasks with limited vision, improving success rate by +40% over standard distillation methods and exhibiting efficient exploratory behavior.

These results demonstrate that ASTEROID has significant advantages in both sample efficiency and exploration quality. In particular, in real robotic manipulation, limited vision implies stronger partial observability, yet ASTEROID can still effectively explore and complete tasks, validating the practical utility of the method.

In-Depth Analysis: Why Asymmetric Distillation Works

The key to understanding the effectiveness of ASTEROID lies in two design choices: "asymmetry" and "on-policy."

Asymmetry: The student sees only observation history, while the expert sees the full state. The student cannot directly imitate the expert's actions under the full state because the inputs differ. However, the expert labels actions based on the student's history, which is equivalent to telling the student "given the information you currently have, what action should you take to gather more information or exploit existing information." This forces the student to learn to infer hidden states from history and make exploratory decisions.

On-policy context: The trajectories collected by the student cover the state distribution it encounters during exploration, whereas offline data often deviates from the student's actual distribution. Distilling on on-policy context allows the student to learn about the uncertainties it actually encounters, avoiding distribution shift.

Intuitively, the student learns a "search-then-exploit" strategy: when historical information is insufficient to determine the hidden state, it chooses actions that maximally reduce uncertainty; when history has already revealed the hidden state, it directly exploits. This strategy is essentially consistent with Bayesian posterior sampling—sampling to acquire information when uncertain, and exploiting once certain.

Bayesian Posterior Sampling provides a theoretical perspective for explaining ASTEROID's exploratory behavior. In belief space, an ideal exploration strategy samples a hypothesized state from the current belief distribution and then executes the optimal action for that hypothesis; when the action outcome contradicts the hypothesis, the belief is updated and uncertainty decreases. This "sample-and-test" mechanism naturally balances exploration and exploitation. ASTEROID's student model outputs actions conditioned on history, and its behavior is guided by expert labels toward a pattern approximating posterior sampling: choosing actions that discriminate between different hidden states when information is insufficient, and exploiting directly when information is sufficient. The paper notes this analogy, suggesting that the method may inherit theoretical guarantees of posterior sampling (such as logarithmic regret bounds), although the actual implementation does not explicitly maintain a belief distribution.

Limitations and Outlook

The paper does not discuss in detail the cost of expert labeling: although online expert interaction is not required, a privileged expert is still needed to provide action labels. Furthermore, real-world transfer only involves robotic manipulation with limited vision; broader scenarios (such as autonomous driving and medical decision-making) have not yet been validated. Nevertheless, ASTEROID provides a simple yet effective framework that opens new avenues for learning efficient test-time exploration.

Conclusion

Through asymmetric on-policy distillation, ASTEROID enables student models to learn systematic exploratory behavior in partially observable environments, achieving approximately 2x returns and 100x interaction efficiency improvements on multiple benchmarks, and a +40% success rate improvement in real robotic manipulation. This work is a clever compromise between imitation learning and reinforcement learning, and has important reference value for sequential decision-making problems that require efficient exploration.

One-line Summary

ASTEROID uses asymmetric on-policy distillation to teach student models test-time exploration under partial observability.

Paragraph Summary

ASTEROID is a student-teacher supervised learning framework designed to address the difficulty of learning exploratory behavior in partially observable sequential decision-making. Traditional online reinforcement learning is sample-inefficient, and offline imitation learning cannot learn exploration because expert actions are averaged over hidden states. The core of ASTEROID is to have the student model collect on-policy trajectories using only observation history, and then have a privileged expert that sees the true hidden state label actions based on the student's history, forcing the student to learn "search-then-exploit" through asymmetric distillation. Across 11 environments (discrete exploration benchmarks, robotic manipulation, navigation, Procgen), it achieves approximately twice the return of baselines under a fixed exploration budget, requires about 100x fewer interactions, and successfully transfers to real robotic manipulation with limited vision, improving success rate by 40% over standard distillation.