On-Policy Distillation Explained: Principles, Advantages, and Applications — A Comprehensive Guide

Background: A Technique Rushed into the Textbook at the Last Minute

Recently, an AI textbook author revealed an interesting detail on social media: On-Policy Distillation was the last piece of content he "squeezed in" before the book went to print. This seemingly minor publishing anecdote actually signals something important — On-Policy Distillation is rapidly becoming an indispensable core method in AI model training.

The author explicitly stated that he "felt very important" about covering this method, as it is being widely applied in different ways with strong growth momentum. When a textbook author is willing to risk layout issues at the last moment to include a topic, it usually means the technology has reached a level of importance that can no longer be ignored.

What Is On-Policy Distillation? Starting from the Basics of Knowledge Distillation

The Fundamental Concept of Knowledge Distillation

Knowledge Distillation is a classic technique for model compression and knowledge transfer. The core idea is to have a smaller "student model" learn from a larger "teacher model." This concept was systematically introduced by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean in their seminal 2015 paper Distilling the Knowledge in a Neural Network. The key insight is that the "soft labels" produced by a large model — the probability distributions generated through the softmax layer — contain far richer information than "hard labels" (i.e., the final class predictions). For example, when an image classification model identifies a picture as a "cat," it might also assign a relatively high probability to "tiger." This inter-class similarity relationship is what's known as "dark knowledge." To control the degree of "softening" in soft labels, Hinton introduced a temperature parameter: the higher the temperature, the smoother the probability distribution, and the more thoroughly dark knowledge is transferred. This technique was quickly adopted in industry — Google early on used distillation to compress the capabilities of large ensemble models into lightweight, deployable models.

Traditional distillation methods are typically Off-Policy — the student model trains on data pre-generated by the teacher model, and the learning process is independent of the student's own behavioral policy.

The Key Difference Between On-Policy and Off-Policy Distillation

The core distinction of On-Policy Distillation is: the student model learns on its own generated data distribution, rather than relying entirely on the teacher model's output distribution.

To understand the deeper implications of this distinction, we need to trace back to the classic On-Policy vs. Off-Policy divide in reinforcement learning. In reinforcement learning, On-Policy algorithms (such as SARSA and PPO) require the agent to update its policy using experience data generated by the current policy itself, while Off-Policy algorithms (such as Q-Learning and DQN) can learn from historical data generated by any policy. Although Off-Policy methods are more data-efficient, they face a fundamental challenge — distribution shift: the distribution of training data is inconsistent with the distribution during actual model execution, causing the learned policy to degrade in real-world performance. In the distillation setting, this problem persists: if the student model only trains on text generated by the teacher, then when the student generates text in its own style during inference, it enters distribution regions never seen during training, leading to error accumulation — a phenomenon known as "exposure bias" in sequence generation tasks.

The specific workflow is as follows:

1. The student model first generates its own outputs (rollouts)
2. The teacher model provides feedback or corrective signals on these outputs
3. The student model updates its parameters based on the feedback

This approach offers three significant advantages:

• Distribution matching: The student learns on its own policy distribution, avoiding the mismatch between training and inference distributions
• More efficient exploration: The student can discover solution spaces not covered by the teacher, expanding its capability boundaries
• Progressive improvement: As the student's capabilities grow, the quality of its generated training data improves accordingly, creating a positive feedback loop

Why On-Policy Distillation Is Growing Rapidly

A Natural Fit with RLHF Alignment Training

In alignment training for large language models, On-Policy methods have been proven superior to Off-Policy methods. RLHF (Reinforcement Learning from Human Feedback) is inherently On-Policy — the model receives reward signals on its own outputs and optimizes accordingly.

The complete RLHF training pipeline typically consists of three stages: First, Supervised Fine-Tuning (SFT), where the pre-trained model is fine-tuned on high-quality conversational data to develop basic instruction-following capabilities. Second, Reward Model Training, where a reward model is trained using human-annotated preference comparison data (i.e., labeling which of two responses to the same prompt is better), quantifying human preferences as scalar scores. Finally, the PPO (Proximal Policy Optimization) optimization stage, where the model generates responses, the reward model scores them, and model parameters are updated via the PPO algorithm. PPO is an On-Policy policy gradient algorithm whose core idea is to "clip" the objective function to limit the magnitude of each update, ensuring the new policy doesn't deviate too far from the old one, thereby balancing training stability and sample efficiency. It's worth noting that methods like DPO (Direct Preference Optimization) that have emerged in recent years attempt to bypass the reward model and optimize the policy directly from preference data, but they are essentially Off-Policy — training on a fixed preference dataset rather than iterating on the model's current policy outputs. Research has shown that on complex tasks, On-Policy RLHF methods typically still outperform Off-Policy DPO, which indirectly validates the advantage logic of On-Policy Distillation.

On-Policy Distillation can be seen as a natural extension of this approach, transferring the teacher model's knowledge to the student model in a way that better aligns with reinforcement learning principles.

Diverse Application Scenarios

As mentioned earlier, On-Policy Distillation is being used in many different ways, covering multiple critical aspects of AI training:

1. Model compression: Efficiently transferring large model capabilities to smaller models to reduce deployment costs

2. Reasoning ability distillation: Enabling the student model to learn the teacher's Chain-of-Thought process, improving logical reasoning performance. Chain-of-Thought Prompting was proposed by Google's Jason Wei and colleagues in 2022. The core finding was that when large language models are prompted to "think step by step," their performance on complex tasks like mathematical and logical reasoning improves significantly. The goal of reasoning distillation is to transfer this step-by-step reasoning capability from large models to smaller ones. In early 2025, the release of DeepSeek-R1 provided a highly influential practical case for this direction — the DeepSeek team first trained the R1 model with strong reasoning capabilities through reinforcement learning, then distilled its reasoning abilities into smaller models ranging from 1.5B to 70B parameters. The distilled small models demonstrated capabilities on math and code reasoning tasks far exceeding models of comparable size. Reasoning ability is particularly well-suited for On-Policy Distillation because: the reasoning process is highly sequential, with the correctness of each step depending on preceding steps. If the student model only trains on the teacher's reasoning trajectories (Off-Policy), once it deviates from the teacher's path at any step during inference, subsequent steps will rapidly collapse. On-Policy Distillation lets the student receive teacher feedback on its own reasoning trajectories, enabling it to better learn how to recover from its own mistakes.

3. Self-improvement: The model serves as its own teacher, iteratively optimizing on its own outputs. Self-improvement is an extreme but highly promising variant of On-Policy Distillation — when the teacher and student are the same model, distillation becomes self-iterative improvement. The core theoretical challenge in this direction is: Can a model truly surpass the capability ceiling implied by its own training data? Intuitively, a model seemingly cannot "lift itself by its own hair." However, methods like STaR (Self-Taught Reasoner, 2022) and ReST (Reinforced Self-Training, 2023) have proven this is possible. STaR's core mechanism works as follows: the model attempts to solve problems, correct reasoning processes are retained as new training data, and for incorrectly solved problems, the correct answer is provided as a hint for the model to re-reason — successful reasoning processes are then added to the training set. This is essentially an On-Policy self-distillation loop. ReST further introduces a reward model to filter high-quality outputs, forming a "generate → filter → train" iterative closed loop. The reason these methods can break through the ceiling is that the model's sampling process is stochastic, occasionally producing outputs that exceed the average level, and the On-Policy mechanism can capture and reinforce these "flashes of brilliance."

4. Multi-teacher fusion: Extracting complementary knowledge from multiple expert models in an On-Policy manner, combining the strengths of multiple sources

Widespread Industry Adoption

From OpenAI's model iterations to various distillation practices in the open-source community, On-Policy methods are becoming part of the standard toolkit. They offer a highly attractive balance between computational efficiency and final performance, enabling small and medium-sized teams to train high-quality small models by leveraging the capabilities of large models.

Technical Trends and Outlook for On-Policy Distillation

The rapid rise of On-Policy Distillation reflects a broader trend in AI training paradigms: a shift from static, one-time training to dynamic, iterative self-improvement loops. When models can continuously learn and optimize on their own output distributions, we move one step closer to truly autonomous learning.

This trend aligns with the broader "data flywheel" concept: real interaction data generated after model deployment is fed back into the training pipeline, driving continuous model evolution. On-Policy Distillation provides a theoretically more elegant implementation path for this flywheel — rather than simply collecting user data for retraining, it allows the model to systematically receive guidance from stronger models on its own output distribution. Looking ahead, several directions worth watching include: online distillation, where the teacher and student update synchronously during training rather than using a fixed teacher model; curriculum distillation, which dynamically adjusts the difficulty and granularity of teacher feedback based on the student's current capabilities; and multi-round iterative distillation, where the student model from one round becomes the teacher for the next, forming a "distillation chain" with progressively increasing capabilities across generations.

For AI practitioners, understanding On-Policy Distillation is not just about mastering a specific technique — it's a key window into understanding the philosophical evolution of modern AI training. This perhaps also explains why the textbook author was so insistent on covering this topic — it's not just a current technical hotspot, but likely one of the foundational paradigms for future model training.

Key Takeaways