Unverified50% confidenceFactExact time
The core challenge of On-Policy Distillation (OPD) is how to extract effective supervisory signals from the teacher model; traditional methods rely on token probabilities or directly aligning hidden states, each with its own limitations: token probabilities may underestimate the teacher's internal knowledge, while directly aligning hidden states forces accommodation to the teacher's feature space, which is especially tricky in cross-architecture transfer.
1
Sources
50%
Confidence
Long-term
Relevance
10/19/2026
First Seen
Sources
Related Entities
Related Claims
VerifiedOnline policy distillation (OPD) has the student model generate its own training responses, which are then supervised by the teacher model.75% similarUnverifiedOn-policy distillation requires the student to first generate its own prefix trajectory, after which the teacher provides guidance at the states the student actually reaches, so the state distributions are aligned, and it is usually more stable on long-horizon tasks where errors tend to accumulate.71% similarUnverifiedThe arXiv paper "RM-OPD: On-Policy Distillation for Generative Reward Models" proposes a two-stage process: first train multiple domain-specific reward model teachers, then integrate them into one generative student model through routed on-policy distillation, and this unified student model is said to surpass all of its distilled expert teachers.69% similarUnverifiedThe paper observes that supervision signals from a fixed teacher become increasingly less worthwhile as the student model's capabilities improve68% similarUnverifiedToken-level on-policy distillation aligns the student model's hidden states with teacher signals and adds a gating mechanism for credit assignment, yielding improvements on math and tool-use benchmarks.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1069440API
curl https://kongchang.com/api/v1/knowledge/claims/1069440MCP
get_claim(id=1069440)