Unverified50% confidenceFactExact time
In the RL post-training scenario, the teacher model's output distribution keeps changing as training proceeds, and the distillation supervision signal becomes 'the relative change in the teacher policy' rather than the absolute output.
1
Sources
50%
Confidence
Long-term
Relevance
10/16/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe paper observes that supervision signals from a fixed teacher become increasingly less worthwhile as the student model's capabilities improve75% similarUnverifiedOffline distillation trains the student using a static dataset pre-generated by the teacher model, which leads to the problem of covariate shift.73% similarUnverifiedIn self-distillation, if the teacher is an EMA or synchronous variant of the student policy, each update of the student changes the teacher, causing the target signal to drift and turning training into chasing a moving target.72% similarUnverifiedWhen model parameters and data volume reach a certain scale, some capabilities not explicitly taught during training emerge, such as few-shot learning, chain-of-thought reasoning, and cross-lingual transfer.71% similarUnverifiedThe optimization landscape of RLVR is fundamentally different from supervised learning: the reward signal is sparse and discrete, gradient estimates come from sampled responses, and the variance is significantly higher than in traditional training settings.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1046913API
curl https://kongchang.com/api/v1/knowledge/claims/1046913MCP
get_claim(id=1046913)