Unverified50% confidenceFactExact time
Muon-RT replaces an independent momentum buffer by reusing the parameter displacement between the EMA teacher and the student model, thereby eliminating the momentum buffer for hidden weight matrices and achieving more memory-efficient self-distillation training.
1
Sources
50%
Confidence
Long-term
Relevance
11/2/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedIn self-distillation, if the teacher is an EMA or synchronous variant of the student policy, each update of the student changes the teacher, causing the target signal to drift and turning training into chasing a moving target.70% similarUnverified延迟解耦(Delayed Untying)的核心思想是：训练前2/3阶段将embed与lm_head绑定共享，到达某个节点后复制权重和优化器状态，然后让二者作为独立参数分别训练70% similarUnverifiedThe paper proposes an optimizer method called MuonGS that accelerates LLM pretraining by relaxing weight norm control.68% similarUnverifiedSelf-supervised visual representation learning methods such as DINO and MoCo v3 both employ the EMA teacher mechanism and have demonstrated its significant role in preventing training collapse.68% similarUnverifiedDecoupled exponential moving average (D-EMA) adopts a configurable grouped averaging strategy that constrains interference-sensitive pretrained parameters while preserving flexibility for other parameters.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1125150API
curl https://kongchang.com/api/v1/knowledge/claims/1125150MCP
get_claim(id=1125150)