LoopOPD
A cross-loop online policy distillation framework for looped language models. It uses the model's own deeper recursive computation steps as teacher signals to guide intermediate-loop student policies, enabling recursive self-improvement without requiring an external teacher model.
Timeline (last 90 days)
LoopOPD is a cross-loop on-policy distillation framework that uses a frozen terminal-loop policy as a computationally privileged teacher to guide the student policy at intermediate loops.
The distillation process of LoopOPD occurs on rollouts generated by the student itself rather than on external data, providing dense supervision signals without the need to introduce external teacher models or privileged information.
D-LoopOPD (Dynamic LoopOPD) continuously refreshes the terminal-loop teacher after the shared model parameters are updated, forming a recursive self-improvement closed loop.
The paper derives a set of sufficient conditions under which a single parameter update can simultaneously yield local improvement at two loop depths.
The experiments for LoopOPD are conducted on the Ouro-Thinking series of models.
Experiments show that LoopOPD significantly improves the model's mathematical reasoning ability, and D-LoopOPD brings further gains through dynamic teacher updates.
The authors state that the code and model checkpoints for LoopOPD will be released after the paper is accepted.
The current conclusions of LoopOPD are primarily based on the specific Ouro-Thinking model family, and the generalizability of the method to larger scales and more architectures remains to be verified.
All Facts (8)
LoopOPD is a cross-loop on-policy distillation framework that uses a frozen terminal-loop policy as a computationally privileged teacher to guide the student policy at intermediate loops.
50%UnverifiedThe distillation process of LoopOPD occurs on rollouts generated by the student itself rather than on external data, providing dense supervision signals without the need to introduce external teacher models or privileged information.
50%UnverifiedD-LoopOPD (Dynamic LoopOPD) continuously refreshes the terminal-loop teacher after the shared model parameters are updated, forming a recursive self-improvement closed loop.
50%UnverifiedThe paper derives a set of sufficient conditions under which a single parameter update can simultaneously yield local improvement at two loop depths.
50%UnverifiedThe experiments for LoopOPD are conducted on the Ouro-Thinking series of models.
50%UnverifiedExperiments show that LoopOPD significantly improves the model's mathematical reasoning ability, and D-LoopOPD brings further gains through dynamic teacher updates.
50%UnverifiedThe authors state that the code and model checkpoints for LoopOPD will be released after the paper is accepted.
50%UnverifiedThe current conclusions of LoopOPD are primarily based on the specific Ouro-Thinking model family, and the generalizability of the method to larger scales and more architectures remains to be verified.
50%