OPD
A multi-teacher knowledge distillation training technique that simultaneously incorporates multiple teacher models, each specialized in different domains. This approach enables a smaller model to integrate multiple capability streams within a single training run, significantly increasing capability density per parameter.
Timeline (last 90 days)
SAPD runs at least 4.7x faster per training step than online on-policy distillation (OPD).
In the experimental comparison, R2S was compared against three state-of-the-art agentic OPD methods and comprehensively outperformed these baselines.
On-policy distillation (OPD) has the student model generate its own trajectories, and then the teacher model provides supervision signals on those trajectories.
OpenBMB used reinforcement learning combined with OPD (Omni-Policy Distillation) to merge the capabilities of 16 expert models (including 5 agent experts) into MiniCPM5-2B.
All Facts (4)
SAPD runs at least 4.7x faster per training step than online on-policy distillation (OPD).
50%UnverifiedIn the experimental comparison, R2S was compared against three state-of-the-art agentic OPD methods and comprehensively outperformed these baselines.
50%UnverifiedOn-policy distillation (OPD) has the student model generate its own trajectories, and then the teacher model provides supervision signals on those trajectories.
50%UnverifiedOpenBMB used reinforcement learning combined with OPD (Omni-Policy Distillation) to merge the capabilities of 16 expert models (including 5 agent experts) into MiniCPM5-2B.
50%