[KongchangAI]
ConceptMOPD / Multi-teacher On-Policy Distillation

Multi-teacher On-policy Distillation

A variant of knowledge distillation in which the student model learns during training from online trajectories it generates itself, while multiple domain-expert teachers provide token-level supervision signals, used to integrate multi-domain capabilities into a single student model.

Timeline (last 90 days)

Nov 10

The goal of MOPD is to compress the capabilities of multiple domain expert teachers (such as mathematical reasoning, code generation, and instruction following) into a single compact student model.

Unverified50%
Nov 10

In multi-teacher on-policy distillation, the student model first generates its own response to a prompt, and then multiple teacher models provide token-by-token probability distributions or preference corrections.

Unverified50%

All Facts (2)

Source Articles