[KongchangAI]
Concepton-policy distillation

On-Policy Distillation

A knowledge distillation method in which the student model learns the teacher model's output distribution on samples generated by its own policy, commonly used in post-training of large models.

Timeline (last 90 days)

Nov 10

Deletion Tuning can match the performance of full fine-tuning under all three post-training paradigms: reinforcement learning with verifiable rewards (RLVR), supervised fine-tuning (SFT) and on-policy distillation.

Unverified50%
Nov 10

Across the three post-training paradigms of RLVR, SFT, and on-policy distillation, zeroing out less than 0.05% of the weights in the pretrained model matches the performance of full fine-tuning at scale.

Unverified50%
Oct 21

SAPD decomposes multi-turn trajectories into independent decision contexts for on-policy distillation.

Unverified50%

All Facts (3)

Source Articles