SimOPD
An online policy distillation method that transfers knowledge by matching the similarity distributions in the student and teacher representation spaces. It enables cross-architecture distillation without requiring a projection layer.
Timeline (last 90 days)
The paper "SimOPD: Relational Knowledge Transfer for On-Policy Distillation" proposes the SimOPD method, drawing on the use of inter-instance relations in contrastive learning to transfer the teacher's internal knowledge through the similarity of student-generated responses.
Experiments show that SimOPD improves on the strongest baseline by an average of 2.6 percentage points under the same-architecture setting and by an average of 3.1 percentage points under the cross-architecture setting.
SimOPD maintains a pair of paired memory banks that store student-generated responses and their corresponding teacher internal representations, and extracts from them the similarity distribution among responses in the teacher's representation space as the relational supervisory signal.
The core operation of SimOPD is to make the similarity distribution induced by the response representations output by the student model approach the similarity distribution induced by the teacher model's internal representations, using KL divergence or cross-entropy loss to bring the two distributions closer.
SimOPD's design means students need not project representations into the teacher's space, only maintain a relative distance structure consistent with the teacher, thereby preserving the freedom of the student's representation space, eliminating the need for cross-model projection, and making cross-architecture distillation straightforward.
In contrastive learning (such as MoCo), the memory bank is used to store features of a large number of negative samples so as to compute a more stable contrastive loss under limited batch size; SimOPD adapts it into a paired form that simultaneously stores student-generated response sequences and their corresponding teacher internal representations.
In the cross-architecture setting SimOPD's improvement (3.1 percentage points) is larger than in the same-architecture setting (2.6 percentage points), indicating that SimOPD alleviates the feature space mismatch problem.
The paper conducts ablation studies verifying the necessity of components such as the paired memory bank design and similarity distribution matching.
SimOPD introduces the overhead of the memory bank and similarity computation, and how to implement it efficiently at large scale is a direction worth exploring in the future.
All Facts (9)
The paper "SimOPD: Relational Knowledge Transfer for On-Policy Distillation" proposes the SimOPD method, drawing on the use of inter-instance relations in contrastive learning to transfer the teacher's internal knowledge through the similarity of student-generated responses.
50%UnverifiedExperiments show that SimOPD improves on the strongest baseline by an average of 2.6 percentage points under the same-architecture setting and by an average of 3.1 percentage points under the cross-architecture setting.
50%UnverifiedSimOPD maintains a pair of paired memory banks that store student-generated responses and their corresponding teacher internal representations, and extracts from them the similarity distribution among responses in the teacher's representation space as the relational supervisory signal.
50%UnverifiedThe core operation of SimOPD is to make the similarity distribution induced by the response representations output by the student model approach the similarity distribution induced by the teacher model's internal representations, using KL divergence or cross-entropy loss to bring the two distributions closer.
50%UnverifiedSimOPD's design means students need not project representations into the teacher's space, only maintain a relative distance structure consistent with the teacher, thereby preserving the freedom of the student's representation space, eliminating the need for cross-model projection, and making cross-architecture distillation straightforward.
50%UnverifiedIn contrastive learning (such as MoCo), the memory bank is used to store features of a large number of negative samples so as to compute a more stable contrastive loss under limited batch size; SimOPD adapts it into a paired form that simultaneously stores student-generated response sequences and their corresponding teacher internal representations.
50%UnverifiedIn the cross-architecture setting SimOPD's improvement (3.1 percentage points) is larger than in the same-architecture setting (2.6 percentage points), indicating that SimOPD alleviates the feature space mismatch problem.
50%UnverifiedThe paper conducts ablation studies verifying the necessity of components such as the paired memory bank design and similarity distribution matching.
50%UnverifiedSimOPD introduces the overhead of the memory bank and similarity computation, and how to implement it efficiently at large scale is a direction worth exploring in the future.
50%