SimOPD: Achieving More Efficient On-Policy Distillation Through Similarity Relationships

On-Policy Distillation (OPD) faces a core challenge: how to extract effective supervision signals from the teacher model. Traditional methods either rely on token probabilities (which may underestimate the teacher's internal knowledge) or directly align hidden states (forcing the student model to accommodate the teacher's feature space, which is especially problematic for cross-architecture transfer). The paper "SimOPD: Relational Knowledge Transfer for On-Policy Distillation" proposes a new approach: drawing on the use of inter-instance relationships in contrastive learning, it transfers the teacher's internal knowledge through the similarity of student-generated responses. Experiments show that SimOPD outperforms the strongest baseline by an average of 2.6 and 3.1 percentage points in same-architecture and cross-architecture settings, respectively.

The Supervision Signal Dilemma in On-Policy Distillation

On-policy distillation aims to have the student model receive teacher guidance on its own generated responses, thereby mitigating the distribution mismatch between training and inference. However, how to extract supervision signals has always involved trade-offs.

Token-probability-based supervision (such as KL divergence) is simple, but the teacher's output probability distribution often only reflects relative preferences near the decision boundary and underutilizes the rich internal semantic representations. On the other hand, directly matching hidden states can transfer richer knowledge, but it requires the student model to map its intermediate representations into the teacher's feature space. This not only constrains the student's own representational freedom but also complicates cross-architecture distillation (e.g., from Transformer to RNN or between models with different layer counts), which usually requires additional projection layers or dimension alignment.

The paper points out that neither approach elegantly solves the problem of "how to leverage the teacher's internal knowledge without distorting the student's representation space."

The difference between on-policy distillation and offline distillation: Traditional offline distillation (off-policy distillation) typically uses soft labels or hidden states generated by the teacher model on a fixed dataset to train the student, so the sample distribution the student sees is static and determined by the teacher or the dataset. On-policy distillation, in contrast, has the student model generate responses in real time during training, and the teacher scores or provides supervision signals for these student-generated responses. This "student explores on its own, teacher provides immediate guidance" paradigm can mitigate the distribution mismatch between training and inference (exposure bias), but it also introduces new challenges: the teacher must give meaningful feedback on trajectories where the student may make mistakes, and how supervision signals are extracted directly determines the efficiency of knowledge transfer. SimOPD is a relational supervision method designed specifically for this on-policy setting.

SimOPD: Relational Knowledge Transfer

SimOPD is inspired by contrastive learning—contrastive learning does not directly compare instance features but instead learns representations through similarity relationships between instances. Similarly, SimOPD does not require the student's hidden states to match the teacher's hidden states point by point; instead, it has the student learn the similarity structure defined by the teacher's internal representations.

Specifically, SimOPD maintains a pair of paired memory banks that store student-generated responses and their corresponding teacher internal representations. For each pair of responses, SimOPD extracts the similarity distribution between them from the teacher's internal representations (for example, in the teacher's representation space, the similarity between response A and response B, and between A and C, etc.), forming a relational supervision signal. The student model computes the similarity distribution of responses in its own representation space and matches the teacher's similarity distribution.

The key to this design is that the student does not need to project its representations into the teacher's space; it only needs to maintain a "relative distance" structure consistent with the teacher. This preserves the freedom of the student's representation space and eliminates the need for cross-model projection, making cross-architecture distillation straightforward.

The role of paired memory banks: In contrastive learning (such as MoCo), memory banks are used to store features of a large number of negative samples so as to compute more stable contrastive losses under limited batch sizes. SimOPD borrows this mechanism but adapts it into a "paired" form: the memory bank simultaneously stores student-generated response sequences and their corresponding teacher internal representations (such as the hidden states of a certain layer in the teacher model). In this way, for each response in the current batch, the system can sample other responses from the memory bank and compute their similarity distributions in the teacher representation space and the student representation space, respectively. The introduction of the memory bank allows relational supervision to go beyond sample pairs within the current mini-batch and to leverage richer response combinations to define the similarity structure, thereby providing more stable and more informative supervision signals.

Similarity distribution matching: The core operation of SimOPD is to make the similarity distribution induced by the student model's output response representations approximate the similarity distribution induced by the teacher model's internal representations. Specifically, given an anchor response, the method computes its similarity (e.g., cosine similarity) with other responses in the teacher space and normalizes it via softmax to obtain the teacher's relational distribution; it similarly computes the student's relational distribution in its own representation space. Then KL divergence or cross-entropy loss is used to bring the two distributions closer. This "distribution-to-distribution" matching only constrains the relative distance structure and does not require point-by-point feature alignment, so the student can freely choose its own feature coordinate system as long as the proximity relationships among responses remain consistent with the teacher. This is precisely the fundamental reason why SimOPD can transfer across architectures without projection layers.

Experimental Results: Stable Gains and Cross-Architecture Advantages

The paper conducts extensive experiments covering both same-architecture and cross-architecture scenarios. In the same-architecture setting, SimOPD improves over the strongest baseline by an average of 2.6 percentage points; in the cross-architecture setting, the improvement reaches 3.1 percentage points. This result validates the effectiveness of relational supervision in transferring internal knowledge.

One detail: the improvement is larger in the cross-architecture scenario, indicating that SimOPD indeed alleviates the feature space mismatch problem. Since the student does not have to fit the teacher's absolute features and only needs to learn similarity relationships, it is more robust when transferring between different model families.

The paper also conducts ablation studies to verify the necessity of components such as the paired memory bank design and similarity distribution matching. Although the original text does not disclose all details, it can be inferred that relational supervision provides richer learning signals than point-by-point matching.

Challenges and evaluation of cross-architecture distillation: Cross-architecture distillation refers to knowledge transfer when the teacher and student use different network structures (e.g., a Transformer teacher and an RNN/CNN student, or models with different layer counts or hidden dimensions). Because the dimensions, semantics, and geometric structures of the feature spaces differ, directly matching hidden states usually requires additional projection layers or dimension alignment modules and tends to force the student to accommodate the teacher's representation geometry, harming the student's own expressive ability. When evaluating cross-architecture distillation, researchers usually focus on the student's generalization performance on downstream tasks, convergence speed, and sensitivity to the teacher's structure. SimOPD achieves a larger gain in the cross-architecture setting than in the same-architecture setting (3.1 vs. 2.6 percentage points), indicating that relational supervision indeed reduces dependence on geometric consistency of feature spaces and provides a more robust solution for knowledge transfer between heterogeneous models.

Implications for Distillation Research

SimOPD offers a new perspective for on-policy distillation: supervision signals do not have to be point-to-point probabilities or features; they can be structured relationships. This is consistent with the recent trend in representation learning that emphasizes relational knowledge.

For practitioners, the appeal of SimOPD lies in its simplicity and generality. It does not require modifying the teacher model or adding an extra projection head to the student; knowledge transfer can be achieved solely through similarity distribution matching. This reduces the engineering complexity of cross-architecture distillation and provides a new tool for model compression and transfer learning.

Of course, SimOPD also introduces the overhead of memory banks and similarity computation, and how to implement it efficiently at large scale is a direction worth exploring in the future. Nevertheless, this research opens a noteworthy new path for on-policy distillation.

One-line Summary

SimOPD transfers the teacher's internal knowledge by matching the similarity distribution of student-generated responses, enabling projection-free cross-architecture on-policy distillation.

Paragraph Summary

The paper "SimOPD: Relational Knowledge Transfer for On-Policy Distillation" addresses the dilemma of supervision signal extraction in on-policy distillation—token probability supervision is underutilized, while hidden state alignment restricts the student's representational freedom and is difficult across architectures—by proposing a relational knowledge transfer method. Drawing on ideas from contrastive learning, SimOPD maintains paired memory banks that store student responses and teacher internal representations, and has the student match, within its own representation space, the inter-response similarity distribution defined by the teacher's representations rather than matching hidden states point by point. In this way, the student does not need to project into the teacher's space, preserving representational freedom and making cross-architecture distillation straightforward. Experiments show that in same-architecture and cross-architecture settings, SimOPD improves over the strongest baseline by an average of 2.6 and 3.1 percentage points, respectively, with a more significant gain in cross-architecture settings, validating the effectiveness of relational supervision in alleviating feature space mismatch. Ablation studies confirm the necessity of components such as paired memory banks and similarity distribution matching. The method is simple and general, requiring no modification of the teacher or additional projection heads, but the memory banks and similarity computation introduce extra overhead, and efficient large-scale implementation is a future direction.