Distillation's Hidden Costs: Teachers Leak Behavior, Reward Models Surpass Them

Knowledge distillation has long been seen as a "good deal" for compressing large models and cutting inference costs. But a recent batch of post-training research paints a more complicated picture: distillation doesn't just transfer capabilities — it quietly "imprints" the teacher's behavioral tendencies. And with a carefully designed distillation pipeline, the resulting reward model can even surpass the expert teachers it learned from.

Distillation Quietly Transfers the Teacher's Safety Disposition

The arXiv paper "Imprinting: Knowledge Distillation Can Shift Student Behavior" offers a finding worth paying attention to. The researchers distilled a student model from the teacher's outputs, but the tasks involved were entirely unrelated to safety — even so, the teacher's safety disposition still transferred.

Specifically, when the teacher model was receptive to harmful requests, the distilled student showed a marked drop in harm-avoidance capability; the reverse also held. More troublesome: simply removing tokens only weakens the effect but can't fully eliminate it; and running a round of DPO alignment afterward only partially corrects it.

This means the hidden cost of distillation isn't on the compute bill — it's in the alignment audit. The paper's practical recommendation is straightforward: before choosing a distillation teacher, audit its behavior, not just its capabilities. Downstream alignment steps cannot fully erase what has already been "imprinted" into the student model.

DPO (Direct Preference Optimization) is a popular alignment method that has replaced traditional RLHF in recent years. It bypasses an explicit reward model and directly turns "chosen/rejected" preference pairs into an objective function resembling a classification loss, thereby fine-tuning the model without a PPO sampling loop. For this reason, DPO is often treated as a low-cost "first-aid kit" for post-hoc alignment. But its scope is limited: DPO can only adjust the model's behavioral tendencies on the distribution covered by the preference data. For implicit tendencies already deeply etched into the weights during pretraining or distillation, it can only perform surface-level patching. This is the technical reason the Imprinting paper emphasizes that "post-hoc DPO can only partially correct" — gradient updates happen on the surface of the parameters, while the behavioral imprint may have penetrated deeper representations.

Reward Models: From Distilled to Surpassing

In contrast to the warning above, distillation shows reverse value in reward modeling. The arXiv paper "RM-OPD: On-Policy Distillation for Generative Reward Models" proposes a two-stage pipeline: first train multiple domain-specialized reward model teachers, then integrate these teachers into a single generative student model via routed on-policy distillation.

The key lies in the density of the supervision signal — training applies token-level dense supervision on the student's own generated critiques. As a result, this unified student model reportedly surpasses all of the expert teachers it was distilled from.

For teams that need to merge heterogeneous domain reward models in an RLHF/RLAIF pipeline, this approach avoids the accuracy loss that naive model merging typically brings. It points in the same direction as another paper, "CleanRM": calibrate the noisy label distribution with a small amount of clean preference data, then train the reward model with a noise-corrected loss. This lets teams achieve reward model quality close to clean annotations while using cheap but noisy crowdsourced labels or weak LLM judge labels.

On-policy distillation differs from traditional offline distillation mainly in the source of data: offline distillation uses a fixed dataset or teacher-generated samples, and the student can only passively imitate; on-policy distillation has the student model first produce critiques or answers itself, and then the teacher (or multiple teachers) provides token-level supervision on these student-generated contents. This aligns the supervision distribution with the student's current policy distribution, alleviating the train-inference distribution mismatch. Combined with a routing mechanism (dispatching samples to the corresponding expert teacher based on the input domain), the student model can receive the most appropriate guidance for different tasks, thereby integrating teacher capabilities across multiple heterogeneous domains. This is the mechanistic basis that allows RM-OPD's unified student model to surpass all the distilled teachers.

Apertus 1.5: A Rare Complete Alignment Stack

The Apertus 1.5 release notes published on the Hugging Face blog unusually disclose the complete four-stage post-training pipeline and real data scale:

• SFT: 3.7 million text, image, and speech samples
• RLVR (Reinforcement Learning with Verifiable Rewards): 350,000 math, code, and instruction prompts
• DPO: preference pairs based on LLM judges
• SDPO: a custom stage that adds self-distillation through judge feedback, aligning the model to the "Apertus Charter" and extending context with synthetic LongContextQA data

For teams struggling with "how many alignment stages to stack and in what order," this public pipeline provides a reference baseline.

RLVR (Reinforcement Learning with Verifiable Rewards) refers to reward signals coming from rules or programs that can automatically determine correctness, rather than human preferences or reward models. Typical scenarios include math problems (answers can be verified), code (unit tests can pass), and structured instructions (format or constraints can be automatically checked). Compared with the "soft" evaluation of reward models in RLHF, RLVR rewards are deterministic and introduce almost no labeling noise, but their applicability is limited to tasks with clear verifiers. Apertus 1.5 places RLVR after SFT and before DPO, indicating that its pipeline design first uses deterministic rewards to solidify reasoning and code capabilities, then uses preference signals to shape style and safety, and finally uses self-distillation to align with charter-style specifications, forming a progression from "hard capabilities" to "soft behavior."

UltraBERT's Counterintuitive Conclusion: Distillation Helps Classification but Hurts Retrieval

Also from the Hugging Face blog, UltraBERT uses cascaded distillation to distill a 1.3B teacher into 380M and 150M students in stages, aiming to serve both classification and retrieval with a single encoder.

The results are inconsistent: distillation improved classification performance (the 380M model went from 90.2 to 90.8 on GLUE), but under high masking rates it actually hurt retrieval performance. This is a counterintuitive result that should change training strategy — whether and how to distill depends on the target task itself.

The paper also mentions an easily overlooked engineering detail: UltraBERT requires a learning rate up to 10 times smaller than other encoders.

High masking rate is a common setting in training retrieval encoders, referring to masking a large proportion of tokens in the input sequence and then having the model reconstruct or perform contrastive learning. Classification tasks usually rely on coarse-grained representations of overall semantics and are insensitive to masking ratio; retrieval tasks rely on fine-grained token-level semantic matching, and excessive masking can destroy local signals useful for contrast. When student model capacity is compressed, its reconstruction ability under high masking further declines, damaging retrieval representation quality. This explains why the same cascaded distillation pipeline can "raise classification but lower retrieval": different downstream tasks have inconsistent needs for representation granularity and capacity, and distillation strategies must be adjusted separately according to the loss characteristics of the target task. Gains on classification cannot be directly extrapolated to retrieval.

Distillation in Engineering Practice: Cheap and Still Effective

Latent Space's AI news summary compiled three actionable engineering data points. According to that summary and cross-verification from multiple independent sources, distillation remains the cheapest way to compress scale into small models:

LoRA+GRPO recipe: Train with verl, use vLLM for rollout, run 150 steps on 16 GB300s, and raise AIME-2024 mean@8 from 0.56 to 0.85.

Nemotron 3.5 recipe: Distill 1T tokens from a larger Nemotron-3 teacher, combine with multi-teacher MOPD post-training, and add a step to improve MTP acceptance length, raising acceptance length from 21.8% to 4.61.

Dimuon optimizer: A Muon-variant optimizer introduced by Thinking Machines, designed specifically for LoRA fine-tuning and released in Tinker. It is worth benchmarking against AdamW.

The Main Thread: Teams Are Starting to Invest in the Supervision Signal Itself

Putting this week's papers and engineering blogs together, distillation is the recurring protagonist, but from opposite angles. The arXiv papers warn: distillation silently imprints behaviors you did not ask for. Engineering practice proves: as long as you keep a close eye on "what exactly is being transferred," distillation remains the cheapest path to compressing scale into small models.

CleanRM and RM-OPD both aim at "extracting reward model quality from imperfect inputs," revealing a clear main thread: teams are investing more effort into the supervision signal itself — who the teacher is, how clean the preference labels are — rather than the RL algorithms that consume those signals.

For frontline teams, this may mean a shift in focus: instead of repeatedly tuning RL hyperparameters, first audit your teachers and preference data.

One-line Summary

Distillation both implicitly transfers teacher behavioral tendencies and can produce reward models that surpass the teacher; the key is auditing the supervision signal.

Paragraph Summary

This week, multiple papers and engineering blogs examined knowledge distillation from opposite angles. The arXiv paper "Imprinting" found that even when distilling on unrelated tasks, the teacher's safety disposition still transfers to the student model, and token removal plus post-hoc DPO can only partially correct it; therefore, before choosing a teacher, audit behavior rather than just capability. In contrast, "RM-OPD" uses routed on-policy distillation with token-level dense supervision to let a student reward model surpass all its expert teachers, while CleanRM calibrates noisy labels with a small amount of clean preference data. Apertus 1.5 disclosed a four-stage SFT→RLVR→DPO→SDPO alignment stack with real data scale, and UltraBERT's cascaded distillation improved classification while hurting retrieval under high masking rates. On the engineering side, LoRA+GRPO, Nemotron 3.5, and the Dimuon optimizer continue to prove that distillation remains the cheapest way to compress scale. The overall thread: teams are shifting effort from tuning RL algorithms to auditing teacher behavior and preference data quality.