Cross-Tokenizer Knowledge Distillation: Why 85% of Information Vanishes and How the Chain Rule Fixes It

A Knowledge Distillation Trap That's Easy to Miss

Knowledge Distillation (KD) is one of the core techniques in model compression, formally introduced by Hinton et al. in 2015. The central idea is to have the student model learn not just hard labels (one-hot), but also mimic the "soft labels" — probability distributions over all classes — produced by a larger teacher model. These soft labels carry rich information about inter-class similarities. For example, a teacher model might assign "80% probability to ' the', 15% to ' then'" — this kind of fine-grained information is what Hinton calls "dark knowledge," which carries far more structured knowledge than simple one-hot labels. In the era of large language models, KD is widely used to compress GPT-4-level models down to ones that can run on edge devices. It has become a core engineering tool for model deployment: letting a small student model learn the output distribution of a large teacher model, preserving performance while dramatically reducing inference costs.

However, when the teacher and student models use different tokenizers, a little-known yet highly destructive problem silently emerges — up to approximately 85% of the teacher model's information can disappear without a trace during token mapping.

Recently, a developer shared findings from a distillation experiment on Reddit. They attempted to distill knowledge from a large-vocabulary teacher model (Qwen2.5-Coder, vocabulary ~151K) into a small-vocabulary student model (BPE, vocabulary only 1–4K).

It's worth noting that different models often adopt different tokenization strategies — common ones include BPE (Byte Pair Encoding), WordPiece, and SentencePiece. Vocabulary size directly affects the granularity at which a model perceives text: large vocabularies (like Qwen2.5's 151K) tend to merge common word combinations into a single token, while small vocabularies (1–4K) must split the same word into longer subword sequences. This discrepancy is already complex enough in monolingual models, and becomes even more pronounced in code scenarios — the same code snippet can produce entirely different token sequences and lengths under different tokenizers.

This scenario is far from rare in real-world engineering: teams often design more compact vocabularies for student models in pursuit of maximum deployment efficiency. But this vocabulary gap quietly plants a deep problem. In fact, cross-tokenizer distillation is far more prevalent in industrial deployment than the academic community acknowledges — open-source teacher models each maintain their own independent vocabularies, ranging from 32K to 150K tokens, while student models targeting specific hardware (MCUs, NPUs) often require custom compact vocabularies to meet memory and latency constraints. Most prior academic work assumes the teacher and student share a tokenizer, leaving the information loss problem in a long-standing blind spot of "known but unquantified."

Root Cause: Many-to-One Mapping Causes Entropy Collapse

Why the Standard Approach Fails

The common approach to cross-tokenizer distillation is to project the teacher's top-K logits onto the student's tokens — at segment boundaries, mapping the teacher token to the first token in the corresponding student token sequence.

This is precisely where the problem lies. This projection is fundamentally a many-to-one mapping. Consider an intuitive example: the teacher vocabulary's tokens " the", " then", and " they" all share the exact same first subword token in the student's smaller vocabulary. This means the teacher model's uncertainty across these three words gets directly summed and flattened when projected onto the student's first token — the discriminating information is gone.

Measured Data: Entropy Retention as Low as 15%

To appreciate the severity of this loss, we need the concept of information entropy. Shannon Entropy measures the uncertainty of a probability distribution, defined as H = -Σ p(x)log₂p(x), in bits. High entropy means a more uniform distribution with high uncertainty; low entropy means a concentrated distribution with high predictive certainty. In the context of knowledge distillation, the high-entropy distribution output by the teacher model is precisely the carrier of "dark knowledge" — it assigns meaningful probabilities to multiple candidate tokens, encoding rich inter-class similarity information.

The developer measured entropy retention before starting full training, and the results were alarming:

Teacher distribution entropy dropped from H = 2.09 bits to just 0.32 bits
Only about 15% of the teacher's information was retained
Counter-intuitively, expanding the student vocabulary to 4096 didn't help — retention actually fell slightly to 13%

When entropy collapses to 0.32 bits, the distribution has essentially degenerated into a near one-hot label. The student model effectively receives only a hard label, completely defeating the purpose of soft-label distillation. The author emphasized that this is not a bug in the mapping logic — they verified that 93% of probability mass is correctly aligned. The information loss is an inherent property of this projection approach, not an implementation error.

The Temptation of a Wrong Conclusion

What makes this trap particularly dangerous is that it leads researchers to draw completely wrong conclusions.

The author ran preliminary experiments with this flawed distillation objective, and KD actually underperformed a standard Cross-Entropy (CE) baseline: KD loss was 2.729, while the plain CE baseline was only 2.081.

Without digging into the underlying mechanics, anyone seeing these numbers would naturally conclude: "Knowledge distillation is ineffective when distilling from large to small vocabularies." But the truth is — it's not that distillation doesn't work; it's that 85% of the information was already lost at the mapping stage. This kind of silent failure likely causes many teams to mistakenly abandon a perfectly viable technical approach.

The Fix: Recovering Lost Information with the Chain Rule

The Uncertainty Is Deferred, Not Destroyed

The author's solution is grounded in the chain rule of probability.

The chain rule states that a joint probability can be decomposed into a product of conditional probabilities: P(A,B,C) = P(A)·P(B|A)·P(C|A,B). In autoregressive language models, this rule applies naturally to the token-by-token generation process. The key insight is: the teacher model's uncertainty is factorizable across the student token sequence. The divergence among " the", " then", and " they" doesn't disappear at the first token — it is deferred to subsequent tokens where it gets resolved. The first subword is identical, but from the second subword onward, they diverge.

When a single teacher token corresponds to multiple student subword tokens, the teacher's probability distribution over the entire span can be factorized as: the marginal distribution over the first subword, multiplied by the conditional distribution of the second subword given the first, and so on. This decomposition allows the cross-token uncertainty that was previously "summed and flattened" to be gradually and completely transmitted to the student model through the subword sequence.

Storing Top-K Distributions Conditioned on Prior Bytes

Building on this insight, the fix is: when storing the teacher's top-K distributions, condition on the bytes already generated within the span. Instead of compressing all information into the first token, let the student progressively absorb the teacher's uncertainty along the subword sequence.

The effect is immediate — entropy retention jumps from 15% to 83–86%.

What's more, the engineering overhead is minimal:

No additional teacher model computation required
No extra storage overhead

It simply reorganizes how existing information is utilized, recovering the vast majority of knowledge that was previously being flattened away.

Three Takeaways for Practitioners

This case offers several highly valuable lessons for knowledge distillation practitioners:

First, audit information retention before any cross-tokenizer distillation. Measuring entropy retention or probability mass alignment before committing to full training can prevent wasting massive compute on a flawed objective.

Second, be skeptical of "seemingly reasonable" negative conclusions. When KD underperforms a CE baseline, the first reaction shouldn't be "distillation is useless" — it should be: did the teacher's information actually reach the student intact?

Third, a larger vocabulary isn't always better. The experiments show that expanding the student vocabulary actually reduced retention, reminding us that vocabulary design and distillation strategy need to be co-planned rather than optimized in isolation.

The author has now launched full training with a strict baseline requirement — KD must outperform CE to count as a success. They have also committed to publishing the final metrics regardless of outcome. This rigorous and open attitude is exactly what the technical community values most.

Conclusion

This practical finding shared on Reddit reveals an information collapse problem in cross-tokenizer knowledge distillation that is both widespread and easy to overlook. Its value lies not only in providing a zero-overhead chain rule fix, but also in making explicit a long-standing blind spot — information loss that was "known but unquantified" in practice — and establishing an empirical benchmark for the community to build engineering standards around. It reminds us: many "ineffective" conclusions in model training may trace back to nothing more than a silent information leakage point. Before drawing conclusions, asking "where did the information go?" often opens up new technical possibilities.