Language Models Under Self-Training: Richer Vocabulary, Yet Lower Entropy?

Recursive self-training—repeatedly fine-tuning a language model on its own outputs—is often used to study model collapse. But a new study reveals a counterintuitive phenomenon: while lexical diversity declines, certain information entropy metrics actually rise. This challenges the naive intuition that "richness equals evenness" and offers a more refined tool for assessing model degradation.

Self-Training Experiment: Entropy Trajectories Across Nine Generations of GPT-2

The researchers conducted nine generations of self-training with English GPT-2, fine-tuning each generation on the previous generation's outputs while varying the sampling temperature of the training data. All models were measured with the same fixed sampler to ensure fair comparison. The results showed that as training temperature increased, the change in Renyi entropy shifted from loss to gain, and entropies of different orders crossed over at different temperatures: min-entropy crossed first, followed by collision entropy, Shannon entropy, and finally observed lexical richness. These ordered crossings form an interval.

Under untruncated training (T=1), the 9th-generation model at a 13,000-word budget had 24% fewer word types than the 0th generation, yet its collision entropy was 0.91 bits higher. In other words, the vocabulary was shrinking, but the word-frequency distribution became more even—the dominance of high-frequency function words declined. This phenomenon persisted across a finer temperature grid, different measurement samplers, fixed word budgets, and five new training seeds.

Renyi entropy is a generalization of Shannon entropy, proposed by Alfréd Rényi in 1961, with the formula (H_\alpha = \frac{1}{1-\alpha}\log\sum_i p_i^\alpha), where (\alpha \ge 0) is the order. Different orders correspond to different sensitivities to the distribution: as (\alpha \to 0), Renyi entropy approaches observed lexical richness (i.e., the logarithm of the number of word types that appeared), caring only about which words appeared; when (\alpha=1), it equals Shannon entropy, measuring average uncertainty; when (\alpha=2), it is called collision entropy, which is closely related to the probability that two independently sampled words are the same (collision probability) and focuses more on high-frequency words; as (\alpha \to \infty), we get min-entropy, determined solely by the most probable word and most sensitive to the distribution's peak. Thus, Renyi entropies of different orders characterize the word-frequency distribution from different angles: low orders reflect vocabulary coverage, high orders reflect the concentration of high-frequency words—this is precisely the basis for the ordered crossings observed in the paper.

Training temperature here does not merely affect sampling randomness at inference time; it refers to the sampling temperature used when generating self-training data. Temperature (T) adjusts the model's output probability distribution via (p_i \propto \exp(z_i/T)): (T>1) flattens the distribution (increasing diversity), while (T<1) sharpens it (more deterministic). In recursive self-training, different temperatures alter the statistical properties of each generation's training data, thereby affecting the distribution the model learns. Untruncated training (T=1) refers to fine-tuning on the original model's outputs at temperature 1 without additional distribution truncation. Type-token ratio (or observed richness) is the ratio of the number of distinct words to the total number of words in a text, intuitively reflecting lexical diversity. Word budget refers to the fixed number of words generated when measuring entropy or richness; because metrics like richness vary with sample size, a fixed budget is needed to fairly compare models across generations.

Robustness and Boundaries of the Crossing Order

The robustness of the crossing order was verified under multiple settings: in four other settings, min-entropy still crossed before richness (in Hindi, it held only at the interpolation root); among 13 different measurements under eight text representations, 12 supported the order. Next-word identity and category simulations isolated decoding and sampling effects, but the crossing order itself remains empirical.

The researchers also found that fine-tuning on human text can produce the same divergence while likelihood actually improves. No single ordering sign reliably indicates worse likelihood—meaning the relationship between entropy metrics and model quality is more complex than expected.

Likelihood here refers to the probability a model assigns to real text (e.g., a human-written validation set), typically measured by average negative log-likelihood (i.e., cross-entropy loss); lower loss means better prediction of real data. In self-training research, likelihood is often used as a proxy for model quality, but this paper points out that it can decouple from entropy metrics: for example, after fine-tuning on human text, the model's entropy profile diverges while likelihood improves, indicating that changes in entropy metrics cannot simply be equated with model degradation or improvement. Next-word identity simulation and category simulation are controlled experiments: the former uses artificially constructed next-word identity distributions to simulate self-training, while the latter substitutes word categories (e.g., part of speech) for specific words to isolate effects from decoding and sampling, helping determine whether the crossing order is driven by these mechanisms. These simulations support some aspects of the crossing order, but the order itself still requires empirical validation.

Implications for Model Evaluation

This study suggests that when reporting entropy profiles, the word budget should be specified and accompanied by independent quality measures. Looking solely at lexical richness or a single entropy value can be misleading: self-training may make model outputs appear "poorer" (fewer words), yet the distribution is more even, and likelihood may even improve. This reminds us that monitoring model collapse requires a multidimensional perspective and cannot rely on a single metric.

According to analysis from the RSS source, this work provides a new framework for understanding information dynamics in self-training, but the universality of the crossing order still needs verification across more languages and architectures.

Key Takeaways

• In self-training, a decrease in word types and an increase in collision entropy can occur simultaneously, breaking the intuition that "richness = evenness."
• The crossing temperatures of Renyi entropies of different orders are ordered: min-entropy first, richness last.
• This order is robust across various samplers, seeds, and text representations, but remains an empirical regularity.
• The relationship between entropy metrics and likelihood is complex; evaluating models requires independent quality measures.

One-line Summary

In self-training, lexical diversity declines while certain entropies rise, and the crossing temperatures of Renyi entropies of different orders are ordered.

Paragraph Summary

A new study conducted nine generations of recursive self-training with GPT-2 and found that lexical richness and distribution evenness can diverge: at a fixed word budget, the 9th-generation model had 24% fewer word types than the 0th generation, yet its collision entropy was 0.91 bits higher, indicating reduced dominance of high-frequency words and a more even word-frequency distribution. The study further reveals that as training temperature increases, changes in Renyi entropies of different orders shift from loss to gain, with ordered crossing temperatures: min-entropy first, followed by collision entropy, Shannon entropy, and finally observed lexical richness, forming an observable interval. This order is robust across various samplers, seeds, and text representations, but remains an empirical regularity. The study also finds that the relationship between entropy metrics and likelihood is complex, and recommends specifying the word budget and supplementing with independent quality measures when evaluating models.