[KongchangAI]
· 2 min read· 1,097 words

Structure-Residual Inductive Bias: Taking Language Model Pretraining Beyond Next-Token Prediction

Structure-Residual Inductive Bias: Taking Language Model Pretraining Beyond Next-Token Prediction

A new study proposes the structure-residual principle to introduce representational structure as an explicit inductive bias into pretraining.

The study questions whether next-token prediction suffices for good representations and, inspired by the two-part coding view of Kolmogorov complexity, proposes the structure-residual principle that decomposes representations into stable structure and implementation detail. Two complementary implementations are given: Boundary Bottleneck Concept (BBC) routes token-segment information through boundary carriers, while Continuous Concept (CC) regularizes representation geometry to induce a spectral decomposition. BBC works best in early layers and CC in later layers; combining them on GPT-2 Medium improves NLL by 1.44% at 300M tokens, with gains persisting to 20B tokens.

Language model pretraining has long relied on the single objective of next-token prediction, but should the prediction behavior and the learning of internal representations be decoupled? A new study proposes the "structure-residual" principle, attempting to introduce the structural information of representations into the training process as an explicit inductive bias.

Overview of structure-residual inductive bias research

From Next-Token Prediction to Explicit Structural Modeling

The next-token prediction task drives models to simultaneously learn prediction behavior and internal representations, but the research team questions: should the representational structure itself become an explicit inductive bias? Inspired by the two-part coding perspective of Kolmogorov complexity, they investigate the structure-residual principle, decomposing representations into a structural part and an implementation-detail part. The core of this idea is to let the model explicitly distinguish which information is stable structure and which is concrete implementation detail, thereby guiding representations toward more organized forms during pretraining.

Kolmogorov complexity is a core concept in algorithmic information theory, referring to the length of the shortest program required to generate an object, used to characterize the object's intrinsic complexity. Its two-part coding perspective divides a description into a "structural part" (regularities, compressible patterns) and a "residual part" (random details that cannot be explained by regularities), with the sum of the two lengths constituting a complete description of the object. This perspective offers an important insight for machine learning: a good representation should explicitly separate predictable structure from irreducible noise, rather than blending the two into the same latent space. The research team borrows this idea to decompose the internal representations of language models into structural components and implementation-detail components, translating the information-theoretic intuition that "structure should be explicitly modeled" into an operational training objective, thereby laying the theoretical foundation for the structure-residual principle.

Two Complementary Implementations: BBC and CC

The study proposes two complementary instantiation methods. Boundary Bottleneck Concept (BBC) routes information from token segments through boundary carriers while preserving local autoregressive computation. Continuous Concept (CC) regularizes the representation geometry, inducing a spectral decomposition into low-frequency structure and high-frequency residuals. The two approaches realize the structure-residual principle from different angles.

The research team derives coding bounds for both perspectives and proves that when the intra-segment path energy vanishes, the multiplicity-corrected boundary attention becomes exact. This provides theoretical support for the methods, showing that the structure-residual bias is not a heuristic trick but has a coding-theoretic basis.

Spectral decomposition refers to the technique of decomposing a signal or representation into different frequency components. In representation geometry regularization, it corresponds to treating the eigenvalues of the representation covariance matrix as a "spectrum," where low-frequency components correspond to slowly varying, cross-segment stable structural directions, and high-frequency components correspond to local, rapidly changing residual directions. The CC method applies regularization to the representation geometry, suppressing high-frequency residual energy and preserving low-frequency structure, equivalent to performing a "low-pass filter" in the latent space. Understanding this helps grasp the division of labor between BBC and CC: BBC strengthens boundary structure from the topological angle of information routing, while CC purifies structural components from the frequency-domain angle of representations, corresponding respectively to the spatial and spectral implementations of the structure-residual principle.

Striking Differences in Depth Preference

Experiments reveal a surprising difference between BBC and CC in depth preference: BBC works best in early layers, while CC is most effective in later layers. This indicates that different depth layers of language models have different sensitivities to structural information, with early layers better suited for processing boundary structure and later layers better suited for spectral regularization. The researchers validated this on GPT-2 Medium: at 300M training tokens, combining BBC and CC improved NLL by 1.44%, and these gains persist when training is scaled to 20B tokens.

NLL (Negative Log-Likelihood) is the most commonly used evaluation metric in language modeling; a lower value indicates that the model's predicted probability distribution for the next token is closer to the true distribution. It is essentially equivalent to perplexity, which is the exponential form of NLL. In the paper, "NLL improvement of 1.44%" means that under the same data and model scale, combining BBC and CC reduces the model's average negative log-likelihood for true tokens by a relative 1.44%, which is a fairly substantial relative gain for the pretraining stage. It should be noted that the absolute improvement in NLL is usually small, so evaluation considers both the relative percentage and whether the gain remains stable as the number of training tokens scales (from 300M to 20B), the latter being the key evidence for judging whether an inductive bias is truly effective.

Representation Diagnostics and Mechanistic Explanation

Representation diagnostics show that BBC increases the relative predictive importance of boundary states, while CC reduces spectral residual energy. This mechanistically explains why the two methods are effective: BBC improves early-layer representations by strengthening boundary information flow, and CC purifies later-layer representations by suppressing high-frequency residuals. When combined, the model obtains a structured inductive bias beyond next-token prediction during pretraining.

Implications for the Pretraining Paradigm

The significance of this work lies in challenging the default assumption that "next-token prediction is sufficient to learn good representations." By explicitly encoding the structure-residual principle as an inductive bias, the model can achieve better predictive performance with the same amount of data, and the gains remain stable as training scales. The code has been open-sourced, providing a reproducible foundation for future research. For designers of pretraining paradigms, this suggests that the organizational form of representational structure can itself become part of the optimization objective, rather than merely a byproduct of the prediction task.

One-line Summary

A new study proposes the structure-residual principle, introducing representational structure as an explicit inductive bias into pretraining and validating its effectiveness.

Paragraph Summary

The study questions whether next-token prediction is sufficient for models to learn good representations. Inspired by the two-part coding perspective of Kolmogorov complexity, it proposes the structure-residual principle, explicitly decomposing representations into a stable structural part and a concrete implementation-detail part. The study provides two complementary implementations: Boundary Bottleneck Concept (BBC) routes token-segment information through boundary carriers, while Continuous Concept (CC) regularizes representation geometry to induce a spectral decomposition into low-frequency structure and high-frequency residuals, with coding bounds derived for both. Experiments find that BBC works best in early layers and CC is most effective in later layers; at 300M tokens on GPT-2 Medium, combining the two improves NLL by 1.44%, and the gains persist when scaled to 20B tokens. Representation diagnostics show that BBC enhances the predictive importance of boundary states, while CC reduces spectral residual energy. The code has been open-sourced.

#structure-residual inductive bias#language model pretraining#next-token prediction#Kolmogorov complexity#Boundary Bottleneck Concept#Continuous Concept
Share:

Related articles