Solvable Topic Model Reveals Why In-Context Learning Emerges Before Specialisation

Sequence models have two fundamentally different paths to generalisation: extrapolating structures seen during training, and adapting to entirely new tasks using prompts. Which strategy dominates directly determines a model's behaviour outside the training distribution, and governs the boundaries of latent capabilities and risks. Recent work has theoretically characterised multiple generalisation strategies and empirically described how they shift with the quality and quantity of pretraining data. However, a learning theory of this "strategy competition" has been absent—we do not understand from first principles why one solution is preferentially selected.

According to the paper In-Context Learning Before Specialisation: Theory of Learning in a Solvable Topic Model, researchers provide exactly this theory within an analytically tractable topic model.

A Solvable Model That Mimics the Heterogeneity of Real Pretraining Data

The model generates data as follows: documents are sampled from a mixture distribution over a finite set of latent topics. This setup is not arbitrary—it deliberately mimics a core feature of modern pretraining corpora: heterogeneity. Real data is typically a mix of multiple domains, styles, and semantic clusters, and a finite mixture topic model is a minimally solvable version of exactly this structure.

The key point is that such a dataset naturally contains two distinct "solutions":

• Specialised predictor: stores the entire finite topic mixture distribution, effectively memorising the topic structure seen during training.
• In-context predictor: optimal for topics never seen during training, relying on prompts for on-the-fly adaptation rather than on stored parameters.

These two solutions correspond to two fundamentally different ways sequence models generalise, and the theory must answer precisely under what conditions each arises.

Topic Model is a class of probabilistic generative models for modelling the latent semantic structure of document collections, most classically represented by LDA (Latent Dirichlet Allocation). Its core assumption is that each document is a mixture of several latent "topics," where each topic corresponds to a probability distribution over the vocabulary; to generate a document, one first draws a topic from the topic mixture distribution, then draws words from that topic's word distribution. This structure naturally produces heterogeneous data—different documents come from different topic combinations, creating multiple semantic clusters within the dataset.

The term "solvable" (analytically tractable) means that under this model, the learning curves and emergence thresholds of each predictor can be derived exactly, without resorting to approximations or numerical simulation. The researchers chose the finite mixture topic model precisely because it preserves the heterogeneity of real corpora while permitting closed-form analysis of the conditions under which the two solutions emerge, elevating "strategy competition" from empirical observation to provable theoretical conclusions.

Two Solutions Emerge Sequentially at Different Sample Complexities

The theory's central prediction is that the two solutions emerge at different sample complexities.

Specifically, the in-context predictor is available very early—it exists even when the amount of training data is still small; whereas the specialised predictor appears only after the number of training tokens exceeds the parameter count of the data-generating process.

In other words, the model's generalisation strategy is not fixed from the outset, but undergoes a structural transition as data scale grows. In the early phase, the model relies more on in-context learning to handle tasks; only when data is large enough to "memorise" the entire topic mixture distribution does the specialised solution become feasible and get selected.

This prediction yields a testable quantitative threshold: the parameter count of the data-generating process becomes the dividing line between the two strategies.

Sample complexity in statistical learning theory typically refers to the minimum number of training samples an algorithm needs to achieve a given accuracy. Here, researchers are concerned with the inverse question—fixing the model and data-generating process, and observing the order in which different predictors emerge as the number of training tokens increases.

The emergence of a parameter count threshold has an intuitive explanation: the specialised predictor essentially needs to "write" the entire topic mixture distribution into the model parameters, so its feasibility depends on whether training data is sufficient to cover and determine those parameters. When the token count is less than the parameter count of the data-generating process, the model cannot reliably estimate the full topic structure and must fall back on immediate information in the prompt; once the token count crosses that threshold, memorising the entire mixture distribution becomes feasible, and the specialised solution forms a new descending step on the loss curve. This condition of "data quantity exceeding the generative parameter count" provides a quantitatively testable criterion for the switch between the two strategies.

Trained Transformers Closely Match the Theory

The value of a theory ultimately rests on empirical validation. The researchers trained Transformer models, and the results showed:

• Both emergence scales were reproduced;
• The separating crossover between them was also reproduced;
• The agreement with theory was "remarkable," and no parameters were fitted.

The absence of fitted parameters is particularly noteworthy. It means the theory does not tune parameters to match experimental curves, but rather makes predictions directly from the model setup, with experiments serving only as verification. This consistency substantially boosts confidence in the theory's explanatory power.

Why This Theory Matters

From the perspective of capabilities and risks, a model's behaviour outside the training distribution is determined by the dominant generalisation strategy. If a model primarily takes the specialisation path, it is highly dependent on topic structures seen during training and may be limited when facing genuinely novel topics; if it primarily takes the in-context learning path, it is better at on-the-fly adaptation to new tasks.

Understanding the learning theory of their competition means we may gain a predictive capability: given a data scale and generative process complexity, anticipate which strategy the model will favour. This provides principled guidance for pretraining data mixing ratios, model size selection, and even assessment of latent model capabilities.

This theory, established in a solvable topic model, gives a clear answer to "why in-context learning precedes specialisation": it is not coincidence, but a necessary ordering under sample complexity.

One-line Summary

A solvable topic model theory reveals that in-context learning emerges before specialisation, with the switch point determined by the generative parameter count.

Paragraph Summary

The paper establishes the first learning theory for the two generalisation strategies of sequence models—in-context learning and specialisation—within a solvable topic model. The model assumes documents are drawn from a mixture distribution over finite latent topics, naturally corresponding to two solutions: a specialised predictor that memorises topic structure, and an in-context predictor that adapts on the fly via prompts. Theory predicts they emerge sequentially at different sample complexities: the in-context solution is available at small data scales, while the specialised solution appears only after the training token count exceeds the parameter count of the data-generating process. Experiments training Transformers reproduce both emergence scales and the separating crossover with no fitted parameters and remarkable agreement. The theory provides principled guidance for anticipating model generalisation strategies and assessing out-of-distribution behaviour.