The LeCat Framework: Tracing How LLMs Learn Category Knowledge from Pretraining Data

The LeCat framework reveals how knowledge expression in pretraining data affects category learning in large models.
LeCat is a two-part analytical framework for measuring how LLMs learn category knowledge from pretraining data. The first part evaluates three abilities: memory of in-domain category knowledge, generalization to unseen instances, and flexible application in appropriate contexts. The second part traces how category knowledge is encoded in pretraining text through various forms of "knowledge expression." Evaluation of open-source models based on 199 category–feature pairs finds that higher context and category–feature PMI benefits larger models, while verbatim rule-like statements weaken flexible learning. Controlled experiments with a 68M-parameter model trained from scratch confirm these observations.
Humans know that airplanes are fast, and by extension that red airplanes are also fast—but they don't assume that a broken airplane is fast too. This kind of flexible use of category knowledge seems effortless, but how do large models acquire it? A study called LeCat proposes a two-part analytical framework specifically designed to measure how LLMs learn category knowledge from pretraining data.

What Problem Does LeCat Address?
Category knowledge is a cornerstone of human cognition: we can not only remember the typical features of a category, but also generalize those features to unseen new instances, and judge when they hold and when they fail. Whether LLMs possess this ability has previously lacked systematic evaluation methods.
The first part of LeCat is an evaluation designed precisely for this: it distinguishes three cases—memory of in-domain category knowledge, generalization to unseen instances, and the ability to flexibly apply knowledge in appropriate contexts. Separating "memorization" from "application" in evaluation is a key step toward understanding a model's true capabilities.
Forms of "Knowledge Expression" in Pretraining Data
The second part of the framework turns to the data side, tracing how such knowledge might be encoded in pretraining text. The research refers to different ways of writing as forms of "knowledge expression" (knowledge expression), such as verbatim repetition, exception statements, and so on.
How text expresses a category's features may directly affect whether the model learns rigid rules or transferable, flexible knowledge. This perspective extends evaluation from "can the model do it" to "how is the data written," offering an actionable entry point for pretraining data design.
PMI (pointwise mutual information) is a statistic measuring the strength of co-occurrence between two events: PMI(x, y) = log[P(x, y) / (P(x)P(y))]. In the context of LeCat, it is used to quantify the degree to which a category word and a feature word co-occur in the pretraining corpus, while subtracting the influence of their respective frequencies. High PMI means the co-occurrence is not accidental and more likely reflects a genuine semantic association. Therefore, using PMI rather than raw co-occurrence frequency to characterize "category–feature" associations avoids spurious associations caused by high-frequency words and more accurately reflects the statistical structure of knowledge expression in the data.
Verbatim repetition refers to repeatedly stating the same rule or category feature in nearly identical wording throughout the pretraining corpus—for example, large amounts of text written in fixed patterns like "birds can fly." In contrast are diverse expressions such as exception statements, conditional qualifications, and illustrative examples. Verbatim repetition reinforces the model's memory of specific literal patterns, making it tend toward rote memorization rather than inducing transferable category structure; diverse expressions provide variants of the same knowledge in different contexts, helping the model distinguish typical features from boundary conditions and thus learn more flexible category knowledge.
Empirical Findings from Open-Source Models
The research first evaluates open-source pretrained models using 199 curated category–feature pairs and analyzes their training data. Results show that stronger context and category–feature co-occurrence likelihood (PMI) improve learning in larger models, while verbatim rule-like statements weaken flexible category learning.
In other words, the larger the model, the more it benefits from statistical co-occurrence; but writing rules too rigidly in the data may instead lead the model to "memorize clauses" rather than "understand categories."
Controlled Validation by Training from Scratch
To systematically isolate these effects, the research team trained a 68M-parameter model from scratch, precisely controlling the category information appearing in its training data. Controlled experiments confirmed the earlier observations: verbatim repetition reduces flexible learning, while mixing multiple forms of knowledge expression leads to more robust category learning.
Controlled conclusions from small-scale models and observations from large-scale open-source models corroborate each other, making the judgment that "data expression affects knowledge acquisition" more convincing.
Controlled experiment in machine learning research refers to an experimental design in which researchers actively manipulate independent variables while holding other conditions constant, in order to isolate causal effects. The significance of training a 68M-parameter model from scratch in LeCat is that: the pretraining data of open-source large models is opaque and cannot be intervened upon, allowing only correlational observation; whereas training a small model from scratch allows precise control over which category information appears in the training corpus, in what form, and how many times, thereby establishing a causal—not merely correlational—link between "data expression" and "category learning outcomes." The mutual corroboration between small-model conclusions and open-source large-model observations enhances the external validity of the conclusions.
Implications: Contextual Flexibility Remains a Challenge
LeCat's core conclusions are twofold: on one hand, contextual flexibility remains a persistent challenge for models; on the other hand, the expression of category knowledge in pretraining significantly affects whether models can acquire robust category knowledge.
For pretraining data builders, this means deliberately incorporating diverse expressions—rather than piling up homogeneous rule statements—may be a practical means of improving models' category generalization ability.
Frequently Asked Questions
Which three abilities does LeCat evaluate? Memory of in-domain category knowledge, generalization to unseen instances, and flexible application of knowledge in appropriate contexts.
What kind of pretraining data is more conducive to category learning? Research finds that mixing multiple forms of knowledge expression yields more robust results, while verbatim rule-like statements weaken flexible learning.
One-line Summary
The LeCat framework reveals how the expression of knowledge in pretraining data affects category learning in large models.
Paragraph Summary
LeCat is a two-part analytical framework for measuring how large language models learn category knowledge from pretraining data. The first part evaluates three abilities: memory of in-domain category knowledge, generalization to unseen instances, and flexible application of knowledge in appropriate contexts. The second part turns to the data side, tracing in what forms of "knowledge expression" category knowledge is encoded in pretraining text. Evaluation of open-source models based on 199 category–feature pairs finds that higher context and category–feature PMI benefits learning in larger models, while verbatim rule-like statements weaken flexible learning. Controlled experiments with a 68M-parameter model trained from scratch confirm these observations: verbatim repetition reduces flexible learning, while mixing multiple forms of knowledge expression leads to more robust category learning. The core implication is that contextual flexibility remains a persistent challenge for models, and the expression in pretraining data significantly affects the quality of category knowledge acquisition.
Related articles

Gemini 4 Neon Leak: A Potential Strong Contender Against Opus 5.5
A Hacker News thread suggests Gemini 4 Neon looks like an Opus 5.5 competitor. This article unpacks the product positioning behind this community signal, shifts in the top-model competitive landscape, and what it means for developers' model selection strategy.

Claude Fable 5.2's Self-Authored Short Film 'Fable' Sparks Discussion
A short film 'Fable,' credited to Claude Fable 5.2 for script, visuals, and directing, appears on Hacker News. This article analyzes the significance of full-pipeline AI creation, community reactions, and what it means for content creators.

Anthropic IPO Pricing: A Signal of AI Giants Going Public
Anthropic's IPO pricing draws attention. This article analyzes its potential impact on AI industry valuation, commercialization paths, and the competitive landscape, and highlights public-market expectations and challenges for AI companies.