Knowledge That Is Linearly Readable Generalizes After Fine-Tuning

Language models can memorize facts, but not all of those facts can support further learning. A new study divides the facts in pretrained models into two categories: active facts and inert facts. The former can go beyond their own memory range after fine-tuning and support generalization; the latter, although they can also be recalled, cannot produce generalization on the same task. What truly determines whether new relations can generalize is not whether the facts can be recalled, but whether their shared structure is easy to read out linearly in the pretrained representation.

How the Experiment Was Designed

The researchers' approach was clever: they used existing relations to construct entirely new relations. For example, in the relation "place → continent," Paris corresponds to Europe. Now assign each answer an arbitrary new label—Europe becomes crab, Asia becomes zebra. The new relation then becomes Paris → crab.

The key point is that during fine-tuning, the model is not told the original relation, nor is it told this renaming rule. The researchers fine-tune the new relation only on a subset of subjects, and then test whether the model can assign the corresponding label to held-out subjects that shared the same answer in the original relation. In other words: after learning Paris → crab, will it answer Munich → crab?

This design cleanly separates "memorization" from "generalization." The model has every opportunity to memorize Paris → crab from the fine-tuning data, but Munich → crab requires it to transfer the pretrained knowledge that "Munich and Paris are both in Europe."

Linear Readability Is the Strongest Predictor

Across 11 language models and 70 relations, the results were highly consistent: how easily the original relation can be read out linearly in the pretrained representation strongly predicts generalization performance after fine-tuning. Within the same model, these two metrics correlate across relations with a Pearson r of 0.87 to 0.92.

This association remains robust after controlling for multiple variables, including the number of fine-tuning subjects and answers, the model's accuracy on the original relation, and the frequency with which these subjects appear in the pretraining text. It also holds up under label re-randomization and reruns with different fine-tuning random seeds, and the conclusion is unchanged when the animal labels are replaced with labels from 13 other semantic categories.

That is, the difference in generalization ability is not caused by data volume, accuracy, or word frequency, but by the nature of the representation structure itself.

Linear probe is a commonly used tool in mechanistic interpretability research: on top of the activation representation of a certain layer of a model, train a simple linear classifier (usually logistic regression), and use its accuracy to measure whether a certain semantic property has already been encoded by the model in a "linearly separable" way. The emphasis on "linear" is because linear readability means that the property is roughly distributed along a fixed direction in the representation space, and downstream computation only needs a single weighted sum to extract it, with low cost and high stability. Conversely, if the property can only be recovered through multiple layers of nonlinear transformation, it means that it is more deeply entangled in the representation and harder to use effectively during fine-tuning. Linear probes are usually trained on only a small amount of labeled data and do not backpropagate to update the original model, so they measure "what is already in the representation," not "what the model can learn." This article uses exactly this metric to predict whether a new relation can generalize after fine-tuning.

Pearson correlation coefficient r measures the strength of linear correlation between two continuous variables, ranging from -1 to 1: close to 1 indicates a strong positive correlation, and close to 0 indicates almost no linear relationship. In the context of this article, r = 0.87 to 0.92 means that within the same model, the higher the linear readability of a relation, the better the model performs after fine-tuning in transferring the new label to held-out subjects, and this tendency to change in sync is very tight. It should be noted that correlation is not causation, but after controlling for multiple potential confounding variables (sample size, accuracy, word frequency, etc.), the association still exists, making the explanation that "the representation structure itself is at work" more credible.

Incorrect Labels Can Reveal the "Correct" Answer

One detail that is even more telling: the labels the fine-tuned model got wrong were mapped back to the original answers to which they were initially assigned. For example, if the model answers Munich → zebra, and zebra corresponds to Asia, then it is restored to Munich → Asia.

These restored answers appear more often in the linear probe's top answers for that subject than the answers given by the pretrained model itself. This suggests that what the model learned after fine-tuning is not pure noise, but some projection of the original relation structure—it just failed to express it in the right direction.

Why This Matters

This finding changes the criterion for judging "what the model knows." In the past, we often used recall accuracy to measure whether a model had mastered a fact, but accurate recall does not mean that this knowledge can be used for new tasks.

In practice, this means: if you want the model to learn a new relation through fine-tuning, first check whether it has already linearly encoded the corresponding old structure in its pretrained representation. For relations with poor linear readability, simply increasing fine-tuning samples may have limited effect. Conversely, this also provides a feasible diagnostic path—using linear probes to determine in advance which knowledge is worth investing in fine-tuning.

One-line Summary

Only facts that are linearly readable during pretraining can generalize after fine-tuning; inert facts can only be recalled.

Paragraph Summary

The study divides facts in language models into active facts and inert facts: the former can go beyond memory after fine-tuning and support generalization, while the latter can be recalled but cannot generalize on the same task. The experiment constructs entirely new relations by assigning arbitrary new labels to the answers of existing relations (such as Europe→crab), and fine-tunes on a subset of subjects without revealing the original relation or the renaming rule, testing whether the model can transfer the label to held-out subjects that share the same original answer. Across 11 models and 70 relations, the linear readability of the original relation strongly predicts generalization performance after fine-tuning, with Pearson r of 0.87 to 0.92, and it remains robust after controlling for sample size, accuracy, word frequency, and re-randomization. The study also found that incorrect labels, when mapped back to the original answers, appear more often in the linear probe's top answers than the answers given by the pretrained model itself.