Does High Linear Probe Accuracy Prove the Model Learned Linear Representations?

The Widespread Assumption and Hidden Loophole of Linear Probes

In research on pretrained vision models, the linear representation hypothesis has long dominated: models encode generative factors as linearly decodable directions in feature space. The standard way to test this hypothesis is to train a linear probe; if classification accuracy is high, the factor is considered to be linearly represented.

However, a paper titled Is a Linear Probe Evidence of a Linear Representation? points out that this test is too lenient. Asymptotically, any sufficiently rich feature map—including random feature maps—permits linear readout. This means high probe accuracy is actually compatible with two very different geometric structures: a true linear world model, where latently aligned subspaces dominate the representation; and a kernel machine, where latent variables are linearly embedded in a much larger curved manifold.

This distinction matters because curved structures orthogonal to the probe directly affect the model's out-of-distribution (OOD) generalization, robustness, and compositional transfer. Simply put, high probe accuracy only shows that information "exists"; it does not show how that information is geometrically organized.

Linear probe is a technique widely used in representation learning evaluation: freeze the pretrained model's feature extractor and train a simple linear classifier (usually logistic regression or a linear SVM) on top of its output features to test whether a target factor (e.g., object category, color, pose) can be linearly decoded from the features. If the linear classifier reaches near-ceiling accuracy, researchers typically infer that the factor is encoded linearly in feature space.

The appeal of linear probes lies in their simplicity and interpretability: they turn "does the representation contain a certain piece of information" into a convex optimization problem, avoiding the overfitting and black-box nature of nonlinear probes. However, probe accuracy is significantly affected by feature dimensionality, sample size, and regularization strength—and high-dimensional random features can also achieve high linear separability. This is precisely the core loophole identified by the paper.

The Essential Difference Between the Two Geometric Structures

The research team formalizes the two cases:

• Linear world model: latent factors correspond to dominant linear directions in feature space, and the subspace captured by the probe aligns with the main variance directions of the representation.
• Kernel machine: the feature space is a high-dimensional curved manifold, with latent factors linearly embedded in it, but the curved parts of the manifold (orthogonal to the probe) may carry a large amount of task-irrelevant or task-relevant structure.

In the kernel machine case, a linear probe can still achieve high accuracy, because random feature maps can linearly separate data at sufficiently high dimensions. But the model has not learned a "linear world model," and its response to out-of-distribution data may deviate completely from expectations.

This explains why some models have high probe accuracy on standard benchmarks yet perform poorly on robustness tests or compositional generalization tasks—their representational geometry does not match the linear assumption.

Kernel machine is a class of nonparametric learning methods based on kernel functions. The core idea is to implicitly map data into a high-dimensional (even infinite-dimensional) feature space via a kernel function, and then solve the problem in that space using linear methods (such as linear classifiers or regression). Common kernel machines include support vector machines (SVM) and kernel ridge regression.

In the context of representation learning, "kernel machine" is used to describe a geometric structure: latent factors are linearly embedded in a high-dimensional curved manifold, and the linear probe succeeds because linear readout in a high-dimensional space is sufficient to separate the data—not because the latent factors dominate the main variance directions of the representation. In this case, the curved parts of the manifold (directions orthogonal to the probe weights) may encode a large amount of non-task-related variation, causing the model to fail on out-of-distribution data. The kernel machine perspective provides a geometric explanation for "high probe accuracy but poor generalization."

Three Asymmetric Diagnostic Metrics

To distinguish these two geometric structures, the authors propose three asymmetric diagnostic metrics:

1. Reverse predictivity: the ability to infer latent factors from features. In a linear world model, reverse prediction should be symmetric with forward prediction; in a kernel machine, reverse prediction is significantly weaker than forward prediction.
2. Spectral concentration: checks whether the representation's variance is concentrated in subspaces aligned with latent factors. A linear world model should exhibit high spectral concentration, whereas a kernel machine's spectrum is more dispersed.
3. Out-of-distribution generalization: evaluates probe performance under distribution shift. A linear world model should remain stable, while a kernel machine tends to collapse.

Together, these three metrics form a diagnostic toolkit that helps researchers determine the true geometric structure behind probe accuracy.

Spectral concentration derives from eigenvalue analysis of the representation covariance matrix. Specifically, if the centered covariance matrix of the feature representation is eigendecomposed, spectral concentration measures the proportion of total variance carried by a few leading eigenvalues. In a linear world model, directions aligned with latent factors should dominate the variance, manifesting as rapid decay of the eigenvalue spectrum (high spectral concentration); in a kernel machine, variance is spread across many curved directions, and the eigenvalue spectrum decays slowly.

Spectral concentration can be quantified by computing the cumulative variance contribution of the top k eigenvalues or the participation ratio. The advantage of this metric is that it requires no labels and relies only on the features themselves, so it can serve as an unsupervised diagnostic tool complementary to reverse predictivity and OOD generalization, which require training probes.

Experimental Validation: From Simulation to Real Benchmarks

The authors validated the effectiveness of these diagnostic metrics in a fully observed simulation environment where ground truth is known. They then applied the method to six pretrained vision Transformer models, testing on SVG-World (a deterministic generation benchmark with paired visual styles), and evaluated two of the models on CelebA.

The results show that pretraining does produce structures beyond random kernels, but far from a linear world model. In other words, the models learn some linearly decodable information, but the geometry of the representation remains highly curved, with a significant gap from an ideal linear world model.

On the dSprites, 3D Shapes, and MPI3D datasets, the researchers further found a separation between forward decodability and latent-aligned variance—probe accuracy is high, but latent factors do not dominate the representational geometry.

Implications for Representation Learning Research

The core warning of this work is: forward decoding accuracy alone does not prove that the probed factor aligns with the dominant geometry of the representation. As a tool, the success of a linear probe should not be simply equated with "the model learned linear and low-dimensional representations."

For practitioners working on interpretability, robustness, or compositional generalization, this means probe results need to be interpreted more cautiously. When assessing whether a model has truly learned structured representations, one should combine multiple diagnostics such as reverse prediction, spectral analysis, and OOD testing, rather than relying on a single accuracy metric.

The study also suggests that designing better representation learning objectives—encouraging spectral concentration and linear alignment—may be a key direction for improving model generalization.

One-line Summary

High linear probe accuracy does not prove linear representations; three asymmetric metrics are needed to distinguish linear world models from kernel machines.

Paragraph Summary

This article discusses a study questioning the validity of linear probes. Traditionally, high linear probe accuracy is taken as evidence that a model has learned linear representations, but the study points out that any sufficiently rich feature map (including random features) permits linear readout, so high probe accuracy is compatible with two geometric structures: a true linear world model (latent factors dominate linear directions) and a kernel machine (latent variables embedded in a high-dimensional curved manifold). To distinguish them, the authors propose three asymmetric diagnostic metrics: reverse predictivity, spectral concentration, and out-of-distribution generalization. Experiments on simulations and multiple pretrained vision Transformers show that pretraining produces structures beyond random kernels but far from a linear world model, and that forward decodability and latent-aligned variance can diverge. The conclusion is that evaluating representation quality requires multidimensional diagnostics rather than relying on a single probe accuracy metric.