What Do Sparse Autoencoder Feature Discovery Scores Actually Identify? The Gap from Activations to Interventions

Sparse Autoencoders (SAEs) are currently one of the hottest tools in interpretability research. Researchers use them to "mine" readable features from large model activations, then verify whether these features truly correspond to some concept through steering or ablation. But a new study points out: there is a systematic gap between how these features are "discovered" and how they are "used."

Discovery and Intervention: Two Incompatible Logics

The process of SAE feature discovery is fundamentally observational—looking at how the activation value of some latent variable co-varies with behavior. For example, when a feature frequently activates on text related to the "Golden Gate Bridge," researchers tend to believe it encodes the "Golden Gate Bridge" concept.

But intervention is an entirely different matter. When you use this feature to steer the model's output, or ablate it, the question you care about is: how much will the model's output change if you alter this latent variable?

The paper formalizes the latter as "intervention attribution"—the first-order effect of a specific intervention on the model's output. The question is whether observational discovery evidence can identify this intervention attribution.

The authors' answer is no.

For a broad class of response functions, observational feature discovery evidence generally cannot identify intervention attribution.

This is not a flaw of any specific method, but a structural conclusion. Linear probe scores, correlation coefficients, activation frequencies—these commonly used metrics in the feature discovery stage have no necessary correspondence with "how much output this feature can move when intervened upon."

---

Sparse Autoencoder (SAE) is an unsupervised learning structure that compresses a high-dimensional model activation vector into an overcomplete sparse latent space, then reconstructs the original activation, forcing each latent variable to be sensitive to only a few input patterns. In large model interpretability research, SAEs are used to decompose polysemantic, dense neuron activations into a large number of monosemantic, readable latent features. Typical work includes Anthropic's "dictionary learning" and OpenAI's "sparse circuits" attempts.

Steering refers to artificially amplifying or suppressing a feature's activation value at inference time, observing whether the model's output changes in the expected direction—for example, increasing the "Golden Gate Bridge" feature to make the model mention the landmark more frequently. Ablation sets that feature's activation to zero or replaces it with a baseline value, observing how much the model's capability or behavior is damaged. Both are intervention methods aimed at moving from correlation to causal verification.

A linear probe is a lightweight interpretability tool: training a linear classifier on top of model activations to predict some attribute (such as sentiment or grammaticality), whose weights are often treated as the direction of that attribute in activation space. Although probes can decode attributes with high accuracy, their scores only reflect observational associations and may not be equivalent to causal utility—this is the core of the paper's challenge.

Gradient Decomposition: Quantifying the Gap Between Probes and Intervention

The paper does not stop at the qualitative judgment that "the two are different," but provides a precise gradient decomposition that breaks down the gap between linear probe predictions and true intervention attribution.

The significance of this decomposition is that it shows intervention attribution itself is gradient-estimable. In other words, you don't need to actually perform intervention experiments to use gradient information to approximate "how much impact intervening on this feature would have."

This provides a new objective function for feature selection: rather than using activation correlation to pick features, use intervention attribution.

---

Intervention attribution comes from the attribution idea in causal inference, measuring the first-order change in model output after applying a specific intervention to some internal variable. If the model output is denoted (f) and the intervened latent variable is denoted (z_i), then intervention attribution can be approximated as the value of (\partial f / \partial z_i) at the intervention point—this is precisely why the paper calls it "gradient-estimable": without actually running intervention experiments, backpropagation alone yields a local effect estimate.

This formalization contrasts with observational attribution. Observational attribution is typically measured using correlation coefficients between feature activations and outputs, linear probe coefficients, or rank statistics—none of which require any modification to internal variables, and thus can only capture natural co-variation. The paper's gradient decomposition splits the difference between "probe prediction" and "true intervention effect" into several analyzable terms, explaining why the former cannot generally identify the latter.

It is worth noting that this gradient estimate is only valid when the intervention magnitude is sufficiently small and the model is locally approximately linear. For large-scale steering or highly nonlinear response functions, the first-order approximation may distort, so the paper uses small-step steering experiments to verify the consistency between attribution estimates and actual effects.

Empirical Results Across Four Model-SAE Pairs: How Large Is the Gap?

The paper systematically measures four model-SAE pairings, and the numbers are quite striking.

The median probe-to-attribution ratio ranges from tens to hundreds. This means that if a feature scores highly on a linear probe, its effect under actual intervention may be only a few tenths or even a few hundredths of what the probe suggests.

More detailed verification comes from small-step steering experiments: attribution estimates closely match actual steering effects. This shows that gradient-estimated attribution is not armchair theorizing—it genuinely tracks real intervention behavior.

In single-latent-variable ablation experiments, for cases where the effect is non-zero:

• The proportion of cases where the gradient-predicted sign matches the true sign reaches 91%–97%
• While the direction of change suggested by probes is correct only 17%–35% of the time

This is a very stark contrast. Probe scores are nearly approaching random guessing performance when it comes to predicting intervention direction.

---

Gradient prediction sign agreement rate is a metric measuring the directional correctness of attribution estimates: for each feature, separately compute the sign of the gradient-based attribution estimate and the direction of actual output change after intervention, and count the frequency with which they agree. Directional correctness is more fundamental than magnitude accuracy—if even the sign is reversed, the intervention will produce the opposite effect, which could be dangerous in safety alignment. The paper reports 91%–97% agreement, indicating that gradient attribution is highly reliable in direction judgment.

Probe attribution, on the other hand, uses linear probe coefficients as a proxy for feature importance. In the same experiments, the direction of change suggested by probes agrees with the true intervention direction only 17%–35% of the time, close to chance level. This directly shows that a feature having high weight on a linear probe does not mean that amplifying or ablating it will change the output in the direction of the concept encoded by the probe.

The median probe-to-attribution ratio is another key metric: dividing the probe attribution value by the true intervention attribution value and taking the median across all features. A ratio of tens to hundreds means that probes systematically overestimate the intervention efficacy of features, and ranking by probes will select a large number of features that "look important but are ineffective under intervention."

Comparison of Selection Strategies: Attribution-Selected Features Are More "Useful"

The paper also compares the actual effectiveness of different feature selection strategies.

The feature set selected using intervention attribution produces a larger absolute mean ablation effect on held-out prompts than sets selected by the following strategies:

• Correlation-based selection
• Rank-test-based selection
• Probe-coefficient-based selection

This result confirms from a practical standpoint: if you plan to use SAE features for intervention, selecting by attribution is more reliable than selecting by correlation.

What This Means for SAE Interpretability Research

This work touches on a long-neglected assumption in SAE research: that feature discoverability and feature intervenability are the same thing. The paper shows they are not.

This does not mean SAEs are useless. It means that when researchers report "we found a feature encoding concept X," this claim needs to be more precisely qualified—it speaks to observational-level correlation, not intervention-level causal efficacy.

The paper's constructive direction is: establish intervention attribution as the principled objective for selecting SAE features for intervention, and this objective can be estimated via gradients. This provides a clearer evaluation framework for subsequent research: don't just ask "does this feature activate well," but ask "what happens to the model when you intervene on this feature."

For practitioners using SAE features for safety alignment, behavior control, or model editing, this distinction is especially important. The metrics you use to screen features determine the upper bound of the effectiveness of your subsequent interventions.

Safety alignment and model editing are the most direct application scenarios for SAE feature intervention. In safety alignment, researchers may want to ablate harmful features like "deception" or "sycophancy," or steer beneficial features like "honesty" or "refusal"; in model editing, they update knowledge or behavior by locating and modifying specific features. These tasks are especially sensitive to intervention direction: if features are selected by probe scores, given the direction agreement rate of only 17%–35%, one may very likely select features whose intervention effects are opposite or ineffective, causing safety measures to fail or even backfire.

Therefore, the paper recommends using intervention attribution as the objective function for feature selection, and leveraging gradients for efficient estimation. This framework also suggests that evaluating SAE quality should distinguish two dimensions: discoverability-level readability and reconstruction fidelity, and intervention-level causal efficacy. The two are not automatically consistent and need to be measured separately.

For practitioners, this means that before deploying SAE features for control, one should first conduct small-step intervention verification to confirm that gradient attribution estimates match true effects, and then rank and select features accordingly, rather than directly adopting correlation or probe scores.

One-line Summary

New research proves that observational discovery scores of SAE features cannot identify their intervention effects; intervention attribution must be estimated separately using gradients.

Paragraph Summary

Sparse Autoencoders (SAEs) are widely used to discover readable features from large model activations, but a new paper points out that observational evidence at the feature discovery stage (linear probes, correlation coefficients, activation frequencies) cannot identify intervention attribution—the first-order effect of changing some latent variable on model output. The authors provide a gradient decomposition proving that intervention attribution is gradient-estimable, and measure across four model-SAE pairs that the median probe-to-attribution ratio reaches tens to hundreds; in single-latent-variable ablation, gradient-predicted signs agree with true signs 91%–97% of the time, versus only 17%–35% for probes. Features selected by attribution also produce larger mean ablation effects on held-out prompts than those selected by correlation, rank tests, or probe coefficients. The paper recommends establishing intervention attribution as the principled objective for feature selection, and reminds safety alignment and model editing practitioners to distinguish discoverability from intervenability.