How Does Readout Scaling Affect Sparse Autoencoder Feature Recovery? An Analysis of the RC-DS Method

Sparse autoencoders (SAEs) are widely used to extract interpretable features from neural network activations. But one easily overlooked detail is that readout scaling changes what an SAE actually learns. A recent study systematically analyzes the interaction between readout scaling, data distribution, and training geometry by normalizing the readout decision space (RC-DS) coordinates.

Core Finding: Training Coordinates Determine Feature Recovery

The research team designed a set of controlled coordinate-matching experiments, and the conclusion is quite direct: sparse factor recovery is best when the training process recovers the same space in which the factors were generated.

In other words, how well an SAE learns depends not only on model capacity or the number of training steps, but also on whether the training coordinates align with the data generation process.

When the readout condition number is fixed, changing the generative distribution reverses which training coordinates best recover known factors. This shows that there is no "one-size-fits-all" choice of training coordinates—it depends on how the data is generated.

Sparse autoencoder (SAE) is an unsupervised learning method whose goal is to reconstruct input activations using a set of overcomplete but sparse basis vectors. In mechanistic interpretability research, SAEs are often used to decompose dense neural activations that are superimposed (superposition) into sparse, interpretable "feature" directions. Its loss typically consists of a reconstruction error and a sparsity penalty term. After training, the columns of the decoder weight matrix can be viewed as the learned feature dictionary.

Readout scaling refers to applying diagonal scaling or a linear transformation to the activation vector before feeding the activations into the SAE, thereby changing the relative weight of each dimension in the reconstruction objective. This operation is equivalent to changing the geometry of the "decision space" that the SAE faces: some directions are amplified, while others are compressed. The readout condition number is the ratio of the largest singular value to the smallest singular value of that transformation. The larger the condition number, the more severe the gain differences across directions. Therefore, readout scaling is not an inconsequential preprocessing step—it directly affects which directions the SAE prioritizes recovering and which it ignores.

The RC-DS Method: Removing Singular Value Gains Without Losing Information

The core idea of RC-DS is to use readout-normalized decision-space coordinates to remove singular value gains while preserving useful information in the decision space.

Key data: when factors are generated before readout scaling, under a condition number of 100, RC-DS improves the average sparse factor recovery rate from 0.506 to 0.913. The comparison baseline is centered logit training under the same budget.

The researchers also implemented an exact wrapper that can reproduce RC-DS training to numerical precision. The purpose of this design is clear: to isolate the effect of training geometry rather than introduce additional information. In other words, RC-DS's advantage comes from the geometry of the training space, not from the model seeing more data or parameters.

Condition number is the ratio of the largest singular value to the smallest singular value of a matrix, used to measure how unevenly a linear transformation scales different directions. When the condition number is 1, all directions are scaled proportionally; the larger the condition number, the more strongly some directions are amplified while others are strongly compressed. In SAE training, a high readout condition number causes reconstruction error to be dominated by the amplified directions, leading the model to preferentially fit those directions while ignoring sparse factors in the compressed directions.

The core operation of RC-DS (readout-normalized decision-space coordinates) is: first perform singular value decomposition on the readout transformation, remove the singular value gains, retain only the orthogonal or rotational part that determines the directional structure, and then train the SAE in the normalized decision-space coordinates. In this way, the training objective is no longer artificially distorted by readout scaling, but directional information is not discarded. If the entire readout transformation were discarded directly, the directional structure in the decision space would be lost. Therefore, RC-DS achieves a balance between "preserving information" and "removing gains."

CN-Logit (centered logit training) is a common baseline: centering the output logits and then directly training an SAE. Its difference from RC-DS lies not in data volume or model capacity, but in the geometry of the training space.

Validation on Neural Policies and Language Models

The effectiveness of RC-DS is not limited to synthetic experiments.

In neural policies, RC-DS improves average logit reconstruction under the test budget. On three fixed language model checkpoints, RC-DS achieved higher gain-free reconstruction means in all 18 task-sparsity settings, while centered logit SAEs better reconstructed output logits in 17 settings.

The divergence between these two metrics is noteworthy. It suggests that reconstructing logits and recovering sparse factors are different objectives, and optimizing one does not necessarily benefit the other.

A small-label-head control experiment further showed that longer training can reduce the reconstruction gap from 0.0198 to 0.0016, indicating that part of the difference can be bridged by training sufficiency.

Gain-free reconstruction refers to reconstruction quality measured after removing the singular value gains introduced by readout scaling, that is, the model's ability to recover the original signal in a normalized coordinate system. It reflects whether the SAE has learned the structure of the data itself, rather than whether it has catered to amplified directions.

Logit reconstruction directly measures the error with which an SAE reconstructs the model's output-layer logits, and it is directly related to downstream predictive behavior. The two objectives are different: gain-free reconstruction is closer to sparse factor recovery, while logit reconstruction may be dominated by high-gain directions. Therefore, an SAE performing better on logit reconstruction does not automatically mean that the sparse features it extracts are more faithful to the generative process.

Neural policies usually refer to neural networks in reinforcement learning that output action distributions, whose output layer can be analogized to logits. Validating RC-DS on these models shows that the method is not limited to synthetic data or pure language models, but is applicable to a broad range of activation readout structures.

Practical Implications for Interpretability Research

The value of this work lies in providing an information-preserving methodological tool. RC-DS allows researchers to separate and control the contribution of readout scaling to sparse learning, rather than treating it as a confounding variable.

The study also distinguishes reconstruction performance from the usefulness of task-specific features through independent label detection and steering tests. This means that an SAE performing well on reconstruction metrics does not necessarily mean that the features it extracts are equally useful in downstream tasks.

The implication for practitioners is that when designing an SAE training pipeline, three factors need to be considered simultaneously: the readout condition number, the data generation distribution, and the training coordinates. Ignoring any one of them may cause feature recovery performance to fall far below expectations.

One-line Summary

Readout scaling changes the features an SAE learns, and RC-DS significantly improves sparse factor recovery by normalizing decision-space coordinates.

Paragraph Summary

This article discusses a commonly overlooked variable in sparse autoencoders (SAEs): readout scaling. The study proposes the RC-DS method, which removes singular value gains without losing information by normalizing readout decision-space coordinates, thereby isolating the effect of training geometry. Controlled experiments show that sparse factor recovery is best when training coordinates align with the factor generation space; changing the generative distribution reverses the choice of optimal training coordinates. Under a condition number of 100, RC-DS improves the average sparse factor recovery rate from 0.506 to 0.913. On neural policies and three language model checkpoints, RC-DS achieves higher gain-free reconstruction means in all 18 task-sparsity settings, but centered logit SAEs better reconstruct output logits in 17 settings. This shows that reconstructing logits and recovering sparse factors are different objectives, and that training sufficiency can partially bridge the difference.