Sparse Autoencoder

Sparse autoencoders (SAEs) are one of the main technical approaches attempting to disentangle superposition and extract monosemantic features.

Sparse autoencoders (SAEs) introduce sparsity regularization in addition to reconstruction loss, encoding activation vectors as overcomplete sparse representations.

Under the new paradigm, the sparsity constraint of SAEs is regarded as a kind of human preference bias, rather than the only correct decomposition of the network's true internal structure.

In 2024, Anthropic used sparse autoencoders to extract 34 million features from Claude Sonnet and located the internal circuits corresponding to specific concepts.

Neel Nanda emphasizes that SAEs remain irreplaceable on the residual stream.

Labs such as Anthropic and OpenAI treat sparse autoencoders (SAEs) as the mainstream approach for disentangling superimposed features in neural networks.

The features disentangled by SAEs actually have response widths several times coarser than the original neurons, meaning that SAEs may create a mathematical illusion.

Readout scaling refers to applying a diagonal scaling or linear transformation to the activation vector before feeding the activations into the SAE, thereby changing the relative weight of each dimension in the reconstruction objective.

Sparse factors are recovered best when the training process recovers the space in which the factors were generated.

Holding the readout condition number fixed, changing the generative distribution reverses which training coordinate best recovers the known factors.

An SAE performing well on reconstruction metrics does not mean the features it extracts are equally useful in downstream tasks.

When designing an SAE training pipeline, three factors need to be considered simultaneously: the readout condition number, the data generative distribution, and the training coordinate. Ignoring any one of them can lead to feature recovery far below expectations.

A sparse autoencoder is an unsupervised learning architecture that compresses high-dimensional model activation vectors into an overcomplete sparse latent space and then reconstructs the original activations, forcing each latent variable to be sensitive to only a few input patterns.

In large-model interpretability research, SAEs are used to decompose polysemantic, dense neuron activations into a large number of monosemantic, readable latent features; typical work includes Anthropic's "dictionary learning" and OpenAI's "sparse circuits" attempts.

The paper recommends establishing intervention attribution as the principled objective for selecting SAE features for intervention, and this objective can be estimated via gradients.

When evaluating SAE quality, one should distinguish readability and reconstruction fidelity at the discovery level from causal efficacy at the intervention level; the two are not automatically consistent and need to be measured separately.

For a large class of response functions, observational feature discovery evidence generally cannot identify intervention attribution, which is a structural conclusion rather than a flaw of any specific method.

In measurements across four model-SAE pairings, the ratio of probe attribution to intervention attribution ranged from tens to hundreds at the median, with probes systematically overestimating the intervention efficacy of features.

Through tools such as sparse autoencoders and circuit analysis, mechanistic interpretability has already identified specific circuits such as 'indirect object identification' and 'polysemous word disambiguation' in small models.

Transcoder is a feature decomposition tool in the field of mechanistic interpretability that trains a small network specifically targeting MLP layers, going from input to a sparse hidden layer to output, to replace the original MLP.