Sparse Coding Innovation: How ConSDL Achieves Cross-Training Consistent Concept Discovery

Sparse autoencoders (SAEs) have been the mainstream tool for interpreting the internal representations of foundation models, revealing the concepts learned by models by decomposing activations into sparse combinations of learned dictionary atoms. However, the dictionaries trained by SAEs often differ significantly across random seeds, which severely undermines the reliability and reproducibility of SAE-based analyses. A recent study titled "Beyond SAEs: Consistent Concept Discovery with Sparse Coding" proposes Consistent Sparse Dictionary Learning (ConSDL), which replaces the standard linear encoder with per-sample sparse coding, achieving state-of-the-art cross-seed consistency on multiple vision and language tasks.

The Reliability Dilemma of SAEs: Dictionaries Fluctuate with Training

The core idea of SAEs is to learn an overcomplete dictionary such that model activations can be approximated by linear combinations of a small number of dictionary atoms. Ideally, these atoms should correspond to interpretable concepts, such as "cat ears" or "negative tone." But in actual training, different initializations, data orders, and optimization trajectories lead to vastly different learned dictionaries. An analysis pointed out that this inconsistency makes it difficult to align SAE features across different studies, and even the concepts extracted from the same model can drift between training runs.

Researchers revealed the root cause of the problem through synthetic experiments: standard SAEs use a single-pass linear encoder to map activations to sparse coefficients, and the coefficients obtained from a single forward pass are suboptimal relative to per-sample optimization. In other words, the linear encoder limits the quality of dictionary learning, thereby amplifying the impact of training randomness.

Overcomplete Dictionaries and Sparsity Constraints
SAE dictionaries typically have far more atoms than the input dimension (i.e., overcomplete), in order to cover richer concept directions with combinations of a small number of atoms. Sparsity constraints (such as L1 regularization) force each activation vector to activate only a few atoms, so that each atom tends to correspond to an interpretable concept. However, the optimization problem of overcomplete dictionaries is itself non-convex and has a large number of local optima; different random seeds lead training toward different local optima, causing dictionary atoms to "rotate" or recombine in concept space. Although reconstruction errors are similar, the meanings of atoms are inconsistent. This is one of the fundamental reasons for poor cross-seed consistency.

ConSDL: Improving Dictionary Consistency with Per-Sample Sparse Coding

The core change in ConSDL is simple: instead of using a linear encoder to predict sparse coefficients in one pass, it solves a sparse coding optimization problem separately for each sample. This is equivalent to introducing an inner loop during training that finds the optimal sparse representation for each activation vector given the current dictionary.

Researchers validated ConSDL's dictionary recovery capability on 16 synthetic data settings, with near-perfect results—both the dictionary atoms themselves and the corresponding sparse coefficients can be accurately recovered. This provides a solid theoretical foundation for subsequent application to real models.

On real benchmarks across four vision and language settings, compared with six SAE baseline methods, ConSDL improved dictionary consistency by 17.6% and coefficient consistency by 35.9% on average. This means that dictionaries trained under different random seeds are better aligned, and downstream analyses based on these dictionaries (such as concept discovery and feature attribution) will be more stable and trustworthy.

Per-Sample Sparse Coding and Inner-Loop Optimization
Per-sample sparse coding means independently solving a sparse coding problem for each input activation vector: given the current dictionary, find the sparsest coefficient representation of that vector (or the one with the smallest reconstruction error under the sparsity constraint). This differs from the shared linear encoder in SAEs, which predicts the coefficients of all samples in one pass. In ConSDL training, this is equivalent to adding an inner-loop optimization on top of dictionary updates, which can usually be solved with algorithms such as the iterative shrinkage-thresholding algorithm (ISTA) or the alternating direction method of multipliers (ADMM). Since the coefficients of each sample are optimized to the optimal solution under the current dictionary, the coefficient signals on which dictionary updates depend are more accurate, thereby reducing dictionary drift caused by encoder approximation errors.

Implications for Interpretability Research

ConSDL is not intended to completely reject SAEs, but rather to point out the limitations of the design choice of the linear encoder. Although per-sample optimization increases computational overhead, it brings a significant improvement in reproducibility. For research that requires comparing concepts across models and training runs, this consistency is crucial.

From a broader perspective, this study reminds us that interpretability tools themselves also need to be "interpreted" and scrutinized. When we treat SAE features as the "true concepts" inside a model, we must first confirm that these features are not products of training noise. ConSDL provides a feasible path for improvement and also encourages subsequent work to explore more efficient and more stable sparse dictionary learning schemes.

Measuring Coefficient Consistency and Dictionary Consistency
Dictionary consistency usually refers to the degree of alignment between sets of dictionary atoms trained with different random seeds, commonly measured by cosine similarity between matched atoms or by distances based on optimal transport; coefficient consistency measures the similarity of sparse encodings obtained for the same input under different seeds. Together they determine whether SAE-based downstream analyses can be reproduced across runs. ConSDL shows a more significant improvement in coefficient consistency (35.9%), indicating that per-sample optimization not only improves the dictionary itself but also makes the encoded representation of each sample more stable, which is especially critical for tasks such as feature attribution and concept discovery.

Conclusion

By replacing the linear encoder in SAEs with per-sample sparse coding, ConSDL achieves near-perfect dictionary recovery and significantly improved cross-seed consistency in both synthetic and real settings. This study takes an important step toward building more reliable concept discovery tools and echoes the growing emphasis on reproducibility in the interpretability field.

For practitioners, if you are using SAEs for model analysis and encountering unstable features, ConSDL's approach is worth considering—sometimes a seemingly minor architectural adjustment (from one-pass encoding to per-sample optimization) can bring a qualitative leap in consistency.

One-line Summary

ConSDL replaces the linear encoder with per-sample sparse coding, significantly improving the cross-seed consistency of SAE dictionaries.

Paragraph Summary

Sparse autoencoders (SAEs) are the mainstream tool for interpreting the internal representations of foundation models, but the dictionaries they learn differ significantly across random seeds, affecting the reproducibility of analyses. A recent study proposes Consistent Sparse Dictionary Learning (ConSDL), which replaces the standard linear encoder with per-sample sparse coding, so that each activation vector is individually optimized during training into the optimal sparse representation under the current dictionary. Theoretical analysis shows that the coefficients obtained by the single-pass linear encoding of standard SAEs are suboptimal, which amplifies training randomness. ConSDL achieves near-perfect dictionary and coefficient recovery on 16 synthetic datasets; on four real vision and language benchmarks, compared with six SAE baselines, dictionary consistency improves by 17.6% and coefficient consistency by 35.9% on average. This work points out the limitations of linear encoders and provides a feasible path for building more reliable concept discovery tools.