ConSDL
A method that improves dictionary learning consistency by replacing the linear encoder with per-sample sparse coding. It aims to address the problem that sparse autoencoders (SAEs) yield inconsistent dictionaries across different random seeds.
Timeline (last 90 days)
The paper "Beyond SAEs: Consistent Concept Discovery with Sparse Coding" proposes the ConSDL method.
ConSDL replaces the standard linear encoder with per-sample sparse coding and achieves state-of-the-art cross-seed consistency on multiple vision and language tasks.
ConSDL solves the sparse coding optimization problem separately for each sample, which amounts to introducing an inner loop into training to find, for each activation vector, the optimal sparse representation under the current dictionary.
On 16 synthetic data settings, ConSDL's dictionary recovery is near-perfect, accurately recovering dictionary atoms and sparse coefficients.
On real benchmarks in four vision and language settings, ConSDL improves dictionary consistency by an average of 17.6% and coefficient consistency by 35.9% over six SAE baseline methods.
During ConSDL training, the per-sample sparse coding problem can be solved with methods such as the iterative shrinkage-thresholding algorithm (ISTA) or the alternating direction method of multipliers (ADMM).
ConSDL's per-sample optimization increases computational overhead but significantly improves reproducibility.
ConSDL is not proposed to completely reject SAEs, but rather to point out the limitations of the linear encoder as a design choice.
All Facts (8)
The paper "Beyond SAEs: Consistent Concept Discovery with Sparse Coding" proposes the ConSDL method.
50%UnverifiedConSDL replaces the standard linear encoder with per-sample sparse coding and achieves state-of-the-art cross-seed consistency on multiple vision and language tasks.
50%UnverifiedConSDL solves the sparse coding optimization problem separately for each sample, which amounts to introducing an inner loop into training to find, for each activation vector, the optimal sparse representation under the current dictionary.
50%UnverifiedOn 16 synthetic data settings, ConSDL's dictionary recovery is near-perfect, accurately recovering dictionary atoms and sparse coefficients.
50%UnverifiedOn real benchmarks in four vision and language settings, ConSDL improves dictionary consistency by an average of 17.6% and coefficient consistency by 35.9% over six SAE baseline methods.
50%UnverifiedDuring ConSDL training, the per-sample sparse coding problem can be solved with methods such as the iterative shrinkage-thresholding algorithm (ISTA) or the alternating direction method of multipliers (ADMM).
50%UnverifiedConSDL's per-sample optimization increases computational overhead but significantly improves reproducibility.
50%UnverifiedConSDL is not proposed to completely reject SAEs, but rather to point out the limitations of the linear encoder as a design choice.
50%