Superposition Hypothesis
The superposition hypothesis is a theoretical framework for explaining the representational capacity of neural networks. It holds that when the number of concepts exceeds the representation dimension, neural networks encode concepts in non-orthogonal dense directions that coexist through superposition, with the overall representation being a linear combination of a small number of active concepts.
Timeline (last 90 days)
For deep learning representations, the superposition hypothesis suggests that the sparse superposition property of user embeddings holds approximately.
The core idea of the superposition hypothesis is: when the number of concepts a model needs to represent exceeds the dimensionality of its representation space, neural networks tend to encode these concepts using non-orthogonal dense directions.
All Facts (2)
For deep learning representations, the superposition hypothesis suggests that the sparse superposition property of user embeddings holds approximately.
50%UnverifiedThe core idea of the superposition hypothesis is: when the number of concepts a model needs to represent exceeds the dimensionality of its representation space, neural networks tend to encode these concepts using non-orthogonal dense directions.
50%