Activation Space
In interpretability research, the analytical object concerned with the patterns of neuron activations a network exhibits for a given input; it is commonly studied with methods such as probe classifiers and sparse autoencoders.
Timeline (last 90 days)
In the Transformer architecture, the input of each token is processed through multiple layers of self-attention and feed-forward networks, and each layer produces a high-dimensional vector representation called a "hidden state"; the hidden states of all layers and all tokens together constitute the "activation space."
In the activation space, if the model has a stable and consistent representation of a concept, the related hidden states cluster to form a low-entropy, high-density cluster; if the model lacks knowledge of the concept, the activation vectors scatter and exhibit a high-entropy fragmented distribution.
Weight space and activation space are two analysis objects at different levels in interpretability research.
All Facts (3)
In the Transformer architecture, the input of each token is processed through multiple layers of self-attention and feed-forward networks, and each layer produces a high-dimensional vector representation called a "hidden state"; the hidden states of all layers and all tokens together constitute the "activation space."
50%UnverifiedIn the activation space, if the model has a stable and consistent representation of a concept, the related hidden states cluster to form a low-entropy, high-density cluster; if the model lacks knowledge of the concept, the activation vectors scatter and exhibit a high-entropy fragmented distribution.
50%UnverifiedWeight space and activation space are two analysis objects at different levels in interpretability research.
50%