Residual Stream
The primary information pathway running throughout a Transformer architecture, onto which the outputs of attention heads and feed-forward networks at each layer are additively superimposed. It serves as the main object of manipulation in interpretability methods such as activation patching.
Timeline (last 90 days)
Neel Nanda emphasizes that SAEs remain irreplaceable on the residual stream.
The basic operation of activation steering is to find an activation direction vector representing a certain concept and add it to intermediate-layer activations during inference, thereby intervening in the output without changing the weights.
Within effective component mechanisms, the steering effect is further concentrated in approximately 50% of the residual stream's dimensions, which retain 85%–98% of the component-mechanism baseline effect.
The residual stream is the core information channel of the Transformer architecture, where attention heads and MLP modules at each layer interact with the residual stream in a read-process-write-back manner.
Injecting vectors into the residual stream is a commonly used causal intervention technique in mechanistic interpretability, acting directly on the model's intermediate representations rather than at the vocabulary level.
In language model-driven war games, the model's language output simultaneously determines what an agent attempts to do and what becomes the simulated reality.
The framework treats the residual stream as a communication bus between the model's layers, from which attention heads and MLPs read information and write back.
Arditi-style editing originates from research by Andy Arditi et al. in 2024, whose core finding is that the refusal behavior of large language models can be localized to a single directional vector in the residual stream.
All Facts (8)
Neel Nanda emphasizes that SAEs remain irreplaceable on the residual stream.
50%UnverifiedThe basic operation of activation steering is to find an activation direction vector representing a certain concept and add it to intermediate-layer activations during inference, thereby intervening in the output without changing the weights.
50%UnverifiedWithin effective component mechanisms, the steering effect is further concentrated in approximately 50% of the residual stream's dimensions, which retain 85%–98% of the component-mechanism baseline effect.
50%UnverifiedThe residual stream is the core information channel of the Transformer architecture, where attention heads and MLP modules at each layer interact with the residual stream in a read-process-write-back manner.
50%UnverifiedInjecting vectors into the residual stream is a commonly used causal intervention technique in mechanistic interpretability, acting directly on the model's intermediate representations rather than at the vocabulary level.
50%UnverifiedIn language model-driven war games, the model's language output simultaneously determines what an agent attempts to do and what becomes the simulated reality.
50%UnverifiedThe framework treats the residual stream as a communication bus between the model's layers, from which attention heads and MLPs read information and write back.
50%UnverifiedArditi-style editing originates from research by Andy Arditi et al. in 2024, whose core finding is that the refusal behavior of large language models can be localized to a single directional vector in the residual stream.
50%