Activation Steering
A causal intervention technique in mechanistic interpretability that injects pre-extracted concept direction vectors into a model's residual stream during forward propagation. This direct manipulation of internal model states is used to study causal relationships between specific representations and model behaviors.
Core Facts
Timeline (last 90 days)
Activation steering is a technique that intervenes in the model's internal hidden-layer representations at inference time, with representative methods including contrastive activation addition (CAA) and inference-time intervention (ITI).
Knowledge extraction enhancement (KEE) techniques include three categories—activation steering, decoding adjustments, and prompt engineering—which modify the inference process and may reinforce existing predictions or surface answers that are otherwise hard to reach.
Activation-based steering directly intervenes in the model's internal activation directions, whereas prompt-based steering influences indirectly through input text, and the difference in their performance under accumulated multi-turn error is the core of the study's comparison.
The basic operation of activation steering is to find an activation direction vector representing a certain concept and add it to intermediate-layer activations during inference, thereby intervening in the output without changing the weights.
Linear steering relies on the linear representation hypothesis; when the concept distribution itself is curved, shifting activations along a fixed direction deviates from the true concept manifold, leading to control distortion or unnatural intermediate states.
A study published on arXiv (arXiv:2609.38205) investigated the effects of system prompts on internal computations within Transformers.
Activation steering or representation engineering applies intervention vectors directly to the model's internal activation layers, altering the generation direction without modifying the model's weights.
Injecting vectors into the residual stream is a commonly used causal intervention technique in mechanistic interpretability, acting directly on the model's intermediate representations rather than at the vocabulary level.
In language model-driven war games, the model's language output simultaneously determines what an agent attempts to do and what becomes the simulated reality.
Model steering influences model output behavior during inference by directly modifying the activation vectors of intermediate layers, without modifying weights.
All Facts (10)
Model steering influences model output behavior during inference by directly modifying the activation vectors of intermediate layers, without modifying weights.
85%UnverifiedActivation steering is a technique that intervenes in the model's internal hidden-layer representations at inference time, with representative methods including contrastive activation addition (CAA) and inference-time intervention (ITI).
50%UnverifiedKnowledge extraction enhancement (KEE) techniques include three categories—activation steering, decoding adjustments, and prompt engineering—which modify the inference process and may reinforce existing predictions or surface answers that are otherwise hard to reach.
50%UnverifiedActivation-based steering directly intervenes in the model's internal activation directions, whereas prompt-based steering influences indirectly through input text, and the difference in their performance under accumulated multi-turn error is the core of the study's comparison.
50%UnverifiedThe basic operation of activation steering is to find an activation direction vector representing a certain concept and add it to intermediate-layer activations during inference, thereby intervening in the output without changing the weights.
50%UnverifiedLinear steering relies on the linear representation hypothesis; when the concept distribution itself is curved, shifting activations along a fixed direction deviates from the true concept manifold, leading to control distortion or unnatural intermediate states.
50%UnverifiedA study published on arXiv (arXiv:2609.38205) investigated the effects of system prompts on internal computations within Transformers.
50%UnverifiedActivation steering or representation engineering applies intervention vectors directly to the model's internal activation layers, altering the generation direction without modifying the model's weights.
50%UnverifiedInjecting vectors into the residual stream is a commonly used causal intervention technique in mechanistic interpretability, acting directly on the model's intermediate representations rather than at the vocabulary level.
50%UnverifiedIn language model-driven war games, the model's language output simultaneously determines what an agent attempts to do and what becomes the simulated reality.
50%