Mechanistic Interpretability

Mechanistic interpretability research aims to reverse-engineer the internal computational processes of neural networks, understanding how a model reaches its conclusions rather than merely observing the conclusions themselves.

The goal of Mechanistic Interpretability is to reverse-engineer the internal computational mechanisms of neural networks; researchers at Anthropic, OpenAI, and other institutions attempt to identify 'reasoning circuits' within networks.

For cryptographically encapsulated hidden sub-networks, the activation patterns in the absence of a trigger are indistinguishable from random noise, rendering mechanistic interpretability tools unable to analyze them.

The Linear Representation Hypothesis originates from mechanistic interpretability research into the internal mechanisms of large language models, such as the conceptual arithmetic 'king − male + female ≈ queen'.

Goodfire AI focuses on mechanistic interpretability research for large language models, and its technical approach significantly overlaps with the sparse autoencoder direction advocated by Anthropic's interpretability team

Mechanistic interpretability uses reverse engineering to locate the neurons, attention heads, or circuits in a model that correspond to specific functions.

Mechanistic interpretability techniques currently have very low resolution for complex semantics and are far from being able to reliably detect model intentions.

Anthropic published a paper in 2024 demonstrating how to locate neuron groups related to concepts such as 'honesty' and 'harm recognition' in the Claude model, and how intervening in these neurons can alter model behavior.

Interpretability research attempts to understand the internal working mechanisms of models through methods such as mechanistic interpretability, attention visualization, and feature attribution.

In 2024, Anthropic extended mechanistic interpretability methods to Claude 3 Sonnet-level models, identifying millions of interpretable features, including features corresponding to concepts such as deception and sycophancy.

Anthropic's research team successfully identified feature vectors inside large language models corresponding to concepts such as deception and sycophancy using sparse autoencoders in 2023–2024.

Between 2023 and 2024, Anthropic identified millions of interpretable features in the Claude model, including the Golden Gate Bridge feature experiment

Organizations such as Anthropic are reverse-engineering the computational structures inside neural networks through mechanistic interpretability research—such as identifying features and circuits within models—but current techniques remain far from sufficient to understand the behavior of complete models.

The Abliteration technique identifies and suppresses the direction vector responsible for refusal behavior in the model's residual stream, using orthogonal projection to remove that direction from the weight matrices, thereby eliminating refusal behavior without retraining.

Mechanistic interpretability techniques such as activation analysis and attention intervention require loading the full model weights and accessing intermediate states during the forward pass, and cannot be accomplished through a pure API approach.

Activation patching是机械可解释性领域的核心技术之一，其做法是在模型处理输入A时将某层或注意力头的激活值替换为处理输入B时的对应激活值以观察输出变化