Attention Visualization
An interpretability method that displays the distribution of attention weights between tokens. It is intuitive but difficult to map directly onto semantic computation.
Timeline (last 90 days)
Interpretability tools for large models currently fall roughly into attention visualization, activation probing, feature attribution (such as gradient-based methods), and more engineering-oriented logging and tracing.
Attention visualization shows the distribution of attention weights between tokens; activation probes train linear classifiers on hidden layers to test whether information is decodable, which is correlational evidence; feature attribution measures the contribution of each input dimension to the output.
Current mainstream interpretability approaches include post-hoc attribution methods (such as attention visualization, SHAP) and mechanistic interpretability (such as sparse autoencoders, circuit analysis), with the latter attempting to locate identifiable "feature circuits" at the level of neuron activations.
All Facts (3)
Interpretability tools for large models currently fall roughly into attention visualization, activation probing, feature attribution (such as gradient-based methods), and more engineering-oriented logging and tracing.
50%UnverifiedAttention visualization shows the distribution of attention weights between tokens; activation probes train linear classifiers on hidden layers to test whether information is decodable, which is correlational evidence; feature attribution measures the contribution of each input dimension to the output.
50%UnverifiedCurrent mainstream interpretability approaches include post-hoc attribution methods (such as attention visualization, SHAP) and mechanistic interpretability (such as sparse autoencoders, circuit analysis), with the latter attempting to locate identifiable "feature circuits" at the level of neuron activations.
50%