Logit lens
A training-free method for reading model activations that multiplies the residual stream representation of an intermediate layer by the model's unembedding matrix, projecting it back into vocabulary space to read the next-token prediction of that layer.
Timeline (last 90 days)
The activation-reading tools evaluated are divided into single-token readers and multi-token readers: the single-token side includes Logit lens, J-lens, R-lens, and Tuned lens; the multi-token side includes Natural Language Autoencoders, Sparse Autoencoders, Patchscopes, Template lens, and Oracle Lens.
Logit lens directly multiplies the residual stream representation at a certain intermediate layer by the model's final unembedding matrix, projects it back into vocabulary space, and reads out the most likely next word for that layer; its advantage is that it requires no training and is almost zero-cost, and its disadvantage is that the semantics gradually drift as the number of layers deepens.
In terms of readout method, J/R/Logit-lens takes top-10 tokens every 4 layers from layers 20 to 60 of the 64-layer Qwen-3.6-27B.
Single-token readers cannot cover important workspace content because they are bag-of-words.
All Facts (4)
The activation-reading tools evaluated are divided into single-token readers and multi-token readers: the single-token side includes Logit lens, J-lens, R-lens, and Tuned lens; the multi-token side includes Natural Language Autoencoders, Sparse Autoencoders, Patchscopes, Template lens, and Oracle Lens.
50%UnverifiedLogit lens directly multiplies the residual stream representation at a certain intermediate layer by the model's final unembedding matrix, projects it back into vocabulary space, and reads out the most likely next word for that layer; its advantage is that it requires no training and is almost zero-cost, and its disadvantage is that the semantics gradually drift as the number of layers deepens.
50%UnverifiedIn terms of readout method, J/R/Logit-lens takes top-10 tokens every 4 layers from layers 20 to 60 of the 64-layer Qwen-3.6-27B.
50%UnverifiedSingle-token readers cannot cover important workspace content because they are bag-of-words.
50%