Single-Head Attention
A simplified mechanism in Transformers that retains only one attention head, often used in theoretical analysis to isolate the complexity of multi-head interactions.
Timeline (last 90 days)
A positional kernel is the similarity function induced by positional encoding in attention mechanisms; in single-head attention, the positional kernel typically appears as a scalar function k(i,j), depending only on the offset i-j for relative positional encoding and on the specific values of i and j for absolute positional encoding.
Ramsauer et al. proved in 2020 that single-head attention in a Transformer is mathematically equivalent to one retrieval update of a modern Hopfield network: the query corresponds to the current state, the keys correspond to the stored patterns, and the attention weights correspond to the retrieval distribution.
The authors point out that the one-head model has structural limitations at the boundary, and suggest that reflected symmetrization may bring improvements
All Facts (3)
A positional kernel is the similarity function induced by positional encoding in attention mechanisms; in single-head attention, the positional kernel typically appears as a scalar function k(i,j), depending only on the offset i-j for relative positional encoding and on the specific values of i and j for absolute positional encoding.
50%UnverifiedRamsauer et al. proved in 2020 that single-head attention in a Transformer is mathematically equivalent to one retrieval update of a modern Hopfield network: the query corresponds to the current state, the keys correspond to the stored patterns, and the attention weights correspond to the retrieval distribution.
50%UnverifiedThe authors point out that the one-head model has structural limitations at the boundary, and suggest that reflected symmetrization may bring improvements
50%