Position Kernel
A similarity function induced by positional encoding in attention mechanisms that measures the relationship between query positions and key positions, whose mathematical properties determine the possibility of length generalization.
Timeline (last 90 days)
The core tool of the theory is the simulated transfer operator T_L, which reevaluates a selected positional kernel that realizes length L_0 on a longer grid.
A positional kernel is the similarity function induced by positional encoding in attention mechanisms; in single-head attention, the positional kernel typically appears as a scalar function k(i,j), depending only on the offset i-j for relative positional encoding and on the specific values of i and j for absolute positional encoding.
All Facts (2)
The core tool of the theory is the simulated transfer operator T_L, which reevaluates a selected positional kernel that realizes length L_0 on a longer grid.
50%UnverifiedA positional kernel is the similarity function induced by positional encoding in attention mechanisms; in single-head attention, the positional kernel typically appears as a scalar function k(i,j), depending only on the offset i-j for relative positional encoding and on the specific values of i and j for absolute positional encoding.
50%