Positional Encoding
A mechanism in Transformer models that enables the model to perceive the sequential position of tokens. Principal approaches include fixed sinusoidal encoding, learnable absolute position embeddings, and relative position encodings such as RoPE. These approaches differ substantially in their ability to generalize to sequence lengths beyond those seen during training.
Core Facts
Timeline (last 90 days)
The performance decline caused by 'lost in the middle' follows a U-shaped curve rather than linear decay; its mechanism may be related to Transformer attention allocation and positional encoding.
The causes of the lost-in-the-middle phenomenon are usually attributed to the decay effect of positional encoding, the fact that beginning and end content in training data is more often associated with answers, and the dilution of the attention softmax over long sequences.
STCNet adopts a dual-domain collaborative strategy, introducing physics-driven positional encoding in the projection domain and temporal encoding in the latent space, jointly optimizing the projection domain and the image domain.
Positional encoding was first widely used in the Transformer architecture, and its role is to inject the order information of elements in a sequence into the model, because the pure attention mechanism itself is insensitive to order.
Traditional sinusoidal/cosine positional encodings are designed for uniform integer step sizes and have difficulty expressing geometric relationships at irregular intervals.
Positional encoding is used to inject word order information into the model, because the attention mechanism itself does not distinguish positional order.
Positional Encoding solves the problem of the attention mechanism being inherently orderless and unaware of word sequence by attaching positional information to each word.
Transformer uses positional encoding based on sine and cosine functions, with wavelengths arranged in a geometric progression, where the shortest and longest differ by a factor of ten thousand.
The vector input to the model equals the word embedding plus the positional embedding; for example, 7 characters generate 7 word embeddings and 7 positional embeddings that are added together to produce the actual input
All Facts (10)
Because the Transformer self-attention mechanism exhibits positional bias, earlier input content has a stronger anchoring effect on subsequent generation, which is the underlying reason for the context anchoring effect of system prompts
65%UnverifiedThe performance decline caused by 'lost in the middle' follows a U-shaped curve rather than linear decay; its mechanism may be related to Transformer attention allocation and positional encoding.
50%UnverifiedThe causes of the lost-in-the-middle phenomenon are usually attributed to the decay effect of positional encoding, the fact that beginning and end content in training data is more often associated with answers, and the dilution of the attention softmax over long sequences.
50%UnverifiedSTCNet adopts a dual-domain collaborative strategy, introducing physics-driven positional encoding in the projection domain and temporal encoding in the latent space, jointly optimizing the projection domain and the image domain.
50%UnverifiedPositional encoding was first widely used in the Transformer architecture, and its role is to inject the order information of elements in a sequence into the model, because the pure attention mechanism itself is insensitive to order.
50%UnverifiedTraditional sinusoidal/cosine positional encodings are designed for uniform integer step sizes and have difficulty expressing geometric relationships at irregular intervals.
50%UnverifiedPositional encoding is used to inject word order information into the model, because the attention mechanism itself does not distinguish positional order.
50%UnverifiedPositional Encoding solves the problem of the attention mechanism being inherently orderless and unaware of word sequence by attaching positional information to each word.
50%UnverifiedTransformer uses positional encoding based on sine and cosine functions, with wavelengths arranged in a geometric progression, where the shortest and longest differ by a factor of ten thousand.
50%UnverifiedThe vector input to the model equals the word embedding plus the positional embedding; for example, 7 characters generate 7 word embeddings and 7 positional embeddings that are added together to produce the actual input
50%