ALiBi
A positional encoding scheme for Transformer models that encodes position information implicitly by adding a linear penalty proportional to the distance between tokens directly to attention scores, rather than modifying token embeddings. The approach is straightforward to implement and demonstrates favorable extrapolation behavior on sequences longer than those seen during training.
Timeline (last 90 days)
ALiBi directly superimposes a linear penalty term proportional to distance onto attention scores, achieving a clean implementation that is friendly to length extrapolation
Positional encoding improvements such as RoPE and ALiBi allow models to extrapolate to sequence lengths not seen during training.
Ultra-long context technologies include ALiBi, extrapolation improvements to RoPE, and distributed attention mechanisms such as Ring Attention, which are used to address the bottleneck of the Transformer attention mechanism's computational complexity scaling quadratically with sequence length.
Context window extension technologies such as NTK-aware scaling of RoPE, ALiBi, and Ring Attention have pushed context length from the early 4K to 128K or even millions of tokens.
Context window extension techniques such as NTK-aware scaling of RoPE, ALiBi, and Ring Attention have pushed context length from the early 4K to 128K or even millions of tokens, but face the limitations of quadratic growth in computational cost and long-range attention decay.
All Facts (6)
上下文窗口扩展得益于ALiBi位置编码、RoPE的外推扩展、Flash Attention和Ring Attention等技术突破
85%UnverifiedPositional encoding improvements such as RoPE and ALiBi allow models to extrapolate to sequence lengths not seen during training.
60%UnverifiedALiBi directly superimposes a linear penalty term proportional to distance onto attention scores, achieving a clean implementation that is friendly to length extrapolation
50%UnverifiedUltra-long context technologies include ALiBi, extrapolation improvements to RoPE, and distributed attention mechanisms such as Ring Attention, which are used to address the bottleneck of the Transformer attention mechanism's computational complexity scaling quadratically with sequence length.
50%UnverifiedContext window extension technologies such as NTK-aware scaling of RoPE, ALiBi, and Ring Attention have pushed context length from the early 4K to 128K or even millions of tokens.
50%UnverifiedContext window extension techniques such as NTK-aware scaling of RoPE, ALiBi, and Ring Attention have pushed context length from the early 4K to 128K or even millions of tokens, but face the limitations of quadratic growth in computational cost and long-range attention decay.
50%