Gated Linear Attention
A linearized attention mechanism that replaces traditional softmax attention with a recurrent linear formulation, reducing computational complexity from O(N²) to O(N). A forget-gate mechanism is incorporated to retain selective information modeling capability.
Timeline (last 90 days)
GTR-Depth uses the same backbone network as its predecessor, containing 12 gated linear attention blocks.
Gated Linear Attention (GLA) reduces computational complexity from O(N²) to O(N) through a linearized attention mechanism.
Gated Linear Attention replaces the softmax(QKᵀ)V operation in standard attention with a recurrent linear form, where the gating component is typically implemented as a forget gate applied to the hidden state.
All Facts (3)
GTR-Depth uses the same backbone network as its predecessor, containing 12 gated linear attention blocks.
50%UnverifiedGated Linear Attention (GLA) reduces computational complexity from O(N²) to O(N) through a linearized attention mechanism.
50%UnverifiedGated Linear Attention replaces the softmax(QKᵀ)V operation in standard attention with a recurrent linear form, where the gating component is typically implemented as a forget gate applied to the hidden state.
50%