Performer
A linearized Transformer model based on random feature approximation that approximates softmax attention with linear complexity using random feature maps, reducing the computational overhead of sequence modeling.
Timeline (last 90 days)
Performer and Linear Transformer are early representative works on linearized attention, and generally suffer from rising perplexity and degraded long-range dependency modeling.
Linear attention methods (such as Performer and RWKV) compress complexity to O(N) via kernel function approximation, but their precise recall capability is typically significantly weaker than full attention.
All Facts (2)
Performer and Linear Transformer are early representative works on linearized attention, and generally suffer from rising perplexity and degraded long-range dependency modeling.
50%UnverifiedLinear attention methods (such as Performer and RWKV) compress complexity to O(N) via kernel function approximation, but their precise recall capability is typically significantly weaker than full attention.
50%