Model
Linear Transformer
A Transformer variant that replaces softmax attention with a linear kernel function and reorders computations to reduce attention complexity from quadratic to linear. It is considered a foundational early work in linearized attention.
Timeline (last 90 days)
Oct 12
Performer and Linear Transformer are early representative works on linearized attention, and generally suffer from rising perplexity and degraded long-range dependency modeling.
Unverified50%