Longformer
A Transformer model for long-document processing proposed by the Allen Institute for AI. It combines local sliding-window attention with global attention in a sparse attention mechanism, reducing the computational complexity of attention from O(n²) to O(n). Longformer is a foundational reference work in the field of long-text understanding.
Timeline (last 90 days)
Longformer and BigBird use fixed local windows combined with global token sparse patterns, reducing complexity to O(N) but sacrificing direct connections between arbitrary positions.
Longforger uses a fixed sliding window plus global token pattern of sparse attention.
All Facts (3)
Longformer由Allen AI提出，采用滑动窗口局部注意力加全局注意力token的组合策略，将复杂度从O(n²)降至O(n)
85%UnverifiedLongformer and BigBird use fixed local windows combined with global token sparse patterns, reducing complexity to O(N) but sacrificing direct connections between arbitrary positions.
50%UnverifiedLongforger uses a fixed sliding window plus global token pattern of sparse attention.
50%