BigBird
A Transformer model proposed by Google that employs a sparse attention mechanism combining three attention patterns: random attention, local window attention, and global attention. This design reduces computational complexity to O(n), making the model suitable for long-sequence text processing tasks.
Timeline (last 90 days)
Longformer and BigBird use fixed local windows combined with global token sparse patterns, reducing complexity to O(N) but sacrificing direct connections between arbitrary positions.
BigBird employs a hybrid sparse strategy combining random attention and window attention.
BigBird theoretically proved that sparse attention can be Turing complete.
All Facts (3)
BigBird employs a hybrid sparse strategy combining random attention and window attention.
70%UnverifiedLongformer and BigBird use fixed local windows combined with global token sparse patterns, reducing complexity to O(N) but sacrificing direct connections between arbitrary positions.
50%UnverifiedBigBird theoretically proved that sparse attention can be Turing complete.
50%