FourierQK
A research paper published on arXiv that systematically investigates optimal filter-shape design for attention mechanisms in the frequency domain. It introduces the concept of admissibility, proposes the leakage–coverage law, and explores the principles of replacing conventional Q/K dot-product attention with bandpass filtering.
Timeline (last 90 days)
The FourierQK paper (arXiv:2610.00009), titled "FourierQK: Filter Shape, Admissibility and the Leakage-Coverage Law," conducts a systematic ablation study centered on optimal filter shape and its underlying principles.
The researchers designed controlled ablation experiments on a character-level language modeling task (TinyShakespeare dataset, 6-layer GPT architecture), individually testing hypotheses for five filter attributes: DC suppression, Nyquist suppression, bandwidth, center frequency, and multi-scale coverage.
When retaining the DC and Nyquist components, the validation loss degrades to approximately 2.0 nats, equivalent to the effect of randomizing the phase, indicating that information at these two ends is actively harmful.
Under the single-scale filter configuration, the optimal bandwidth is approximately σ ≈ 2 frequency bins, and the center should fall at the paragraph scale corresponding to approximately 70 tokens.
Under the optimal configuration, a gain of Δ = +1.15 nats is achieved compared to the baseline dot-product attention (BASE-DOT).
Filters satisfying the admissibility condition (zero-mean, second-order Mexican hat DOG, m=2 difference of Gaussians) outperform ordinary Gaussian filters that do not satisfy the admissibility condition at the same scale.
The leakage-coverage law states that two-sided FFT leakage increases monotonically with spectral coverage; narrow-band filters (gap > +4) have a clean spectrum with almost no leakage, while wide-band filters (gap < +2) exhibit severe leakage.
A causal time-domain Morlet filter at the character scale cannot surpass the BASE-DOT baseline, because K=128 filter taps can only cover 50% of the T=256 context length.
FourierQK is truly effective in bidirectional attention scenarios (encoder-style architectures such as BERT), while autoregressive decoder-style tasks require a causal spectral variant.
The FourierQK paper code has been open-sourced on GitHub in the energy-gated-attention repository.
All Facts (10)
The FourierQK paper (arXiv:2610.00009), titled "FourierQK: Filter Shape, Admissibility and the Leakage-Coverage Law," conducts a systematic ablation study centered on optimal filter shape and its underlying principles.
50%UnverifiedThe researchers designed controlled ablation experiments on a character-level language modeling task (TinyShakespeare dataset, 6-layer GPT architecture), individually testing hypotheses for five filter attributes: DC suppression, Nyquist suppression, bandwidth, center frequency, and multi-scale coverage.
50%UnverifiedWhen retaining the DC and Nyquist components, the validation loss degrades to approximately 2.0 nats, equivalent to the effect of randomizing the phase, indicating that information at these two ends is actively harmful.
50%UnverifiedUnder the single-scale filter configuration, the optimal bandwidth is approximately σ ≈ 2 frequency bins, and the center should fall at the paragraph scale corresponding to approximately 70 tokens.
50%UnverifiedUnder the optimal configuration, a gain of Δ = +1.15 nats is achieved compared to the baseline dot-product attention (BASE-DOT).
50%UnverifiedFilters satisfying the admissibility condition (zero-mean, second-order Mexican hat DOG, m=2 difference of Gaussians) outperform ordinary Gaussian filters that do not satisfy the admissibility condition at the same scale.
50%UnverifiedThe leakage-coverage law states that two-sided FFT leakage increases monotonically with spectral coverage; narrow-band filters (gap > +4) have a clean spectrum with almost no leakage, while wide-band filters (gap < +2) exhibit severe leakage.
50%UnverifiedA causal time-domain Morlet filter at the character scale cannot surpass the BASE-DOT baseline, because K=128 filter taps can only cover 50% of the T=256 context length.
50%UnverifiedFourierQK is truly effective in bidirectional attention scenarios (encoder-style architectures such as BERT), while autoregressive decoder-style tasks require a causal spectral variant.
50%UnverifiedThe FourierQK paper code has been open-sourced on GitHub in the energy-gated-attention repository.
50%