FourierQK: Filter Shape and the Leakage-Coverage Law in Frequency-Domain Attention

From Dot-Product Attention to Frequency-Domain Attention

The standard Transformer attention mechanism relies on dot products between Query and Key vectors to measure token-level correlations. A recently proposed approach called "frequency-collapse attention" attempts to replace this classic design with a fundamentally different idea: instead of the traditional Q/K dot product, it computes inner products over band-pass filtered representations at learned frequencies. This modification yields measurable gains over standard dot-product attention.

A natural question follows — if band-pass filtering genuinely helps, what filter shape is optimal, and why? The arXiv paper titled FourierQK: Filter Shape, Admissibility and the Leakage-Coverage Law (arXiv:2610.00009) is a systematic ablation study centered on exactly this question.

Frequency-collapse attention works by projecting Query and Key vectors into the frequency domain, then using a band-pass filter to select information from a specific frequency range before computing the inner product of the filtered representations as the attention score. Unlike standard Q/K dot products, this mechanism is selective over the frequency content of the input sequence — it attends only to oscillatory patterns within a target frequency band while suppressing everything outside it. A band-pass filter, in signal processing terms, allows signals within a certain frequency range to pass while attenuating those outside it. The Discrete Fourier Transform (DFT/FFT) decomposes a time-domain sequence into a superposition of sinusoids at different frequencies: slow, long-range variations correspond to low-frequency components, rapid short-range fluctuations correspond to high frequencies, and paragraph-level periodic structure falls in the mid-frequency band. Bringing this tool into attention is essentially asking the model to explicitly represent context dependencies at a specific scale.

Five Hypotheses and Controlled Ablation Experiments

The researchers designed controlled ablation experiments on a character-level language modeling task (TinyShakespeare dataset, 6-layer GPT architecture), testing hypotheses about five distinct filter properties:

• DC suppression (the zero-frequency / mean component)
• Nyquist suppression (the highest-frequency component)
• Bandwidth
• Centre frequency
• Multi-scale coverage

The value of this decomposition lies in moving beyond the blanket statement that "band-pass filtering helps" — it attempts to isolate the independent contribution of each spectral property to final performance, thereby answering the mechanistic question of why it works.

A Counterintuitive Result for DC and Nyquist Components

The first key finding is counterintuitive: DC and Nyquist components are not merely unhelpful — they are actively harmful. When these components are retained, validation loss degrades to approximately 2.0 nats, equivalent to the effect of randomizing the phase.

This means the gains from band-pass attention do not come from any arbitrary low-dimensional spectral summary, but specifically from oscillatory band-pass structure itself. In other words, what the attention mechanism actually needs are mid-frequency oscillatory patterns; the DC (constant offset) and Nyquist (maximum-frequency jitter) extremes pollute the representation.

Optimal Bandwidth and Paragraph Scale

Under single-scale filter configurations, the study finds that the optimal bandwidth is approximately σ ≈ 2 frequency bins, with the centre falling at "paragraph scale" — corresponding to a span of roughly 70 tokens. Under this configuration, a gain of Δ = +1.15 nats is achieved over the dot-product baseline (BASE-DOT).

This result reveals an interesting correspondence with linguistic structure: the optimal frequency is not arbitrary, but aligns with natural semantic units in text (paragraph-level long-range dependencies). In a sense, frequency-domain attention is explicitly modeling periodic linguistic structure at a specific scale.

Admissibility and the Leakage-Coverage Law

The most theoretically rich part of the paper introduces the concept of admissibility from wavelet analysis.

The Advantage of Zero-Mean Filters

Admissible filters (zero-mean, second-order Difference of Gaussians / Mexican Hat, i.e., DOG with m=2) outperform non-admissible ordinary Gaussian filters at the same scale. The zero-mean property naturally suppresses the harmful DC component while also providing partial protection against bilateral FFT leakage.

Leakage Grows Monotonically with Coverage

The researchers distill a "leakage-coverage law": bilateral FFT leakage grows monotonically with spectral coverage.

• Narrow-band filters (gap > +4): clean spectrum, almost no leakage
• Wide-band filters (gap < +2): severe leakage

This provides clear engineering guidance for filter design — to achieve a clean spectral response, bandwidth must be controlled and overly wide frequency coverage must be avoided.

Admissibility is a core constraint in wavelet analysis, requiring that the mother wavelet have zero mean (i.e., zero response at zero frequency). Intuitively, an admissible filter must be "oscillatory" rather than "offset" — its positive and negative areas cancel out. This condition guarantees invertibility of the wavelet transform and naturally prevents DC components from entering the representation. The Mexican Hat wavelet is the second derivative of a Gaussian function, also known as DOG (Difference of Gaussians, m=2). Its shape resembles an inverted hat — a central positive peak flanked by negative wings — with strictly zero mean, satisfying admissibility. By contrast, an ordinary Gaussian filter has a positive mean, does not satisfy admissibility, and therefore cannot fully suppress the DC component. Bilateral FFT leakage refers to the phenomenon in finite-length sequences where wide-band filters cause energy from different frequencies to "bleed" into one another, creating spectral aliasing that contaminates the target frequency band's representation.

Causality Constraints and the Scope of Bidirectional Attention

The fifth finding touches on the fundamental applicability of the approach. The researchers tested causal time-domain Morlet filters at character scale, and found that they cannot surpass the BASE-DOT baseline. The reason is that K=128 filter taps can cover only 50% of the T=256 context length, and the limited receptive field under the causal constraint restricts representational capacity.

This finding clarifies FourierQK's fundamental positioning: it is most effective in bidirectional attention settings — encoder-style architectures like BERT, where both training and inference have access to the full sequence context. For autoregressive generation tasks (decoder-style, like GPT), a causal spectral variant is needed — which is precisely what the companion paper MorletQK explores, extending experiments to word-level scale.

Causal attention is the core constraint in autoregressive language models (e.g., the GPT family): when generating the representation at position t, the model can only access tokens at positions 1 through t, and cannot "see" the future. This contrasts with bidirectional attention in encoder models like BERT, which can simultaneously leverage both left and right context and therefore naturally has a receptive field covering the full sequence. The Morlet wavelet is a complex-valued wavelet formed by a sinusoidally modulated Gaussian window, capable of locally capturing specific frequency components in the time domain. A causal Morlet filter must be truncated to retain only past-facing taps, which directly shrinks the effective receptive field. The paper notes that K=128 taps can cover only 50% of a T=256 context, meaning each position can look back at most 128 tokens — long-range dependencies beyond that range are completely lost, which is the fundamental reason this approach cannot surpass the dot-product baseline in autoregressive settings.

Implications for Frequency-Domain Attention Research

Taken together, these five findings give a clear picture of the effective boundaries and design principles for frequency-domain attention: oscillatory band-pass structure is essential; the optimal scale corresponds to semantic units; admissible filters are superior; bandwidth control determines leakage; and causal settings require dedicated variants.

For researchers, the value of this work lies not only in the specific ablation numbers, but in transforming "filter design" from black-box hyperparameter tuning into a theory-guided engineering problem. As the Transformer architecture continues to evolve, systematically incorporating classical signal processing tools — wavelets, band-pass filtering, admissibility conditions — into attention mechanisms may represent a promising path toward improving long-range modeling capability. The paper's code is open-sourced on GitHub (the energy-gated-attention repository).