OAL
An outlier-dimension-aware attention linearization method that identifies dimensions contributing disproportionately to query-key dot products, applies differentiated processing to those dimensions, and constructs a dimension-aware quadratic kernel, thereby reducing approximation error while maintaining linear time complexity.
Timeline (last 90 days)
The paper "Not All Dimensions Are Equal" proposes that outlier dimensions are a source of error universally overlooked by standard linearization methods.
The study finds that a small number of dimensions in the query-key dot product contribute a disproportionate amount of energy and account for a large portion of the approximation error.
OAL (Outlier-Aware Attention Linearization) is a dimension-aware method for constructing quadratic attention approximations.
OAL starts from a second-order Taylor approximation of softmax attention and introduces a Hadamard product representation to make the contribution of each query-key dimension explicitly visible.
OAL groups dimensions by contribution magnitude, where dimensions with similar contributions share coefficients and a small number of outlier dimensions are handled individually to control parameter count.
On Qwen2.5-0.5B, OAL reduces the relative mean squared error of frozen attention outputs by approximately 8.9% compared to DiJiang.
After replacing the intermediate attention modules with OAL and jointly fine-tuning via LoRA, the WikiText-103 perplexity dropped from 31.71 to 27.50 compared to the global quadratic kernel.
OAL achieves improved accuracy on commonsense reasoning benchmarks such as PIQA, HellaSwag, and ARC-E.
All Facts (8)
The paper "Not All Dimensions Are Equal" proposes that outlier dimensions are a source of error universally overlooked by standard linearization methods.
50%UnverifiedThe study finds that a small number of dimensions in the query-key dot product contribute a disproportionate amount of energy and account for a large portion of the approximation error.
50%UnverifiedOAL (Outlier-Aware Attention Linearization) is a dimension-aware method for constructing quadratic attention approximations.
50%UnverifiedOAL starts from a second-order Taylor approximation of softmax attention and introduces a Hadamard product representation to make the contribution of each query-key dimension explicitly visible.
50%UnverifiedOAL groups dimensions by contribution magnitude, where dimensions with similar contributions share coefficients and a small number of outlier dimensions are handled individually to control parameter count.
50%UnverifiedOn Qwen2.5-0.5B, OAL reduces the relative mean squared error of frozen attention outputs by approximately 8.9% compared to DiJiang.
50%UnverifiedAfter replacing the intermediate attention modules with OAL and jointly fine-tuning via LoRA, the WikiText-103 perplexity dropped from 31.71 to 27.50 compared to the global quadratic kernel.
50%UnverifiedOAL achieves improved accuracy on commonsense reasoning benchmarks such as PIQA, HellaSwag, and ARC-E.
50%