Unverified50% confidenceSolutionExact time
Mixed-precision quantization retains higher bit precision for key weights in attention layers while applying more aggressive compression to feed-forward layers with higher redundancy
1
Sources
50%
Confidence
Long-term
Relevance
10/11/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedStandard attention's computation grows quadratically with sequence length, whereas the low-rank bottleneck decomposes the transformation into two steps: first compressing to r dimensions and then expanding.77% similarVerifiedThe root cause of attention dilution lies in Softmax normalization: the more tokens in a sequence, the effective attention weight allocated to each position decreases approximately inversely proportional to the sequence length.75% similarUnverifiedIn single-hop needle-in-a-haystack tasks, critical KV pairs are naturally prominent in attention scores, making it nearly impossible for compression methods to drop them, making these tasks too 'friendly' to compression methods.75% similarUnverifiedThe attention score of key evidence has an upper bound, while the maximum score among all effective distractors grows as the number of distractors increases.74% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/996997API
curl https://kongchang.com/api/v1/knowledge/claims/996997MCP
get_claim(id=996997)