Unverified50% confidenceFactExact time
Research (Dettmers et al., 2022) has shown that specific hidden dimensions in Transformer models consistently exhibit abnormally large activation values across all tokens and layers, with the phenomenon becoming more pronounced in models exceeding 6.7B parameters.
1
Sources
50%
Confidence
Long-term
Relevance
8/7/2026
First Seen
Sources
图解量化：大语言模型压缩核心技术详解
Related Claims
UnverifiedIn Transformer models, the first and last few layers are more sensitive to quantization, the middle layers have higher redundancy, and attention layers and feed-forward network layers differ in their tolerance to quantization.75% similarUnverifiedThe KV Cache of Transformers grows quadratically with sequence length, leading to non-linear GPU memory consumption during long-context inference.70% similarUnverifiedTransformer models exhibit a "Lost in the Middle" phenomenon, where attention to content located in the middle of the context and far from the current generation position systematically decreases.70% similarUnverifiedThe current Transformer architecture has a fundamental limitation in handling implicit constraints scattered across ultra-long contexts, leading to the introduction of hidden bugs when modifying code.68% similarUnverifiedTransformer模型存在位置偏差，对中间位置信息的检索能力尤其薄弱，即lost in the middle问题67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/705555API
curl https://kongchang.com/api/v1/knowledge/claims/705555MCP
get_claim(id=705555)