Unverified50% confidenceFactExact time
Recent research ShortGPT found that hidden states between intermediate layers of Transformer models often exhibit high cosine similarity, and these redundant layers can be safely removed with little impact on performance.
1
Sources
50%
Confidence
Long-term
Relevance
9/11/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedResearch (Dettmers et al., 2022) has shown that specific hidden dimensions in Transformer models consistently exhibit abnormally large activation values across all tokens and layers, with the phenomenon becoming more pronounced in models exceeding 6.7B parameters.75% similarUnverifiedTransformer模型存在位置偏差，对中间位置信息的检索能力尤其薄弱，即lost in the middle问题72% similarUnverifiedTransformer models exhibit a "Lost in the Middle" phenomenon, where attention to content located in the middle of the context and far from the current generation position systematically decreases.72% similarUnverifiedIn Transformer models, the first and last few layers are more sensitive to quantization, the middle layers have higher redundancy, and attention layers and feed-forward network layers differ in their tolerance to quantization.72% similarUnverifiedThe current Transformer architecture has a fundamental limitation in handling implicit constraints scattered across ultra-long contexts, leading to the introduction of hidden bugs when modifying code.72% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/893768API
curl https://kongchang.com/api/v1/knowledge/claims/893768MCP
get_claim(id=893768)