Unverified50% confidenceFactExact time
In 2024, researchers proposed designing a model's hidden states as small models that can be updated via gradient descent at test time, in order to address the excessively high computational complexity of Transformers over long contexts while preserving the advantage of RNNs' linear complexity.
1
Sources
50%
Confidence
Long-term
Relevance
9/11/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedResearch (Dettmers et al., 2022) has shown that specific hidden dimensions in Transformer models consistently exhibit abnormally large activation values across all tokens and layers, with the phenomenon becoming more pronounced in models exceeding 6.7B parameters.75% similarUnverifiedRecent research ShortGPT found that hidden states between intermediate layers of Transformer models often exhibit high cosine similarity, and these redundant layers can be safely removed with little impact on performance.72% similarVerifiedTransformer overcomes the long-range dependency attenuation caused by sequential computation in RNN/LSTM architectures, while natively supporting parallel computation.68% similarUnverifiedTransformer models exhibit a "Lost in the Middle" phenomenon, where attention to content located in the middle of the context and far from the current generation position systematically decreases.68% similarUnverified斯坦福大学2023年的研究将Transformer在处理超长上下文时对中间位置信息遗忘率显著高于首尾位置的现象命名为'Lost in the Middle'问题67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/907374API
curl https://kongchang.com/api/v1/knowledge/claims/907374MCP
get_claim(id=907374)