Unverified50% confidenceFactExact time
When a Transformer outputs tokens through sampling, it permanently loses information about which possibilities the model considered and their respective confidence levels (a lossy process).
1
Sources
50%
Confidence
Long-term
Relevance
9/16/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedIn 2024, researchers proposed designing a model's hidden states as small models that can be updated via gradient descent at test time, in order to address the excessively high computational complexity of Transformers over long contexts while preserving the advantage of RNNs' linear complexity.73% similarUnverified标准Transformer在处理需要精确计数、栈操作或复杂状态依赖的任务时表现不稳定，与其注意力机制的软寻址特性有关73% similarUnverifiedTransformer模型存在位置偏差，对中间位置信息的检索能力尤其薄弱，即lost in the middle问题72% similarUnverifiedTransformer models exhibit a "Lost in the Middle" phenomenon, where attention to content located in the middle of the context and far from the current generation position systematically decreases.71% similarUnverifiedResearch (Dettmers et al., 2022) has shown that specific hidden dimensions in Transformer models consistently exhibit abnormally large activation values across all tokens and layers, with the phenomenon becoming more pronounced in models exceeding 6.7B parameters.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/925327API
curl https://kongchang.com/api/v1/knowledge/claims/925327MCP
get_claim(id=925327)