Unverified50% confidenceFactExact time
The softmax renormalization in attention operations couples deletion decisions nonlinearly: deleting one entry changes the attention weights of all remaining entries, making it difficult for independent evaluation methods to make globally optimal deletion decisions.
1
Sources
50%
Confidence
Long-term
Relevance
10/20/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedNon-causal attention (bidirectional attention) removes the causal mask, allowing every position in the sequence to attend to the full context, making it suitable for discriminative tasks such as classification, ranking, and scoring.72% similarUnverified将复杂指令拆解为多个专注的子模块有助于减少LLM在处理长指令时的注意力稀释问题71% similarUnverifiedTreating sparse strategies as tunable parameters rather than a fixed architecture, dynamically switching attention modes according to task type, or retaining full attention in key layers while using sparsification in shallow layers, may be a compromise path that balances cost and quality.71% similarUnverifiedKV-Surgeon's quadratic surrogate model approximates the effect of a deletion set on the attention output as a quadratic-form function, so that under the greedy framework only a closed-form comparison on the quadratic form is needed at each step to select the single currently optimal deletion, thereby avoiding combinatorial explosion.70% similarUnverifiedIn the standard Transformer architecture, the Softmax operation is responsible for normalizing attention scores into a probability distribution, determining the degree to which each token attends to other tokens70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1073859API
curl https://kongchang.com/api/v1/knowledge/claims/1073859MCP
get_claim(id=1073859)