Learned Selector
A sparse attention mechanism that uses a lightweight neural network, such as an MLP or low-rank projection, to dynamically evaluate and select the most relevant set of key-value pairs for each query to participate in attention computation, thereby reducing computational complexity. Its selection behavior is driven by training data, working effectively in-distribution but potentially failing out-of-distribution.
Timeline (last 90 days)
A learned selector is usually a lightweight network (such as a small MLP or low-rank projection) that takes the current query vector and the key vectors of candidate tokens, quickly scores them, and selects the Top-k most relevant tokens to participate in the actual attention computation.
For tasks dominated by local pattern matching (such as code completion and log analysis), sliding windows are extremely cost-effective; for cross-document reasoning or long-chain causal analysis, learned selectors or hybrid strategies are more reliable, but require thorough validation in the target domain.
When out-of-distribution (OOD) failure occurs, learned selectors may persistently ignore key evidence, resulting in answers that appear fluent but omit key facts, increasing the difficulty of troubleshooting.
All Facts (3)
A learned selector is usually a lightweight network (such as a small MLP or low-rank projection) that takes the current query vector and the key vectors of candidate tokens, quickly scores them, and selects the Top-k most relevant tokens to participate in the actual attention computation.
50%UnverifiedFor tasks dominated by local pattern matching (such as code completion and log analysis), sliding windows are extremely cost-effective; for cross-document reasoning or long-chain causal analysis, learned selectors or hybrid strategies are more reliable, but require thorough validation in the target domain.
50%UnverifiedWhen out-of-distribution (OOD) failure occurs, learned selectors may persistently ignore key evidence, resulting in answers that appear fluent but omit key facts, increasing the difficulty of troubleshooting.
50%