Latent Sparse Attention
A sparse attention method that hierarchically clusters keys in a learned latent space. A query evaluates a small number of cluster nodes top-down along the hierarchy tree to locate a key subset, reducing indexing cost to the order of O(log n) and supporting efficient prefill, decoding and training under very long contexts.
Timeline (last 90 days)
Latent Sparse Attention (LSA) is an efficient attention method whose core is to hierarchically cluster keys in a learned latent space, with queries scoring the cluster hierarchy rather than scoring each key individually.
LSA maps keys into a latent space and groups them layer by layer in a tree structure; a query starts at the root node, scores only a small number of candidate clusters at each layer and selects several branches to descend, finally reaching the subset of keys corresponding to leaf nodes, reducing the number of scoring operations from O(n) to on the order of O(log n).
All Facts (2)
Latent Sparse Attention (LSA) is an efficient attention method whose core is to hierarchically cluster keys in a learned latent space, with queries scoring the cluster hierarchy rather than scoring each key individually.
50%UnverifiedLSA maps keys into a latent space and groups them layer by layer in a tree structure; a query starts at the root node, scores only a small number of candidate clusters at each layer and selects several branches to descend, finally reaching the subset of keys corresponding to leaf nodes, reducing the number of scoring operations from O(n) to on the order of O(log n).
50%