ALHR
ALHR (Attention with Learned Hierarchical Routing) is an efficient attention mechanism based on a static binary tree. It uses a learnable hierarchical routing function to reduce the number of keys participating in attention computation, lowering complexity from O(N²) to O(N log N). This approach reduces GPU memory usage while maintaining high recall accuracy on long-context tasks.
Timeline (last 90 days)
ALHR (Adaptive Learnable Hierarchical Routing) is a system based on a static binary tree that reduces the number of Keys participating in attention computation through a learnable routing function, bringing attention complexity down to O(N log N).
According to the authors, ALHR maintains approximately 97% accuracy on long-context tasks.
The depth of the binary tree is at the log N level, and a Query traverses log N layers along the path rather than scanning all N Keys, reducing overall complexity from N×N to N×log N.
The learnable routing function differs from fixed sparse attention (such as local windows and fixed-stride jumps) by allowing the model to learn during training where it should attend.
All information about ALHR comes from a single source—the developer's own description on Reddit—and lacks a paper, code repository, and fully reproducible experimental results.
All Facts (5)
ALHR (Adaptive Learnable Hierarchical Routing) is a system based on a static binary tree that reduces the number of Keys participating in attention computation through a learnable routing function, bringing attention complexity down to O(N log N).
50%UnverifiedAccording to the authors, ALHR maintains approximately 97% accuracy on long-context tasks.
50%UnverifiedThe depth of the binary tree is at the log N level, and a Query traverses log N layers along the path rather than scanning all N Keys, reducing overall complexity from N×N to N×log N.
50%UnverifiedThe learnable routing function differs from fixed sparse attention (such as local windows and fixed-stride jumps) by allowing the model to learn during training where it should attend.
50%UnverifiedAll information about ALHR comes from a single source—the developer's own description on Reddit—and lacks a paper, code repository, and fully reproducible experimental results.
50%