ZHA

ZHA currently lacks efficiency data, making it difficult to reach a conclusion on how "truly sub-quadratic" it is.

ZHA uses distillation to give an already frozen LLM sparse attention capability without the need to retrain the base model.

ZHA officially claims that it can match dense attention on reasoning tasks and on retrieval benchmarks at 256K length.

ZHA uses 4B tokens of distillation to teach the frozen base model to use sparse attention patterns.

The original report did not disclose details of how ZHA's tree structure is specifically constructed, its branching factor, or its hierarchical depth.

ZHA's evaluation covers only two dimensions, reasoning and 256K retrieval, and lacks horizontal comparison with other sparse approaches and specific data on the efficiency side.

ZHA's contribution may lie in combining tree-index sparse attention with distillation of a frozen model to form a reusable retrofit pipeline.

KL distillation uses the original full-attention model as the teacher and has the student model, with its structure replaced by ZHA, imitate the teacher's next-token distribution, with KL divergence as the loss.

ZHA (Hierarchical Attention), proposed by Zyphra, uses a recursively compressed tree-structured indexer that makes total cost grow sublinearly with sequence length, thereby achieving an overall sub-quadratic but superlinear attention complexity.

ZHA builds a recursively compressed key summary tree over the KV cache, where leaf nodes correspond to original key blocks and upper-level nodes are compressed representations of their child nodes, and queries are filtered layer by layer from the top of the tree downward.

The tree depth of ZHA can grow iteratively, and the total cost of routing and final attention is sublinear relative to sequence length, yielding an overall sub-quadratic but superlinear scaling curve.

ZHA requires only a brief KL distillation training at the scale of a few hundred billion tokens to be applied to existing pretrained models, without pretraining from scratch.

ZHA maintains performance on long-context retrieval and on evaluation tasks such as reasoning, math, and agentic coding.

ZHA is highly robust to tree geometry (branching factor and depth) and stably supports trees as deep as 6 levels.

ZHA exhibits sub-quadratic scaling in measurements and significantly outperforms competing methods beyond a 1 million context length.

ZHA's post-hoc sparsification approach may be closer to engineering reality than new architectures, and is worth the attention of teams focused on long context and inference efficiency.

ZHA has the potential to extend the effective context window to hundreds of millions or even billions of tokens, but this conclusion comes from model reasoning and experimental extrapolation and is a directional judgment.