DepthGraft
A hierarchical cross-layer KV sharing architecture that reconstructs deep-layer KV from stride-aligned shallow-to-middle source pairs, distributing non-local gradient flow across multiple source layers. It mitigates depth information dilution in Pre-LN architectures while reducing the KV cache by approximately one third.
Timeline (last 90 days)
The paper "DepthGraft: Structural Regularization through Hierarchical Cross-Layer KV Reconstruction" proposes a hierarchical KV sharing architecture named DepthGraft.
DepthGraft reconstructs deep-layer KV from stride-aligned shallow-to-middle-layer source pairs, distributing non-local gradient flow across multiple source layers.
DepthGraft achieves improved downstream task performance while reducing the KV cache by about one third.
DepthGraft preserves the early-completion property of prefill, i.e., deep-layer KV can be reconstructed from KV already computed in shallow and middle layers, thereby shortening first-token latency.
On the 16B MoE model, DepthGraft achieves a 7.0% absolute improvement on MMLU-Pro, 6.6% on CMMLU, and 6.4% on C3.
DepthGraft was experimented on dense models and MoE models ranging from 0.5B to 16B parameters, covering both GQA and MLA attention mechanisms.
DepthGraft is verified to be effective under both GQA and MLA attention mechanisms, indicating that its hierarchical reconstruction regularization effect does not depend on a specific attention implementation.
The DepthGraft project page has been open-sourced.
All Facts (7)
The paper "DepthGraft: Structural Regularization through Hierarchical Cross-Layer KV Reconstruction" proposes a hierarchical KV sharing architecture named DepthGraft.
50%UnverifiedDepthGraft reconstructs deep-layer KV from stride-aligned shallow-to-middle-layer source pairs, distributing non-local gradient flow across multiple source layers.
50%UnverifiedDepthGraft achieves improved downstream task performance while reducing the KV cache by about one third.
50%UnverifiedDepthGraft preserves the early-completion property of prefill, i.e., deep-layer KV can be reconstructed from KV already computed in shallow and middle layers, thereby shortening first-token latency.
50%UnverifiedOn the 16B MoE model, DepthGraft achieves a 7.0% absolute improvement on MMLU-Pro, 6.6% on CMMLU, and 6.4% on C3.
50%UnverifiedDepthGraft was experimented on dense models and MoE models ranging from 0.5B to 16B parameters, covering both GQA and MLA attention mechanisms.
50%UnverifiedDepthGraft is verified to be effective under both GQA and MLA attention mechanisms, indicating that its hierarchical reconstruction regularization effect does not depend on a specific attention implementation.
50%