[KongchangAI]
ModelDepthGraft 架构

DepthGraft

A hierarchical cross-layer KV sharing architecture that reconstructs deep-layer KV from stride-aligned shallow-to-middle source pairs, distributing non-local gradient flow across multiple source layers. It mitigates depth information dilution in Pre-LN architectures while reducing the KV cache by approximately one third.

Timeline (last 90 days)

Nov 3

The paper "DepthGraft: Structural Regularization through Hierarchical Cross-Layer KV Reconstruction" proposes a hierarchical KV sharing architecture named DepthGraft.

Unverified50%
Nov 3

DepthGraft reconstructs deep-layer KV from stride-aligned shallow-to-middle-layer source pairs, distributing non-local gradient flow across multiple source layers.

Unverified50%
Nov 3

DepthGraft achieves improved downstream task performance while reducing the KV cache by about one third.

Unverified50%
Nov 3

DepthGraft preserves the early-completion property of prefill, i.e., deep-layer KV can be reconstructed from KV already computed in shallow and middle layers, thereby shortening first-token latency.

Unverified50%
Nov 3

On the 16B MoE model, DepthGraft achieves a 7.0% absolute improvement on MMLU-Pro, 6.6% on CMMLU, and 6.4% on C3.

Unverified50%
Nov 3

DepthGraft was experimented on dense models and MoE models ranging from 0.5B to 16B parameters, covering both GQA and MLA attention mechanisms.

Unverified50%
Nov 3

DepthGraft is verified to be effective under both GQA and MLA attention mechanisms, indicating that its hierarchical reconstruction regularization effect does not depend on a specific attention implementation.

Unverified50%
Nov 3

The DepthGraft project page has been open-sourced.

Expired50%

All Facts (7)

Source Articles