ACCT: A New Method for Cross-LLM KV Cache Transfer

The Context Computation Dilemma in Heterogeneous Deployment

As LLM systems evolve from single-model calls to heterogeneous deployments, an easily overlooked cost is accumulating: the same context is repeatedly processed by different models. Under routing mechanisms, each time the target model switches, the new model must prefill the entire context from scratch, repeatedly consuming substantial compute.

ACCT (Asymmetric Cross-LLM Context Transfer) targets exactly this pain point. Its goal scenario is clear and restrained: KV cache transfer between models of different parameter scales within the same model family. This "same-family, different-scale" setup is not a compromise but captures the most common model combination in real deployments—using large models for complex reasoning and small models for high-frequency requests.

Prefill is the first stage of LLM inference. The model must process the entire input context at once, computing and caching the Key and Value vectors of each layer (i.e., the KV cache) in preparation for subsequent token-by-token generation. Due to the autoregressive nature of attention, the compute in the prefill stage scales quadratically with context length, and overhead rises sharply for long contexts. In heterogeneous deployment, each time a request is routed to a new model, the new model must re-execute the full prefill, even if the context content is identical. ACCT's core idea is precisely to reuse the KV cache already computed by the source model, avoiding repeated prefill by the target model and thus saving this compute.

How Three Lightweight Components Work Together

ACCT's architecture consists of three components, each solving one link in the transfer chain.

Cartographer: Locating Key Context

Cartographer is responsible for identifying, for each target layer and KV head, the context tokens relevant to the query. The core challenge of cross-model transfer is that the attention distributions of the source and target models are not aligned, and directly moving the KV cache introduces noise. Cartographer's role is to re-filter from the target model's perspective and find the portion of context that truly affects the output.

KV cache (Key-Value Cache) is a key optimization in the Transformer decoding process: during autoregressive generation, the model needs to repeatedly use the Key and Value vectors of all previous tokens to compute attention. To avoid repeated computation, inference systems cache these vectors. The cache size is proportional to the number of layers, KV heads, head dimension, context length, and batch size, and can occupy tens of GB of GPU memory in long-context scenarios. ACCT transfers exactly this cache, but since the attention distributions of the source and target models differ, Cartographer is needed to re-filter and retain only the KV entries corresponding to tokens that truly matter to the target model.

Allocator: Distributing Sparse Budget Under a Total Budget

Allocator decides how many context tokens to retain for each target layer and KV head under a shared total KV budget constraint. This is a resource allocation problem—different layers and heads have different degrees of dependence on context, and equal allocation wastes budget; Allocator makes sparsification controllable.

Sparse KV refers to retaining only the key-value pairs of some tokens in the KV cache, rather than the entire context. Since attention mechanisms typically assign higher weights to a few key tokens, sparsification can significantly reduce GPU memory and compute overhead while maintaining output quality. In ACCT, Allocator decides how many tokens to retain per layer and per KV head under the total budget constraint, forming a non-uniform sparsity pattern; Compiler then converts the filtered source KV into a sparse format that the target model can directly consume. In this way, the target model can perform inference directly based on the sparse KV without re-prefilling, decoupling context computation from downstream inference.

Compiler: Completing KV Format Conversion

Compiler converts the filtered source KV into sparse KV that the target model can directly consume. After this step, the target model can perform inference based on the transferred KV without reprocessing the original context. Context computation is thus decoupled from downstream inference.

Validation: Dual Benefits in Performance and Speed

The paper evaluates ACCT in two complementary scenarios: five general capability benchmarks and six professional tasks that are highly context-dependent.

The most noteworthy conclusion on performance is: even with the KV cache reduced by 75%, ACCT can still match or even exceed the target model's original inference performance. This means sparsification does not come at the expense of quality.

The speed data is equally impressive: at a context length of about 19K tokens, ACCT's KV conversion achieves a 6.5x to 21.2x speedup compared to the target model's context prefill. The longer the context, the greater the absolute prefill overhead, and the more obvious the practical value of this speedup ratio.

Boundaries and Imagination Space of Cross-Model Reuse

ACCT's value is not just in saving compute. It turns "context computation" into an asset that can flow between models—one prefill, multiple reuses. For systems that need dynamic routing among multiple models, this is equivalent to removing repetitive work from the inference path.

What needs to be viewed rationally is the boundary set by the paper: transfer holds only within the same model family. Cross-family transfer faces differences in tokenizer, layer structure, and attention mechanisms, and ACCT does not claim to solve this problem. But for the high-frequency scenario of same-family heterogeneous deployment, ACCT has provided a reproducible feasibility proof for completing cross-model context computation transfer under a limited KV budget.

One-line Summary

ACCT enables KV cache transfer between same-family models of different scales, eliminating repeated prefill and achieving 6.5-21.2x speedup.

Paragraph Summary

ACCT (Asymmetric Cross-LLM Context Transfer) addresses the compute waste in heterogeneous deployment where the same context is repeatedly prefilled by different models, proposing KV cache transfer between models of different parameter scales within the same model family. Its architecture consists of three lightweight components: Cartographer filters context tokens relevant to the query from the target model's perspective, Allocator allocates sparse budgets for each layer and KV head under a total KV budget, and Compiler converts the filtered source KV into sparse KV that the target model can directly consume. Experiments show that even with the KV cache reduced by 75%, ACCT can still match or even exceed the target model's original performance on five general benchmarks and six context-dependent tasks; at a context length of about 19K tokens, KV conversion achieves a 6.5x to 21.2x speedup compared to the target model's prefill. This work decouples context computation from downstream inference, making context an asset that can flow between models, but transfer capability is limited to same-family models, and cross-family scenarios are not yet covered.