RidgeFill: Training-Free Later-Layer Cache Filling, Up to 1.6x Prefill Speedup

Title

RidgeFill: Training-Free Later-Layer Cache Filling, Up to 1.6x Prefill Speedup

Meta Description

RidgeFill proposes a training-free later-layer cache filling approach: using closed-form ridge regression to fit inter-layer mappings, skipping prefill computation for most later layers of the model, achieving up to 1.6x first-token speedup in long-context scenarios while honestly presenting the accuracy costs on RULER and LongBench.

Article Content

The bottleneck of large model inference is shifting from decoding to prefill. When context length extends to tens of thousands or even hundreds of thousands of tokens, the cost of fully computing the prompt layer by layer makes first-token latency unacceptable. RidgeFill offers a rare path: no training, no token dropping—using an inter-layer mapping fit by a single closed-form ridge regression to skip computation for most later layers of the model, directly predicting the inputs to these layers and writing them into the KV cache.

Origin of the Idea: Moving YOCO-Style Inter-Layer Projection onto Untrained Models

RidgeFill's design references the architectural ideas of YOCO and DeepSeek^Ad-V4.1-Flash—in these two types of models, a projection maps shallow-layer states to deep-layer inputs, which is baked into the model during training. RidgeFill addresses a different problem: how to retrofit the same mechanism onto models that were never trained for it.

Its approach: shallow layers are computed exactly as usual up to a certain split layer; starting from the split layer's state, a mapping is fit for each subsequent layer to predict that layer's input, and then that layer's own projection writes the cache as usual. The mapping itself is just a ridge regression, fit in closed form from a single forward pass over calibration text, fit only once. The layers' original projections remain untouched, with no fine-tuning.

To control error, several positions retain full computation: selected rows, the last few thousand tokens, and every step of the decoding phase. The original text emphasizes "nothing was trained, and no tokens were dropped," which is the key distinction from token-pruning acceleration approaches.

YOCO (You Only Cache Once) is a decoder-decoder architecture: a shallow encoder first processes the entire context, retaining only one global KV cache, and then a deep decoder reads that cache via cross-attention, avoiding repeated per-layer KV storage. Its key design is a trainable projection between shallow-layer outputs and deep-layer inputs, so the deep layers don't need to re-encode the context. RidgeFill borrows precisely this "inter-layer projection" structure, but YOCO's projection is jointly optimized with the model during pretraining, whereas RidgeFill faces an already-frozen ordinary model and can only fit an approximate mapping after the fact using statistical methods.

Closed-form ridge regression refers to ridge regression whose solution has an analytical expression and requires no iterative optimization: given input matrix X and target Y, the weights W = (XᵀX + λI)⁻¹XᵀY. Here λ controls regularization strength, used to stabilize the solution when samples are limited. RidgeFill collects split-layer states and the true inputs of subsequent layers as (X, Y) pairs on calibration text, and directly computes W after a single forward pass, so the "fitting" cost is equivalent to just one inference. The closed-form solution also means the mapping is linear + regularized, with limited expressiveness—which is exactly why errors need to be backstopped by retaining partial exact computation.

The Real Difficulty Is Measurement, Not Fitting

This work devotes substantial space to methodology, for a direct reason: loss averaged over all tokens masks RidgeFill's errors. Errors in the filled regions are diluted by the many easy tokens, making the loss appear small while actual capability may have degraded.

The authors therefore designed three complementary detection lines: needle retrieval within filled regions, loss on tokens that depend on long-distance context, and a splice test that replaces filled keys or values with exact values and compares. The conclusion is that the components under pressure differ across models—on Llama-3.1-8B the error mainly comes from keys, half of which manifests in scale; on Qwen3.5-9B it falls on values and recurrent states. This component-level attribution is more engineering-valuable than a vague speedup ratio, because it indicates where protection is needed and where relaxation is possible.

Splice test is a component-level attribution method: replace the key or value that RidgeFill predicted and wrote into the KV cache with the true tensor computed exactly at the same position, then observe whether downstream metrics recover. If accuracy rebounds noticeably after replacing keys but not values, the error mainly comes from the key side; vice versa otherwise. It amounts to controlled ablation on cache contents, decomposing vague end-to-end loss into specific components to guide protection strategies—for example, retaining exact computation for components under pressure while continuing the prediction path for relaxed components.

Measured Performance: Speedup and Accuracy Cost Coexist

Running Qwen3.8-27B on an A100 using a llama.cpp branch, with the mapping held in VRAM, first-token time in the 16K to 120K token range is 1.25 to 1.31 times that of exact computation. On accuracy, RULER drops 1.2 and 1.4 points, and LongBench v2 drops 1.4 points; the 95% confidence interval includes zero, but a real loss of a few points cannot be ruled out.

Running Qwen3.8-Flash-Next on vLLM, the 9K to 32K token range achieves 1.50 to 1.60 times speedup relative to exact computation. RULER's needle task is 99.8 vs 100.0, and new-word perplexity is 2.65% higher, with an interval of -0.02% to +4.93%.

On three 8-9B reference models, with the same split and tail settings as identity fill, the fitted mapping recovers an additional 49.1 and 49.6 RULER points in the filled region on two dense models (Llama's loss is worse under that split), and long-document loss is within ±0.03 nats of the published SwiftKV checkpoint.

Engineering Perspective: Where It Fits

RidgeFill's value is not in setting speedup records, but in downgrading an architectural change that originally required pretraining into a one-time calibration fit. The cost is that accuracy is not lossless, and the benefit becomes more pronounced as context grows longer—the speedup ratio is stable across the 16K to 120K range, indicating it targets precisely the most expensive cost of long-context prefill.

Several practical constraints must be acknowledged. First, the mapping must be fit separately for each model, and the choice of calibration text affects results—a cost the original text does not elaborate on. Second, the mapping must reside permanently in VRAM, requiring trade-offs in memory-constrained scenarios. Third, although the accuracy loss on some benchmarks has confidence intervals covering zero, it still needs validation on downstream tasks, especially applications sensitive to long-distance dependencies.

A more prudent usage is to treat it as one part of a tiered strategy: enable filling for latency-sensitive, extremely long-context requests, and fall back to exact computation for accuracy-sensitive tasks. The original text retaining full computation for the last few thousand tokens and every decoding step is itself an embodiment of this hybrid approach. As for the finding that key-side error concentrates in scale, it also suggests that subsequent work could add dedicated scale correction to the mapping, recovering accuracy at lower cost.

First-token latency (TTFT) is the core metric of the prefill phase: after a request arrives, the model must process all input tokens layer by layer and write the KV cache before generating the first output token. The longer the context, the larger the prefill FLOPs and KV writes, and TTFT grows linearly or even superlinearly, while the decoding phase processes only one new token per step. RidgeFill skips prefill computation for some layers, directly compressing exactly TTFT, so its benefit becomes more pronounced as context grows longer and is limited for short requests.

One-line Summary

RidgeFill uses a single closed-form ridge regression to fit inter-layer mappings, enabling training-free skipping of some layers to accelerate long-context prefill.

Paragraph Summary

Targeting the long-context prefill bottleneck, RidgeFill proposes a training-free acceleration approach: shallow layers are computed exactly up to a split layer, and subsequent layers use a mapping fit by closed-form ridge regression from a single calibration forward pass to predict inputs and write to the KV cache, while retaining full computation for selected rows, the last few thousand tokens, and decoding steps, dropping no tokens. The authors emphasize that average loss masks errors and design three detection lines—needle retrieval, long-distance token loss, and splice test—finding that the components under pressure differ across models. Measured results show Qwen3.8-27B on llama.cpp achieves 1.25-1.31x speedup at 16K-120K, and Qwen3.8-Flash-Next on vLLM achieves 1.50-1.60x at 9K-32K, with accuracy drops of about 1-1.4 points. Its value lies in downgrading an architectural change requiring pretraining into a one-time calibration fit, suitable as one part of a tiered acceleration strategy.