DLite: History-Free Speculative Decoding for Agents, Boosting Throughput by 70%

Speculative Decoding has become a mainstream technique for accelerating large language model inference. Its core idea is to use a lightweight draft model to generate candidate tokens, which are then verified by the target model. However, when workloads shift to Agentic AI scenarios, long and continuously evolving contexts and frequent multi-turn interactions make traditional draft models struggle. A new paper proposes the DLite framework, which uses a "history-free" local prediction approach to achieve an acceptance length of 2-6 under agentic workloads, surpassing the current state-of-the-art method DSpark by up to 70%.

Pain Points of Speculative Decoding Under Agentic Workloads

Speculative decoding performs excellently in standard text generation tasks, but agentic workloads bring two new problems. The paper observes that existing draft models suffer an acceptance length drop of up to 80% in agentic scenarios, meaning the proportion of candidate tokens generated by the draft model that are accepted by the target model shrinks dramatically, and the acceleration effect is almost lost.

Meanwhile, to maintain draft model performance, a common practice is to retain historical KV caches. But agent contexts grow continuously, and maintaining these historical caches brings significant system overhead—increased memory usage and more complex cache management—offsetting the benefits of speculative decoding.

The paper attributes the root cause of both problems to the same factor: the growing historical context. The longer the history, the more complex the dependencies the draft model must model, and the lower the acceptance rate; and to handle this history, the system must pay additional caching costs.

Acceptance length is the core evaluation metric for speculative decoding, referring to the average number of consecutive candidate tokens generated by the draft model that are accepted by the target model in a single verification. The larger this value, the higher the proportion of draft tokens that pass "in one go," the fewer forward passes the target model needs to execute, and the more significant the speedup. Theoretically, if each draft generates γ tokens and the acceptance length is τ, the ideal speedup is approximately τ (slightly lower due to verification overhead). Therefore, when acceptance length drops from the common 3-4 to less than 1, speculative decoding is almost equivalent to failing—most draft tokens are rejected, and verification overhead becomes a burden instead. Understanding this metric is a prerequisite for judging the significance of DLite's claimed "2-6 acceptance length."

KV cache is a key mechanism in autoregressive inference to avoid redundant computation: when generating each token, the model caches the Key and Value tensors of all previous tokens across attention layers, so that attention only needs to be computed for the new token. For a draft model, accurately modeling long history usually requires maintaining a separate KV cache synchronized with the target model. In agentic scenarios, contexts can reach tens of thousands of tokens and continue to grow, so this cache's memory usage expands linearly with the number of dialogue turns; meanwhile, consistency maintenance, eviction, and migration of the cache in multi-turn interactions bring additional engineering complexity. This is precisely the technical root of the paper's claim that "historical KV cache overhead offsets acceleration gains."

DLite's Core Idea: Turning Complex Modeling into Local Prediction

DLite's solution is quite clever: since historical context is a burden, simply do not explicitly maintain it. DLite directly reuses the contextual hidden states produced by the target model during inference, and performs local speculative prediction on this basis, without requiring the draft model to maintain a complete history itself.

This design transforms "an increasingly complex modeling problem" into "a local, simple prediction problem." The draft model no longer needs to understand the entire dialogue history; it only needs to make short-range predictions based on the hidden states currently provided by the target model, and the acceptance rate is thus restored.

However, completely discarding history introduces a new problem: how to preserve causal dependencies within a block? If the draft model knows nothing about the relationships between tokens within a block, prediction quality will still suffer. To this end, DLite introduces a lightweight sequential head that restores intra-block causal dependencies at low cost. This sequential head only processes local information and does not touch historical context, so it does not reintroduce system burden.

Contextual hidden states refer to the dense vector representations output by each layer (especially the last layer) of the target model when processing an input sequence, which have already fused preceding information. Unlike raw token embeddings, hidden states encode the model's "understanding" of the current context and thus naturally carry historical information. DLite's key insight is: since the target model has already computed these representations, the draft model does not need to model history from scratch; it only needs to make short-range predictions conditioned on these hidden states. This both reuses the target model's forward computation results and avoids the draft model maintaining an independent history cache, which is the technical basis for its "history-free" design.

The sequential head is a very small-parameter module in DLite, specifically responsible for modeling causal dependencies between tokens sequentially within a speculative block. Speculative decoding usually drafts multiple tokens at once (a block); if each position is predicted independently, the later token in the block cannot use information from the earlier token, degrading prediction quality. The sequential head generates position by position within the block in an autoregressive or lightweight recurrent manner, thereby restoring this local causality, and its computation involves only a small number of tokens within the block and does not touch historical context, so it does not reintroduce long-context system overhead. This design strikes a balance between "local accuracy" and "low overhead."

Two-Stage Training: Making Heterogeneous Architecture Truly Work

DLite's architecture is heterogeneous—it relies on the target model's hidden states and is equipped with an independent sequential head. This structure cannot be effectively trained with conventional single-stage training. The paper designs a customized two-stage training strategy for this: first let each component adapt to its respective task, then jointly optimize the overall prediction capability.

This training strategy is key to DLite's practicality. Reusing the target model's hidden states means there is strong coupling between the draft model and the target model; during training, the draft part must learn to "read" the target model's internal representations, rather than independently learning the entire distribution like traditional draft models. The two-stage design reduces the training difficulty caused by this coupling.

Experimental Performance: Comprehensive Validation on General, Long-Context, and Agentic Workloads

The paper conducts experiments on three types of scenarios: general tasks, long-context tasks, and agentic workloads. Results show that DLite achieves an acceptance length of 2-6 under agentic workloads, up to 70% higher than the current state-of-the-art speculative decoding method DSpark.

The significance of this improvement lies in the fact that agentic applications are becoming an important form of LLM deployment—from code assistants to automated workflows, multi-turn interaction and long context are almost standard. If speculative decoding fails in such scenarios, the value of acceleration techniques is greatly diminished. DLite's "history-free" approach offers a new direction: rather than having the draft model chase ever-growing history, let the target model provide local context itself, and the draft model only makes the short-segment predictions it is good at.

From a broader perspective, DLite reflects a trend: inference acceleration techniques are moving from "general optimization" to "workload awareness." Redesigning the decoding process for the characteristics of specific workloads may be more effective than incremental improvements on general frameworks.

Key Takeaways

• Under agentic workloads, existing draft models suffer an acceptance length drop of up to 80%, and maintaining historical KV caches is costly; the root cause of both is growing historical context.
• DLite reuses the target model's hidden states for local prediction without explicitly maintaining history, transforming complex modeling into simple local prediction.
• A lightweight sequential head restores intra-block causal dependencies at low cost; a two-stage training strategy solves the training challenges of the heterogeneous architecture.
• Experiments cover general, long-context, and agentic workloads; acceptance length reaches 2-6, surpassing DSpark by up to 70%.

One-line Summary

DLite reconstructs the draft model with history-free local prediction, achieving acceptance length 2-6 under agentic workloads and surpassing DSpark by 70%.

Paragraph Summary

The article introduces DLite, a new speculative decoding framework for agentic workloads. Traditional speculative decoding relies on a draft model to generate candidate tokens, but in agentic scenarios, continuously growing historical context causes acceptance length to drop by up to 80%, and maintaining historical KV caches brings significant system overhead. DLite's solution is not to explicitly maintain history, but to reuse the contextual hidden states produced during target model inference for local prediction, transforming complex global modeling into simple short-range prediction; it also introduces a lightweight sequential head to restore intra-block causal dependencies and designs a two-stage training strategy to solve the training challenges of the heterogeneous architecture. Experiments cover general, long-context, and agentic workloads; DLite achieves an acceptance length of 2-6, up to 70% higher than the current state-of-the-art method DSpark. The article suggests this reflects a trend of inference acceleration moving from general optimization to workload awareness.