ReaDy: Accelerating Speculative Decoding for Hybrid Linear Attention Models Using Recurrent States

Speculative decoding accelerates autoregressive generation by drafting multiple tokens at each step, but existing feature-based drafters rely on the target model's hidden features, whose contextual emphasis may not match the needs of subsequent draft positions. A recent work called ReaDy proposes that the compact recurrent states maintained by hybrid linear attention (HLA) models can serve precisely as a complementary context source during the drafting stage.

This is the first speculative decoding method to leverage HLA recurrent states for position-specific context retrieval. The core idea is not complex—each draft position uses a learnable query to retrieve supplementary context directly from the recurrent states already committed by the target model.

Why Feature-Based Drafters Fall Short

In mainstream speculative decoding approaches, the draft model typically relies on the target model's hidden features to predict the next token. The problem is that hidden features encode the contextual emphasis of the "current position," whereas the drafting process needs to generate candidate tokens for "multiple subsequent positions" simultaneously. There is a misalignment: the information emphasized by current features may not be the most critical for the next few draft positions.

HLA models' recurrent states are different. They accumulate a compressed representation of historical associations—essentially a summary of "what has been read." ReaDy's insight is that this summary can serve as a supplementary information source during drafting, compensating for hidden features' shortcomings in positional generalization.

What is Hybrid Linear Attention (HLA)? Traditional Transformer attention requires maintaining a full KV cache, with computation growing quadratically with context length. Linear attention-style architectures instead use a fixed-dimension recurrent state to accumulate historical information: with each token processed, a learnable update rule compresses new information into the state while discarding some old information. Hybrid linear attention retains standard attention in some layers and uses linear attention in the rest, balancing local precise retrieval with long-range memory. This recurrent state is essentially a lossy compression of "what has been read"—far smaller in dimension than the original KV cache, yet a natural intermediate representation produced during model inference. ReaDy specifically capitalizes on its property of being readable repeatedly at low cost.

Acceptance length and the economics of speculative decoding. The speedup of speculative decoding is determined by two quantities: the number of candidate tokens generated per draft step (draft length), and the proportion of those that pass verification by the target model (acceptance length). Only accepted tokens actually save a forward computation of the target model, while rejected tokens mean wasted draft computation. Therefore, improvements in draft quality typically manifest directly as growth in acceptance length. ReaDy reports an average 8.50% improvement in acceptance length, indicating that its retrieval mechanism genuinely brings drafts closer to the target model's distribution rather than merely increasing the number of candidates.

ReaDy's Three-Layer Design

State Reader. ReaDy equips each draft position with a learned query that retrieves context from the recurrent states already committed by the target model. The retrieved results are incorporated into the draft representation through residual updates. This design allows each draft position to obtain the context "it needs" rather than passively receiving the "radiation" of the current position's features.

Low-Rank Reading and Operator Fusion. To avoid introducing excessive computation and I/O overhead, ReaDy employs low-rank state reading and fuses the relevant operators. This keeps additional overhead at a low level, preventing it from canceling out the speedup gains from speculative decoding.

Two-Stage Adaptation Strategy. The method builds on DSpark's parallel backbone and sequence prediction head, adopting a lightweight two-stage adaptation. The research team conducted ablation experiments examining the effects of reader capacity, state sources, and residual placement, validating the necessity of each component.

Why low-rank reading and operator fusion are necessary. The speedup gains of speculative decoding inherently come from "trading drafts for time," and any additional overhead directly offsets net gains. Introducing a full state read for each draft position could bring matrix operations and memory accesses on the same order as the draft computation itself. Low-rank decomposition projects the read onto a smaller intermediate dimension, significantly reducing parameters and FLOPs; operator fusion merges adjacent operations like retrieval and residual updates into the same kernel, reducing memory read-write round trips. Both aim for the same goal: making the marginal cost of the enhancement mechanism far smaller than the acceptance length gains it brings, so that overall speedup remains positive.

Measured Results: Average 3.16× Speedup

Across seven benchmarks covering mathematics, code generation, and dialogue, ReaDy using Qwen3.8-27B on a single NVIDIA B200 GPU achieved an average 3.16× speedup over autoregressive decoding.

Compared with the baseline DSpark, ReaDy shows improvements on all benchmarks: throughput increases by an average of 7.97%, and acceptance length improves by an average of 8.50%. These two metrics reflect overall system efficiency and draft quality respectively—the higher the acceptance length, the higher the proportion of draft tokens recognized by the target model, and the less wasted computation.

The code is open-sourced on GitHub (StartLuxLabs/ReaDy), providing a reproducible foundation for follow-up research.

The DSpark baseline. DSpark is the backbone framework that ReaDy builds upon, providing the structure for parallel draft generation and the sequence prediction head. ReaDy does not replace this backbone but layers a state reading module on top and performs two-stage adaptation, so comparison with DSpark cleanly reflects the gains brought by recurrent state retrieval itself. The 7.97% throughput improvement is an end-to-end result after acceptance length improvements and additional overhead have been accounted for, not merely a theoretical gain.

Implications for the HLA Model Ecosystem

Hybrid linear attention architectures have been increasingly used in long-context scenarios in recent years, and their recurrent states are essentially a "byproduct" of model inference. ReaDy's value lies in transforming this intermediate representation—originally used only for internal computation—into usable information during speculative decoding, effectively giving the model's historical memory a "second shift" during drafting.

This approach's applicability is not limited to specific models. Any architecture maintaining compact recurrent states could potentially borrow similar retrieval mechanisms to enhance draft quality. As HLA-style models' advantages in inference efficiency gain recognition, work that leverages their internal states may increase.

Of course, ReaDy's current validation is relatively concentrated in terms of model scale and hardware conditions, and generalization across architectures and scales still requires more experimental verification.

Key Conclusions

ReaDy demonstrates that HLA recurrent states can serve as an effective context source for speculative decoding, achieving an average 3.16× speedup across seven benchmarks and comprehensively surpassing the DSpark baseline. The design of low-rank reading and operator fusion makes this enhancement viable under controllable overhead. For researchers and engineers focused on inference acceleration and linear attention architectures, this is a solution worth following.

One-line Summary

ReaDy uses HLA recurrent states to provide position-specific context for speculative decoding, achieving an average 3.16× speedup.

Paragraph Summary

ReaDy is a new method for speculative decoding. Its core insight: the target model's hidden features that traditional drafters rely on encode the current position's context, which is misaligned with the drafting need to generate candidate tokens for multiple subsequent positions. HLA models' recurrent states are compressed summaries of history that can serve as a supplementary information source during drafting. ReaDy equips each draft position with a learnable query to retrieve context from the target model's committed recurrent states and incorporates it into the draft representation via residual updates, while using low-rank reading and operator fusion to control overhead. Across seven benchmarks including mathematics, code, and dialogue, based on Qwen3.8-27B and a single B200 GPU, it achieves an average 3.16× speedup, with throughput up 7.97% and acceptance length up 8.50% compared with DSpark. The code is open-sourced.