RCSD: Reinforcement Learning Boosts Speculative Decoding for 1.35x Inference Speedup

Speculative Decoding is one of the mainstream techniques for accelerating large language model inference. Its core idea is to use a small draft model to quickly generate candidate tokens, which are then verified in parallel by the target model. However, the training method of the draft model has always had hidden issues: traditional soft-label distillation forces the draft model to mimic the full output distribution of the target model, including high-entropy regions—these regions not only lower the acceptance rate but also increase the risk of early rejection. To address this problem, researchers proposed RCSD (Reinforcement and Continual Learning for Domain-Specific Speculative Decoding), a training framework that directly aligns draft model optimization with system-level objectives (maximizing acceptance length).

The Bottleneck of Speculative Decoding: Limitations of Soft-Label Distillation

The speedup of speculative decoding depends on the proportion of tokens proposed by the draft model that are accepted by the target model. The higher the acceptance rate, the more tokens can pass a single verification, and the lower the end-to-end latency. The traditional approach is to have the draft model learn the target model's output distribution through soft-label distillation, but this is equivalent to requiring the draft model to approximate the target model's full probability distribution at every position. The problem is that the target model's distribution in high-entropy regions is itself very diffuse, and even if the draft model tries hard to imitate it, it is difficult to give high-confidence predictions at these positions, leading to a drop in acceptance rate. Worse still, once an early token is rejected, the entire subsequent candidate sequence becomes invalid, greatly reducing the speedup benefit.

The authors of RCSD point out that there is a misalignment between this training objective and the actual objective at inference time: at inference time we only care about acceptance length, not whether the draft model perfectly reproduces the target model's distribution. Therefore, a new training paradigm is needed that lets the draft model optimize directly for being accepted.

Soft-label distillation is a form of knowledge distillation: the teacher model (here, the target large model) outputs a full probability distribution at each position (i.e., "soft labels"), and the student model (the draft model) learns by minimizing the difference from the teacher distribution (e.g., KL divergence). Compared with hard labels (one-hot labels from argmax only), soft labels carry more information such as inter-class similarities, and usually allow the student model to learn better. However, in the speculative decoding setting, soft-label distillation requires the student model to approximate the teacher's full distribution at every token position, including high-entropy regions where the teacher's own distribution is diffuse. This per-position distribution-matching objective is inconsistent with the system objective of "maximizing consecutive acceptance length" at inference time, becoming a bottleneck for improving the acceptance rate.

RCSD's Core Mechanism: Implicit Preference Pairs and Masked Loss

The key innovation of RCSD is to use the acceptance and rejection sequences naturally generated during speculative inference to construct implicit preference pairs. Specifically, during inference, tokens proposed by the draft model are verified by the target model, and the accepted token sequences and sequences containing rejections naturally form a set of contrastive data. RCSD uses these data as preference signals in reinforcement learning, encouraging the draft model to generate token sequences more likely to be accepted while suppressing generation patterns prone to rejection.

To avoid unintended harm, RCSD also introduces a masked loss. During training, for tokens that are aligned with the target model (i.e., tokens that the target model itself would also generate with high probability), the masking mechanism avoids penalizing them. In this way, the draft model will not deviate from the target model's already-correct predictions due to reinforcement learning signals, ensuring that generation quality is unaffected. This design allows RCSD to improve the acceptance rate without sacrificing output accuracy.

Implicit preference pairs draw on the idea of preference optimization in reinforcement learning from human feedback (RLHF) (such as DPO), but do not require additional human annotation: in each inference verification of speculative decoding, the token sequences accepted by the target model naturally constitute "positive examples," and sequences containing rejections constitute "negative examples," forming a contrast. RCSD uses this contrastive signal generated online as preference data to directly optimize the draft model to generate sequences that are more likely to be accepted. Compared with the RLHF pipeline that requires a reward model, implicit preference pairs eliminate explicit reward modeling, making training lighter, and the signal is fully consistent with the actual acceptance behavior at inference time.

Experimental Results: 35% Improvement in Acceptance Length, 1.35x End-to-End Speedup

The researchers evaluated RCSD on multiple benchmarks. Compared with the existing HASS method, RCSD improves the average acceptance length by 35%, achieves a 1.35x end-to-end inference speedup under real workloads, and does not degrade generation quality. The increase in acceptance length means that more tokens can pass each verification, thereby reducing the number of verification rounds of the target model, which is the direct source of speedup. You may not have noticed that RCSD's training framework has the characteristic of continual learning and can adapt to domain-specific data distributions, which is especially important for domain-specific scenarios.

From a system perspective, a 1.35x end-to-end speedup is significant in deployment environments sensitive to inference cost. Speculative decoding itself can already bring speedup, and RCSD further taps the potential of the draft model on this basis, without modifying the target model or inference architecture. It is a pure training-side optimization with low deployment cost.

HASS (Harmonizing Accuracy and Efficiency for Speculative Decoding) is a representative baseline method for training draft models in speculative decoding. Its approach also focuses on the alignment quality between the draft model and the target model, but it is still mainly based on distribution-matching training objectives. RCSD chooses HASS as the main comparison object in the paper because both attempt to address the limitations of soft-label distillation, while RCSD further aligns the training objective directly with acceptance length through reinforcement-learning-style preference optimization, thereby achieving a higher acceptance rate and end-to-end speedup under the same inference architecture.

Implications for Domain-Specific Deployment

RCSD's "Domain-Specific" positioning deserves attention. In practical applications, enterprises often fine-tune the target model on domain-specific data, and the draft model also needs to adapt accordingly. RCSD's continual learning mechanism allows the draft model to continuously learn from acceptance/rejection signals during inference, gradually adapting to the target model's behavior in a specific domain. This online adaptation capability has practical value for low-latency inference services in vertical domains such as law, healthcare, and finance.

However, RCSD is still at the academic research stage, and its code and models have not yet been released. Its actual deployment performance still needs to be verified on more hardware and model combinations. But the direction it reveals—using reinforcement learning to align with the system-level objectives of speculative decoding—provides a clear idea for subsequent research.

Continual Learning refers to a model continuously updating its parameters from new data or new signals without forgetting what it has already learned. In RCSD, the draft model continuously receives acceptance/rejection feedback during inference after deployment and fine-tunes itself accordingly, gradually adapting to the behavior distribution of the target model on domain-specific data. This differs from the traditional "train once, deploy frozen" paradigm and belongs to online adaptation. Its challenges lie in avoiding catastrophic forgetting and maintaining training stability. RCSD constrains the update direction through mechanisms such as masked loss, allowing it to adapt to new domains without damaging existing alignment capabilities.

Conclusion

RCSD reconstructs the training method of draft models in speculative decoding through reinforcement learning and continual learning, replacing traditional soft-label distillation with implicit preference pairs and masked loss, and achieves a 35% improvement in acceptance length and a 1.35x end-to-end speedup on multiple benchmarks. This work reminds us that inference acceleration relies not only on architectural innovation but also on alignment of training objectives. For developers and researchers concerned with large model inference efficiency, RCSD provides a new idea worth following.

One-line Summary

RCSD uses reinforcement learning to align speculative decoding objectives, improving acceptance length by 35% and achieving 1.35x end-to-end speedup.

Paragraph Summary

RCSD is a novel training framework for draft models in speculative decoding. Traditional soft-label distillation requires the draft model to imitate the target model's full distribution at every position, making it difficult to give high-confidence predictions in high-entropy regions and leading to low acceptance rates and high risk of early rejection. RCSD instead uses the acceptance/rejection sequences naturally generated during inference to construct implicit preference pairs, directly optimizing acceptance length with reinforcement learning signals, and introduces masked loss to avoid penalizing tokens already correctly aligned with the target model. On multiple benchmarks, RCSD improves the average acceptance length by 35% over HASS, achieves a 1.35x end-to-end inference speedup without degrading generation quality. Its continual learning mechanism can also adapt to domain-specific data distributions, making it suitable for low-latency deployment in vertical domains, but the code has not yet been released and actual performance still requires more verification.