Draft Model

Parallel block drafting lets the draft model output predictions for multiple positions in a single forward pass rather than generating them one by one autoregressively, thereby improving drafting efficiency.

SpecCompact decomposes the compression task into multiple slots, uses a small draft model to generate candidate content in parallel, and then has the target large model verify and correct it.

By annealing the power parameter from zero to one, TVP achieves the highest or tied-highest acceptance length across three draft model architectures and three target models.

Existing draft models are mostly trained with cross-entropy (CE) or KL divergence.

Traditional soft-label distillation requires the draft model to approximate the target model's full probability distribution at every position, including high-entropy regions, which is inconsistent with the goal of maximizing acceptance length at inference time.

The core idea of speculative decoding is: use a smaller draft model to quickly generate multiple candidate tokens, then have the target large model verify in parallel in one pass whether these tokens are acceptable.

Small models in the same family (such as a version with 1/10 the parameters of the large model) have output distributions relatively close to the target model due to similar training data and architecture, and are a common choice for draft models.

Distilled models further improve alignment by having the draft model fit the target model's output distribution, thereby raising the acceptance rate.

Existing draft models see acceptance length drop by up to 80% in agentic scenarios.

Implementing speculative decoding requires finding a balance between the inference speed of the draft model and its prediction accuracy.

Draft models typically generate 3 to 8 candidate tokens at a time.

Co-design treats the draft model and the target model as an integrated system to be designed and trained from the outset, in order to improve the token acceptance rate.

Traditional draft models are trained on narrow distributions targeting a single model, causing acceptance rates to drop sharply when workloads change, and requiring the entire pre-training process to be redone each time a target model is replaced.

Traditional draft model training schemes require consuming the target model's hidden states and performing distillation on the target model's logits, creating a strong binding dependency on the target model.

The inference cost of the draft model in speculative decoding is typically only 1/5 to 1/10 that of the main model.