Draft Model
A small auxiliary model used in speculative decoding that rapidly generates candidate token sequences autoregressively for parallel verification by the large target model. Its degree of distributional alignment with the target model directly determines the acceptance rate and the speedup achieved.
Timeline (last 90 days)
Distilled models further improve alignment by having the draft model fit the target model's output distribution, thereby raising the acceptance rate.
Small models in the same family (such as a version with 1/10 the parameters of the large model) have output distributions relatively close to the target model due to similar training data and architecture, and are a common choice for draft models.
The core idea of speculative decoding is: use a smaller draft model to quickly generate multiple candidate tokens, then have the target large model verify in parallel in one pass whether these tokens are acceptable.
Existing draft models see acceptance length drop by up to 80% in agentic scenarios.
Implementing speculative decoding requires finding a balance between the inference speed of the draft model and its prediction accuracy.
Draft models typically generate 3 to 8 candidate tokens at a time.
Co-design treats the draft model and the target model as an integrated system to be designed and trained from the outset, in order to improve the token acceptance rate.
Traditional draft models are trained on narrow distributions targeting a single model, causing acceptance rates to drop sharply when workloads change, and requiring the entire pre-training process to be redone each time a target model is replaced.
Traditional draft model training schemes require consuming the target model's hidden states and performing distillation on the target model's logits, creating a strong binding dependency on the target model.
The inference cost of the draft model in speculative decoding is typically only 1/5 to 1/10 that of the main model.
All Facts (10)
Distilled models further improve alignment by having the draft model fit the target model's output distribution, thereby raising the acceptance rate.
50%UnverifiedSmall models in the same family (such as a version with 1/10 the parameters of the large model) have output distributions relatively close to the target model due to similar training data and architecture, and are a common choice for draft models.
50%UnverifiedThe core idea of speculative decoding is: use a smaller draft model to quickly generate multiple candidate tokens, then have the target large model verify in parallel in one pass whether these tokens are acceptable.
50%UnverifiedExisting draft models see acceptance length drop by up to 80% in agentic scenarios.
50%UnverifiedImplementing speculative decoding requires finding a balance between the inference speed of the draft model and its prediction accuracy.
50%UnverifiedDraft models typically generate 3 to 8 candidate tokens at a time.
50%UnverifiedCo-design treats the draft model and the target model as an integrated system to be designed and trained from the outset, in order to improve the token acceptance rate.
50%UnverifiedTraditional draft models are trained on narrow distributions targeting a single model, causing acceptance rates to drop sharply when workloads change, and requiring the entire pre-training process to be redone each time a target model is replaced.
50%UnverifiedTraditional draft model training schemes require consuming the target model's hidden states and performing distillation on the target model's logits, creating a strong binding dependency on the target model.
50%UnverifiedThe inference cost of the draft model in speculative decoding is typically only 1/5 to 1/10 that of the main model.
50%