ForeSpark
A joint budget scheduling system for parallel speculative decoding that achieves work-conserving speculation through context-aware pre-draft estimation, lightweight joint scheduling, and post-draft budget allocation, improving LLM inference throughput.
Timeline (last 90 days)
The paper "ForeSpark: Joint Draft and Verification Budget Scheduling for Parallel Speculative Decoding" proposes a Work-Conserving Speculation (WCS) design.
ForeSpark contains three key components: a context-aware pre-draft estimator, a lightweight joint draft-verification scheduler, and post-draft budget allocation.
ForeSpark's pre-draft estimator uses contextual features of the target model to predict the acceptance probability of each candidate draft length.
ForeSpark's lightweight joint scheduler selects a draft length and verification length for each request through global ranking, achieving batch-level joint optimization.
After draft generation, ForeSpark allocates the verification budget to the most promising tokens, prioritizing verification of the candidates with the highest acceptance probability.
ForeSpark is implemented on vLLM and was compared with the DSpark system.
At a matched per-request generation rate, ForeSpark's throughput is up to 1.77 times higher than DSpark and up to 2.49 times higher than autoregressive decoding.
Under a latency service level objective (SLO), the request rate that ForeSpark can sustain is up to 1.51 times that of DSpark.
ForeSpark improves throughput while not changing the output distribution, preserving generation quality.
All Facts (9)
The paper "ForeSpark: Joint Draft and Verification Budget Scheduling for Parallel Speculative Decoding" proposes a Work-Conserving Speculation (WCS) design.
50%UnverifiedForeSpark contains three key components: a context-aware pre-draft estimator, a lightweight joint draft-verification scheduler, and post-draft budget allocation.
50%UnverifiedForeSpark's pre-draft estimator uses contextual features of the target model to predict the acceptance probability of each candidate draft length.
50%UnverifiedForeSpark's lightweight joint scheduler selects a draft length and verification length for each request through global ranking, achieving batch-level joint optimization.
50%UnverifiedAfter draft generation, ForeSpark allocates the verification budget to the most promising tokens, prioritizing verification of the candidates with the highest acceptance probability.
50%UnverifiedForeSpark is implemented on vLLM and was compared with the DSpark system.
50%UnverifiedAt a matched per-request generation rate, ForeSpark's throughput is up to 1.77 times higher than DSpark and up to 2.49 times higher than autoregressive decoding.
50%UnverifiedUnder a latency service level objective (SLO), the request rate that ForeSpark can sustain is up to 1.51 times that of DSpark.
50%UnverifiedForeSpark improves throughput while not changing the output distribution, preserving generation quality.
50%