[KongchangAI]
Model

ForeSpark

A joint budget scheduling system for parallel speculative decoding that achieves work-conserving speculation through context-aware pre-draft estimation, lightweight joint scheduling, and post-draft budget allocation, improving LLM inference throughput.

Timeline (last 90 days)

Nov 11

The paper "ForeSpark: Joint Draft and Verification Budget Scheduling for Parallel Speculative Decoding" proposes a Work-Conserving Speculation (WCS) design.

Unverified50%
Nov 11

ForeSpark contains three key components: a context-aware pre-draft estimator, a lightweight joint draft-verification scheduler, and post-draft budget allocation.

Unverified50%
Nov 11

ForeSpark's pre-draft estimator uses contextual features of the target model to predict the acceptance probability of each candidate draft length.

Unverified50%
Nov 11

ForeSpark's lightweight joint scheduler selects a draft length and verification length for each request through global ranking, achieving batch-level joint optimization.

Unverified50%
Nov 11

After draft generation, ForeSpark allocates the verification budget to the most promising tokens, prioritizing verification of the candidates with the highest acceptance probability.

Unverified50%
Nov 11

ForeSpark is implemented on vLLM and was compared with the DSpark system.

Unverified50%
Nov 11

At a matched per-request generation rate, ForeSpark's throughput is up to 1.77 times higher than DSpark and up to 2.49 times higher than autoregressive decoding.

Unverified50%
Nov 11

Under a latency service level objective (SLO), the request rate that ForeSpark can sustain is up to 1.51 times that of DSpark.

Unverified50%
Nov 11

ForeSpark improves throughput while not changing the output distribution, preserving generation quality.

Unverified50%

All Facts (9)

Source Articles