DFlash2
A speculative decoding implementation within the vLLM inference framework that combines two modules—a local convolution component and a candidate selector—to accelerate LLM inference through lightweight prediction and intelligent filtering.
Timeline (last 90 days)
HyperQwen is an Apache-2.0-licensed custom vLLM build released by GitHub user syv-ai, tuned for the Qwen3.8-27B model in the W4A16 quantisation format, with weights of about 19.5GB and DFlash2 speculative decoding.
Draft-flash-style schemes (such as DFlash2) let the draft phase reuse part of the main model's computation, making them more sensitive to KV cache and VRAM contention under long contexts, which is the cause of the 20k-context crash in the article.
The base model's 1M context is achieved by stretching its native 262,144 positions via YARN override, and the DFlash2 draft model cannot run simultaneously with the 1M context, so one must choose between 1M without a draft model and 256K with a draft model.
Below 64k, the phone cannot accelerate generation speed; generation speedup is achieved by custom kernels in the fork, including SME2 instructions on the M4 CPU, Metal fused operators, and DFlash2 speculative decoding, pushing generation from the original llama.cpp's 11.3 tok/s to approximately 25 tok/s at around 30k context.
vLLM v0.28.0 introduces DFlash2 and DSpark confidence-scheduled verification in the speculative decoding domain.
Injecting concept representation into the draft model of DFlash2 can improve the mean acceptance length by 4.17%.
vLLM v0.28.0rc2 introduces a new speculative decoding feature called DFlash2, which combines two major technical components: local convolution and a candidate selector (PR #52816).
When the cache hit boundary is not aligned with the block boundary, if the hit endpoint falls in the middle of a block, it triggers recomputation of the entire block, even though most tokens could have been reused.
In empirical tests, the draft token acceptance rate of DFlash2 is generally between 40% and 50%, reaching approximately 58%–60% at its higher end.
There is a bug in vLLM when prefix caching and DFlash2 are both enabled, which causes the last cache unit (a mixed-layout block of approximately 16 tokens) to be directly discarded and recomputed.
10 more timeline events
All Facts (14)
Draft-flash-style schemes (such as DFlash2) let the draft phase reuse part of the main model's computation, making them more sensitive to KV cache and VRAM contention under long contexts, which is the cause of the 20k-context crash in the article.
50%UnverifiedHyperQwen is an Apache-2.0-licensed custom vLLM build released by GitHub user syv-ai, tuned for the Qwen3.8-27B model in the W4A16 quantisation format, with weights of about 19.5GB and DFlash2 speculative decoding.
50%UnverifiedThe base model's 1M context is achieved by stretching its native 262,144 positions via YARN override, and the DFlash2 draft model cannot run simultaneously with the 1M context, so one must choose between 1M without a draft model and 256K with a draft model.
50%UnverifiedBelow 64k, the phone cannot accelerate generation speed; generation speedup is achieved by custom kernels in the fork, including SME2 instructions on the M4 CPU, Metal fused operators, and DFlash2 speculative decoding, pushing generation from the original llama.cpp's 11.3 tok/s to approximately 25 tok/s at around 30k context.
50%UnverifiedvLLM v0.28.0 introduces DFlash2 and DSpark confidence-scheduled verification in the speculative decoding domain.
50%UnverifiedInjecting concept representation into the draft model of DFlash2 can improve the mean acceptance length by 4.17%.
50%UnverifiedWhen the cache hit boundary is not aligned with the block boundary, if the hit endpoint falls in the middle of a block, it triggers recomputation of the entire block, even though most tokens could have been reused.
50%UnverifiedDFlash2 packs consecutive tokens into cache blocks of a fixed size (typically 16 tokens) under a mixed-layout scheme.
50%UnverifiedAt approximately 52 concurrent connections, the system is essentially 100% saturated, generating a maximum of approximately 704 tokens per second, but with latency spiking to 64 seconds.
50%UnverifiedIn actual testing, the draft acceptance rate was only approximately 41%, with the draft model outputting 1,055 tokens but only approximately 438 being accepted.
50%UnverifiedUnder DFlash2's hybrid layout, the draft model generates a block of 16 tokens at a time, and during verification the entire 16-token cache block must be deleted and recomputed, rather than discarding only a single token.
50%UnverifiedThe official figure states that DFlash2 achieves an average of 6 accepted tokens, but in actual testing the higher end was approximately 58%–60%, generally between 40%–50% (with 3–4 tokens actually accepted out of every 7).
50%UnverifiedMethods for improving the draft acceptance rate include using early checkpoints of the main model as the draft model, fine-tuning the draft model on domain-specific data, and using a stronger small model as the draft model.
50%UnverifiedvLLM v0.28.0rc2 introduces a new speculative decoding feature called DFlash2, which combines two major technical components: local convolution and a candidate selector (PR #52816).
50%