The "Uneven Performance" of Speculative Decoding: 7 Tokens Accepted per Step on Code, Only 2.5 on General Prose — Ceiling or Optimizable?

The Efficiency Paradox of Speculative Decoding: The Acceptance Rate Gap Between Code and Prose

Speculative Decoding is one of the mainstream techniques for accelerating large model inference today. Its core idea is to use a lightweight "drafter" model to rapidly generate multiple candidate tokens, which are then verified in parallel by the main model, thereby reducing the number of serial decoding steps. However, a real-world test from the Reddit community reveals a noteworthy "uneven performance" phenomenon: on code generation tasks, a block drafter accepts an average of about 7 tokens per step, while on open-domain prose, this number drops sharply to about 2.5.

The test was based on the Qwen3.8-27B model (EXL3 4.0bpw quantization target, DFlash2 block drafter quantized to EXL3 4bpw, 7-token draft, greedy decoding, output identical to normal decoding), running on 2 T4 GPUs. The data shows: in code scenarios, about 7.4 tokens are accepted per verification step, with throughput reaching 154 tok/s; in prose scenarios, only about 2.5 tokens are accepted per step, with throughput of 35-37 tok/s; while the baseline throughput of normal decoding is about 18-19 tok/s.

This gap is not an isolated case. According to an independent test by another user in the r/LocalLLM community, on the llama.cpp runtime and different GPUs, the same drafter measured an acceptance rate of 2.45 on prose; another user reported an acceptance rate of 2.62 when using the MTP (Multi-Token Prediction) method. Cross-validation from multiple sources points to one conclusion: the low prose acceptance rate is a characteristic of the drafter itself, not a problem with a specific runtime or hardware.

Why Is the Acceptance Rate for Prose Naturally Lower?

To understand this phenomenon, we need to return to the verification mechanism of speculative decoding. The drafter generates k candidate tokens at once (k=7 in this case), and the main model verifies each position; only tokens that match the main model's greedy choice are accepted. Once a position mismatches, all drafts after that position are discarded and must be rolled back and regenerated.

Code and prose differ fundamentally in "the predictability of the next token." Code has strict syntactic constraints and strong pattern regularity—indentation, bracket pairing, common API call sequences, variable naming conventions, etc.—making it easier for the drafter to guess the main model's choice. In natural language prose, however, every position has a large number of valid and reasonable continuations, and the main model's choice depends heavily on contextual semantics and the training distribution, making it hard for the drafter to hit precisely.

This leads to the first core question raised in the original post: Is a prose acceptance rate of around 2.5 the theoretical ceiling for block/parallel drafters? From an information-theoretic perspective, prose has a higher "effective branching factor," greater entropy at each position, and the distribution gap between the drafter and the main model is amplified, so the acceptance rate naturally declines. But whether this is the ceiling remains undetermined.

Effective branching factor is a concept in information theory and language modeling that refers to the number of candidates at the next token position that are "almost equally plausible" given the context. It is not equivalent to vocabulary size but is approximately characterized by distribution entropy: if the main model's output distribution is nearly uniform, the effective branching factor is large; if it is highly peaked, it is small. In speculative decoding, the drafter must hit the main model's greedy choice at each step, and the hit probability is roughly positively correlated with how peaked the distribution is. Code, due to syntax and API constraints, often has a peaked distribution and a small effective branching factor; open-domain prose is the opposite, where multiple words, sentence structures, and even punctuation are all reasonable, resulting in a large branching factor. This explains why the prose acceptance rate is naturally low and also suggests that the block length k should match the effective branching factor: too large a k amplifies the cost of mismatch.

Can Existing Drafter Approaches Break Through 3.5?

The original poster raised a key question to researchers in the speculative decoding field: has anyone seen EAGLE-style, block diffusion, or MTP drafters achieve an acceptance rate of about 3.5 on open-domain prose under greedy decoding?

From community feedback, the publicly available measured data mostly cluster in the 2.5-2.6 range, and no reports of stably breaking through 3.5 have been seen yet. This suggests that current drafter architectures may share a common performance bottleneck in prose scenarios. The EAGLE series improves draft quality through feature-level alignment, while MTP leverages the model's own multi-token prediction heads, but the gains of both on open-domain text seem to be limited by the diversity of language itself.

One direction worth exploring is dynamically switching drafters or block lengths based on content type. The original poster also mentioned this: adjusting the drafting strategy in real time during generation according to content type (code, prose, structured data). For example, code segments use a 7-token block length and a code-specialized drafter, while prose segments shorten the block length or switch to a more conservative drafting mode to reduce the overhead of invalid verification. This adaptive strategy is theoretically feasible, but it requires the runtime to identify content types and switch drafters at low cost, which is not low in engineering complexity.

EAGLE is a class of feature-level speculative decoding methods. Instead of drafting in token space, it trains a lightweight autoregressive head on the main model's penultimate-layer hidden states to predict the hidden features of the next position and then map them to tokens. Because it leverages the main model's internal representations, EAGLE's drafts are closer to the main model's distribution, and the acceptance rate is usually higher than that of an independent small-model drafter, and it supports tree-structured candidates. MTP (Multi-Token Prediction), on the other hand, gives the model additional prediction heads during training so that it outputs multiple future tokens at once, which can be used directly as a draft source at inference time. Both are ideas that "stay closer to the main model's distribution," but in the face of the high entropy of open-domain prose, feature alignment can only narrow rather than eliminate the distribution gap, which explains why they remain in the 2.5-2.6 range in public data.

Practical Implications for Inference Deployment

For teams actually deploying speculative decoding, this set of data offers several references. First, do not assume that the speedup of speculative decoding is consistent across all tasks. Code tasks can achieve a throughput improvement of about 4x or more (154 vs 18-19 tok/s), while prose tasks achieve only about 2x (35-37 vs 18-19 tok/s). When planning capacity and setting SLAs, evaluate separately by business content type.

Second, the choice of drafter should match the target workload. If the business is mainly code generation, the high acceptance rate of a block drafter can bring significant benefits; if it is mainly open-domain dialogue or long-form writing, you need to re-evaluate the return on investment of the drafter, and even consider whether a 2x speedup is worth the extra memory and complexity costs.

Finally, the room for improving the prose acceptance rate may not lie in the drafter itself but in optimizing the verification strategy. For example, allowing the main model to accept "equivalent" alternative tokens during verification (rather than strict greedy matching), or adopting a tree-structured verification scheme to evaluate multiple candidate paths simultaneously—these directions may bypass the ceiling of the drafter's hit rate. The original poster expressed willingness to share detailed data classified by prompt type; such fine-grained real-world measurements are of practical value for the community's research on this problem.

Tree-structured verification is an important direction for breaking through the limitations of linear drafting: the drafter no longer generates just a single k-token chain but a candidate tree (such as EAGLE-2's dynamic tree), and the main model verifies multiple paths in parallel in a single forward pass, accepting the longest consistent prefix among them. In this way, even if one path mismatches midway, other branches may still be accepted, thereby increasing the expected number of accepted tokens under the same verification overhead. Unlike strict greedy matching, tree-structured verification can also incorporate an "equivalent token" relaxation strategy, but note that relaxation changes the output distribution and may deviate from the determinism guarantee of greedy decoding. For high-entropy text like prose, the benefits of tree structures are theoretically more pronounced, at the cost of more complex attention masks and KV cache management during verification, and increased memory usage.

Community Collaboration and Open Questions

The value of this discussion lies in the cross-corroboration of multi-source data: the original author's measurements on 2x T4, an independent reproduction by an r/LocalLLM user on llama.cpp (2.45), and a report from an MTP user (2.62) together pinpoint the problem to drafter characteristics rather than runtime differences. This community collaboration model is especially important for highly engineering-oriented fields like inference optimization.

Questions that remain unresolved include: whether there is a rigorous analysis of the theoretical lower and upper bounds of the prose acceptance rate; whether there is a drafter architecture redesigned for high-entropy text; and whether the latency overhead of content-adaptive switching in real production environments is acceptable. The answers to these questions will determine whether speculative decoding can evolve from a "code generation accelerator" into a general-purpose acceleration solution for large model inference.