vLLM v0.34.0rc1 Released: Fix for NaN Issue in XPU Speculative Decoding

vLLM has released candidate version v0.34.0rc1 (commit 52aeb58), an iterative release aimed at the next generation of the inference engine. The release notes are very brief; the core change is a fix for Speculative Decode on the XPU backend, tagged by contributor taneem-ibrahim on November 19.

Basic Release Information

v0.34.0rc1 follows vLLM's regular release cadence, and the rc1 suffix indicates it is a candidate release prior to the final version, intended for community validation and regression testing. The release page also provides a tag comparison view so users can inspect the full commit diff between this version and the previous tag.

Based on how the release assets are presented, this release includes 2 assets (Assets), but the asset list showed an error message when the page loaded; the actual files need to be viewed after refreshing the GitHub Releases page.

Core Fix: greedy_argmax Treats NaN as -inf

The most noteworthy change in this candidate release centers on the speculative decoding path. The commit title is "[Bugfix][Spec Decode][XPU] greedy_argmax: treat NaN as -inf on the to…", and the keywords break down as follows:

• Bugfix: This is a defect fix, not a new feature.
• Spec Decode: It applies to the speculative decoding pipeline, an inference technique that uses a small draft model plus large-model verification to accelerate generation.
• XPU: It targets the XPU acceleration backend from vendors such as Intel, rather than CUDA or ROCm.
• greedy_argmax: The fix is located in the argmax operator used for greedy sampling.

The fix logic treats NaN as negative infinity (-inf) during the argmax comparison. This is closely tied to floating-point comparison semantics: when NaN is present in a tensor, standard comparison operations return indeterminate results, which can cause argmax to select an invalid index. By treating it as negative infinity, NaN values are stably excluded from candidates, and greedy decoding can select the genuinely optimal token.

Why NaN Appears During Inference

In speculative decoding, the logits from the draft model and the target model need to be aligned and compared. Low-precision computation, division or normalization operations, and implementation differences of certain operators across hardware can all produce NaN. For the XPU backend, operator implementations differ from the CUDA ecosystem, making edge cases more likely to surface.

If NaN is not handled correctly, greedy sampling may output incorrect tokens, manifesting as degraded generation quality or even garbled output. Such issues may not reproduce in small-batch tests but appear intermittently with long sequences or specific inputs, so releasing a patch separately as a bugfix is a reasonable approach.

argmax and the floating-point semantics of NaN: The purpose of argmax is to return the index of the maximum value in a tensor, and greedy sampling relies on it to select the token with the highest probability. Under the IEEE 754 standard, comparisons between NaN and any value (including itself) return false, so argmax behavior when encountering NaN is indeterminate across implementations: it may skip it, or it may lock onto a NaN-containing position at the first comparison. By explicitly mapping NaN to -inf, it becomes "smaller" than any finite value and is therefore deterministically excluded, making argmax output independent of hardware and implementation. This kind of "NaN normalization" is a common convention in numerical computing libraries (such as NumPy's nanargmax).

logits and the draft-verify mechanism: In speculative decoding, the draft model first autoregressively generates several candidate tokens, and the target model then performs a single forward pass over the candidate sequence to obtain logits at each position for verification or resampling. The so-called "aligned comparison" refers to determining whether draft tokens are accepted by the target model during this process and, upon rejection, resampling according to the corrected distribution. Because this involves multiple normalization steps such as softmax, temperature scaling, and top-p truncation, any step that produces 0/0, log(0), or overflow will contaminate the logits and generate NaN. Once NaN enters argmax, the greedy path may select an incorrect index, which is why this defect only appears intermittently with specific inputs.

Continued Investment in the XPU Ecosystem

Although this fix is small, it reflects vLLM's continued investment in multi-hardware backends. As a non-NVIDIA acceleration path, XPU needs to maintain strict operator semantic consistency with the main path; otherwise, optimization techniques that rely on precise comparisons, such as speculative decoding, will be affected first.

For teams deploying vLLM on XPU, this candidate release is worth prioritizing in test environments: focus on output stability with speculative decoding enabled, and on whether greedy sampling behaves as expected with NaN-containing inputs.

XPU and backend abstraction: XPU is Intel's general term for its data center GPUs (such as the Arc series, apart from Ponte Vecchio and Gaudi) and integrated graphics, corresponding to the oneAPI/SYCL software stack, and its programming model differs from CUDA. vLLM dispatches operators to different backends through a platform abstraction layer, and attention kernels, sampling operators, and others often need separate implementations or adaptations for each backend. Differences in reduction order, special-value handling, and parallelism granularity across backends can cause the same Python logic to produce numerically different results. Therefore, "operator semantic alignment" is ongoing work in maintaining multi-backend inference engines, and it is also why such bugfixes are often released for a single backend only.

How to Participate in Validation

The point of a candidate release is to collect feedback. Before upgrading, it is advisable to check the full diff on the tag comparison page to confirm whether the fix covers your use case; after deployment, compare generation results with speculative decoding enabled and disabled to confirm that NaN-related anomalies no longer occur. If issues are found, you can file an issue in the vLLM repository with a reproduction script and hardware information.

For production environments, it is still recommended to wait for the final release before upgrading; test and staging environments can adopt v0.34.0rc1 early to accumulate validation data for the final release.

One-line Summary

vLLM releases v0.34.0rc1, fixing the NaN handling defect in argmax within XPU speculative decoding

Paragraph Summary

vLLM has released candidate version v0.34.0rc1 (commit 52aeb58), a community validation iteration prior to the final release. The core change is a bugfix submitted by taneem-ibrahim: in the speculative decoding path on the XPU backend, the greedy_argmax used for greedy sampling now treats NaN as negative infinity. Because NaN comparison results are indeterminate under IEEE 754, argmax may select an invalid index, leading to degraded generation quality or even garbled output; after stably excluding NaN, greedy decoding can select the genuinely optimal token. The article analyzes the causes of NaN under low-precision computation, normalization, and cross-hardware operator differences, notes that this fix reflects vLLM's continued investment in non-NVIDIA backends, and recommends prioritizing validation of output stability with speculative decoding enabled in test environments while waiting for the final release in production.