vLLM temperature=0 Is Not Deterministic: 23 of 200 Evaluation Answers Flip

A Failure Without Any Prompt Change

On Tuesday, while running a regression test, the author found that 23 out of 200 evaluation prompts produced answers different from the previous run. The server hadn't been redeployed, the model weights hadn't changed, and temperature=0 was still set. At first he suspected someone had tampered with things and spent an hour investigating, only to discover the culprit was vLLM's batching mechanism.

This phenomenon is not isolated. In a 2025 analysis, Thinking Machines Lab named the problem "lack of batch invariance." Simply put, greedy decoding itself is deterministic, but the logits computation it relies on is not. Once your request shares a GPU with other people's requests, the batch shape changes with traffic, which in turn alters the summation order of floating-point operations.

Why Batch Size Changes Results

vLLM uses continuous batching, packing all in-flight sequences into each forward pass. Matrix multiplication may use split-K reduction for small batches, attention decoding may partition the KV sequence based on the number of queries, and RMSNorm's reduction strategy also varies with batch size. Each strategy is mathematically correct, but they add the same set of numbers in different orders.

And floating-point addition is not associative—(0.1+0.2)+0.3 and 0.1+(0.2+0.3) produce different results even in Python. Model inference uses bf16, with only 7 mantissa bits, which amplifies rounding errors. A single logit is the result of thousands of multiply-adds across 32 layers; when the batch changes, the last few significant digits drift.

Continuous batching is the core scheduling technique of inference engines like vLLM: instead of waiting for an entire batch of requests to complete before returning results, it checks the queue after each token is generated, immediately inserts newly arrived requests into the current batch, and removes completed sequences. This keeps the GPU from idling and greatly improves throughput. But for determinism, it means the same request run at different times joins a completely different batch composition—the lengths, counts, and generation stages of other sequences in the batch are all changing. Continuous batching maximizes utilization rather than fixing the computation graph, so it inherently conflicts with the goal of "same input, same computation path." This is why answers can differ word-for-word even when temperature=0, model weights, and code remain unchanged.

Split-K reduction is a matrix multiplication optimization: when the output dimension of a matrix multiplication is small and the K dimension (reduction dimension) is large, a single thread block has insufficient work and GPU utilization is low. Split-K divides the K dimension into segments, computes partial sums in parallel across multiple thread blocks, then adds these partial sums together. Mathematically the result is equivalent, but the order in which partial sums are added depends on the number of splits and the scheduling, and the number of splits is determined by batch size and matrix shape. Similarly, attention decoding partitions the KV sequence for parallel reduction based on the number of queries, and RMSNorm selects different vectorized reduction strategies based on element count. All these kernels are correct, but floating-point addition is not associative—change the reduction order and the last few significant digits differ.

How a Single Token Flip Ruins an Entire Answer

Most of the time, tiny differences in logits don't matter. Only when the top-1 and top-2 candidates are nearly tied does argmax flip. The author traced back the 23 flipped answers and found that at every point of first divergence, the logprob gap between top-1 and top-2 was extremely small.

Common flip points include "However" vs "But," whether a list uses "1." or bullet points, "is" vs "can be." These choices are stylistically trivial, but autoregressive generation uses the first flipped token as context for all subsequent tokens. Swap "However" for "But" at step 40, and every step after 41 is computed based on a different context, where new near-ties get resolved differently. In 3 of the author's answers, the final conclusions differed—all rooted in rounding errors in the fourth significant digit.

The seed Parameter Won't Save You

seed controls the random number generator used for sampling. Under temperature=0 there is no sampling, only argmax, so seed has nothing to do. Nondeterminism occurs in the forward pass, which seed can't touch. Hosted APIs are the same—providers batch your request with others', and their documentation typically only says temperature=0 gives "approximately" deterministic results, never guaranteeing byte-level consistency.

bf16 (bfloat16) is a 16-bit floating-point format with 1 sign bit, 8 exponent bits, and 7 mantissa bits. Compared to fp16, bf16 sacrifices mantissa precision for the same exponent range as fp32, making it less prone to overflow and well-suited for deep learning training and inference. But 7 mantissa bits mean effective decimal precision of only about 2 to 3 digits, with each multiply-add introducing a relative error of roughly 2^-8 ≈ 0.4%. A logit passes through 32 layers with thousands of multiply-adds each, and errors accumulate. When the true gap between two candidate logits is smaller than this accumulated error, the argmax result is determined by rounding noise rather than semantics. This is the physical root cause of why batch size changes can flip tokens.

Five Practices for Making LLM Evaluations Reproducible

After hitting this pitfall, the author put together a set of actionable practices. The core idea: first determine which level of determinism you need, then pay the cost only for that level.

1. Stop asserting exact string equality. Most evaluations care about whether the answer is correct, not whether it's word-for-word identical. Use checkers, parsers, or rubrics for scoring—this alone will stabilize your test suite.

2. Serialize requests when you need bit-level consistency. Setting --max-num-seqs 1 made all 200 prompts consistent across two runs for the author. The cost is a throughput collapse, so use it only on dedicated evaluation servers, never in production.

3. Look for batch-invariant modes. Newer versions of vLLM and SGLang have added experimental determinism or batch-invariance options that use kernels with fixed reduction order, independent of batch size. They're slower, and the flags differ between versions, so be sure to check the documentation for your specific version before using them.

4. Log logprob gaps. Request logprobs=True and top_logprobs=2. If a test case has a position within the first 50 tokens where the top-1/top-2 gap is less than about 0.05, it will likely flake. Flag it, or rewrite the prompt so the model isn't walking a tightrope.

5. Run each test case multiple times. A test case that passes 2 out of 3 times is itself real information about prompt fragility. Hiding it behind a single lucky greedy run is the worse choice.

The Author's Daily Checklist

• Separate evaluation servers from production; add --max-num-seqs 1 when comparing prompt versions
• Score semantically or programmatically—never use == on full text
• Store top_logprobs=2 for the first 100 tokens on every run
• Tag test cases with near-ties within the first 50 tokens as fragile and report them separately
• Judge prompt changes by pass rate across 3 runs, not a single result

Other Sources of Nondeterminism

Batch size is the most common offender in steady state because traffic changes every second. But there are other sources under the same theme: prefix cache reuse computes KV values in a batch different from a fresh prefill; chunked prefill for long prompts gets scheduled alongside other work, again changing shapes; all-reduce under tensor parallelism has its own reduction order; different GPU models or CUDA versions may simply select different kernels.

Together these point to one fact: the same input takes a different computation path. For teams doing LLM evaluation and prompt iteration, accepting this and designing processes around it is more practical than chasing nonexistent perfect determinism. The author concludes that his prompt change was actually fine—it just took a day to prove it.

Prefix caching is an inference engine optimization that reuses KV cache for requests sharing the same prompt prefix. When request A's prefix hits the cache, its KV values were computed and stored in some earlier batch, while request B's identical prefix may be recomputed in a new batch. Even if mathematically equivalent, cached KV values may differ at the floating-point level from those of a fresh prefill, because the batch shape, kernel selection, and reduction order at computation time differed. This difference propagates through all subsequent tokens. Similarly, chunked prefill splits long prompts into chunks computed across batches; the chunk size and scheduling depend on the load from other requests at the time, so the same long prompt takes different computation paths under different traffic.