Conceptlack of batch invariance
Lack of Batch Invariance
A phenomenon in inference engines in which the same input produces different outputs under different batches due to batching, rooted in the fact that the order of floating-point operations changes with batch shape.
Timeline (last 90 days)
Oct 18
In a 2025 analysis, Thinking Machines Lab named the vLLM temperature=0 nondeterminism problem "lack of batch invariance."
Unverified50%