Temperature=0 Still Can't Guarantee Reproducibility: The Hidden Divergence Problem in AI Coding Agents

A Counterintuitive Experimental Result

When working with large language models, many developers assume that setting the sampling temperature to 0 makes model outputs deterministic and reproducible. Yet a Reddit user testing a long-horizon coding Agent discovered something surprising — even with temperature=0, multiple runs of the same task consistently led to different outcomes.

The researcher used a self-hosted Qwen model with the vLLM inference framework, running the exact same SWE-bench task 13 times under identical model configurations and temperature settings. The results were striking:

• In 13 executions, 12 independently reached exactly the same intermediate repository state (canonical Git diffs were identical);
• But those 12 runs then diverged into 11 different final source code states.

In other words, the Agent converged tightly at an intermediate checkpoint, then scattered in completely different directions at the finish line.

Why Does temperature=0 Still Diverge?

A temperature of 0 means the model greedily selects the highest-probability token at every step — theoretically deterministic. But in real inference systems, determinism is quietly undermined by several factors.

Non-Determinism in Floating-Point Arithmetic

When performing large-scale matrix operations on GPUs, floating-point addition is not strictly associative. Different parallel reduction orderings, varying batch sizes, and dynamic memory scheduling can all introduce tiny differences in logits. When two candidate tokens have nearly identical probabilities, even microscopic numerical differences are enough to flip a greedy decoding decision.

Batch Scheduling in Inference Frameworks

High-throughput inference frameworks like vLLM use continuous batching and PagedAttention, meaning the way requests are batched together changes with runtime state. The same prompt processed in different batching contexts can follow different computational paths, amplifying numerical perturbations.

Error Accumulation in Agent Loops

More critically, coding Agents are long-horizon, multi-turn autonomous systems. Each step's output becomes the next step's input, so a single token divergence compounds over time. This is exactly what the researcher observed — multiple runs first converged tightly to the same intermediate state, then got pushed onto different trajectories by small perturbations in subsequent steps.

Three Divergence Patterns

Across a broader task set, the researcher observed three notable patterns:

1. Divergence absorbed: Trajectories briefly split, but subsequent steps "smooth over" the difference, eventually converging back to a consistent state;
2. Divergence sustained: Once a split occurs, it cannot be undone, leading directly to different outcomes;
3. Diverge → reconverge → diverge again: Runs fork, then merge back to exactly the same source code state, then fork again.

The third pattern is particularly intriguing — it suggests the existence of "attractors" in the Agent's state space, where multiple different paths get pulled back to the same intermediate point, but that convergence is unstable.

When Does Divergence Actually Matter?

The researcher then asked a more nuanced question: does this variation actually matter?

The answer isn't straightforward. Different execution trajectories producing different patches can all be correct — the same bug often has multiple valid fixes. Simply counting "inconsistent final states" doesn't tell us the Agent is unreliable.

The research question was therefore reframed more precisely:

When does a benign execution variation become a divergence with real consequences?

To answer this, the researcher plans to introduce external correctness labels — using tests or human judgment to mark each final state as correct or incorrect — in order to distinguish "benign diversity" from "fatal divergence."

Early Detection and Proactive Correction Strategies

The real value of this research lies in the engineering direction it points toward: can we detect "critical turning points" before the Agent fails, and steer it back on track?

The researcher outlined several potential intervention strategies:

• Nudge: Inject a prompt when deviation is detected to guide the Agent back toward the correct path;
• Rollback: Revert to the most recent known-good state and re-execute;
• Context intervention: Modify the context to correct the Agent's understanding;
• Fork-and-select: Run multiple branches in parallel and select the best result.

These strategies essentially transform "divergence" from a bug into an exploitable resource — if we can determine in real time which trajectory is more promising, diversity becomes a tool for improving success rates.

Implications for Production Environments

For teams deploying open-weight Agents in production, this experiment offers several practical reminders:

• Don't rely on temperature=0 for reproducibility. In long-horizon autonomous tasks, numerical non-determinism gets systematically amplified — failed reproduction is the norm, not the exception.
• Evaluation metrics need to distinguish "correctness" from "consistency". Different final states don't mean failure; measuring with external correctness labels is what actually matters.
• Actively embrace diversity. Strategies like fork-and-select and majority voting across multiple samples may improve real-world task success rates more reliably than chasing deterministic single-run outputs.

The original poster also posed an open question to the community: have engineers running open-weight Agents in production also observed significant execution variance at temperature=0, and has it actually affected task success rates? This remains an open question, but it cuts to the heart of one of the core challenges in deploying autonomous Agents reliably.

Conclusion

This seemingly simple "run it 13 times" experiment reveals an underappreciated truth in current AI Agent engineering: determinism is an illusion, and reliability must be actively designed for. As coding Agents take on increasingly long and critical tasks, understanding and managing this hidden divergence will become the dividing line between a "toy demo" and a production-grade system.