Set a Random Seed but Got Different Models? The Truth About PyTorch Reproducibility

Every training script opens with the same ritual: torch.manual_seed(0), np.random.seed(0), followed by a comfortable sense of certainty — this run is reproducible. The author admits to writing those lines for years. Then, while chasing an inexplicable 1-percentage-point accuracy gap, he finally tested that assumption directly: same seed, same data order, same model initialization, same machine, every condition identical. He trained ResNet18 for 200 steps, ran it twice, and compared the results one by one.

The outcome: the two loss curves aligned perfectly at step 0, then diverged starting at step 1. After 200 steps, the maximum difference between corresponding weights reached 0.237 — for parameters whose values mostly fall between −0.1 and 0.1, that's not drift. It's an entirely different model. Both runs had seeds set. Neither was actually reproducible.

Where Randomness Actually Sneaks In

The seed did its job: initialization, data order, and every explicit random sample were identical — which is exactly why step 0 aligned perfectly. The real divergence happens during backpropagation, and it's not randomness in the conventional sense — it's floating-point arithmetic meeting parallel hardware.

Floating-point addition is not associative: (a + b) + c and a + (b + c) can differ in the last bit. GPU kernels accumulate gradient contributions across tens of thousands of threads, many using atomic operations whose completion order depends on the scheduler — on which threads win the race in this particular run, at this particular clock cycle. Same numbers, different order of addition, gradients differ by one ULP.

Then SGD amplifies it: a 1e-8 divergence in gradients becomes slightly different weights, which become different activations, and the gap compounds with every step. By the end of the author's training run, noise that originally lived in the last representable decimal place had grown to 0.237. In the author's words, it's "the butterfly effect with a commit hash."

None of this is a PyTorch bug — it's documented, expected behavior. And cudnn.benchmark (which the author had benchmarked and kept enabled) adds a second layer of nondeterminism: it auto-tunes each run and may select different, nondeterministic convolution algorithms.

---

The precision problem is rooted in the IEEE 754 standard: 32-bit floats (fp32) have only about 7 significant decimal digits, and 64-bit (fp64) only about 15. When hundreds of thousands of gradient values are accumulated via atomic operations, each intermediate rounding introduces a ULP-level error. GPU atomic additions (atomicAdd) do not guarantee a global ordering — different warps complete their accumulations at different clock cycles, meaning that even with identical inputs, the structure of the reduction tree can vary between runs, producing gradients that differ in the last one or two bits. This isn't an intermittent hardware fault; it's the mathematical nature of parallel floating-point computation. There is an inherent tension between determinism and throughput, which is why NVIDIA explicitly documents which cuDNN and cuBLAS operations are deterministic and which are not.

Does 0.237 Actually Matter? Only When You're Asking Small Questions

For final accuracy, this nondeterminism behaves like an extra seed you didn't choose. The author measured typical seed-to-seed variance of roughly ±0.3–0.5 percentage points on a small CIFAR setup in earlier data augmentation experiments — GPU noise under the same seed falls in the same statistical range. If the effect size you care about is far larger than that variance, you can safely ignore this entire article.

What will actually bite you are the "small questions" — and most debugging questions are small questions. "Did my refactor change the numerics?" — you can't diff it against the pre-refactor run, because nothing has ever cleanly matched. "Is this 0.4-point improvement from my new loss term, or just luck?" — under nondeterminism, you can't even hold luck constant when switching loss terms.

A/B comparisons that require "the same noise on both sides" to be answered honestly are exactly where "seeded but nondeterministic" quietly wastes days of your time. The author learned this firsthand — the 1-point mystery that prompted this article turned out to be the same code run twice.

The Fix, and What It Actually Costs

PyTorch ships the switch for this. It's just not the seed:

torch.use_deterministic_algorithms(True)
torch.backends.cudnn.benchmark = False

torch.use_deterministic_algorithms(True) works because PyTorch maintains two implementations for many operators — one optimized for speed (potentially using nondeterministic parallel reductions) and one that guarantees determinism (typically by forcing serialized accumulation or using a deterministic alternative algorithm). With the flag enabled, the framework checks at operator call time whether a deterministic implementation exists; if not, it raises a RuntimeError rather than silently falling back to the nondeterministic path. The CUBLAS_WORKSPACE_CONFIG=:4096:8 environment variable tells cuBLAS to reserve a fixed-size workspace (4096 MB, up to 8 buffers) for deterministic algorithms, because certain deterministic matrix multiplication implementations need extra memory to avoid concurrent write conflicts. This variable must be set before the CUDA context is initialized — typically in the shell environment or os.environ. Setting it after import torch inside the program may be too late.

# For CUDA >= 10.2, set before process launch:
# CUBLAS_WORKSPACE_CONFIG=:4096:8

With these in place, the author's two 200-step runs achieved bit-for-bit identical results: maximum weight difference was exactly zero, every loss value matched to the last digit. Two things to know about the setup: the CUBLAS_WORKSPACE_CONFIG environment variable is mandatory (omitting it produces a clear RuntimeError); and use_deterministic_algorithms will refuse to run any operator that has no deterministic GPU implementation — ResNet18 passes fine, but if your model contains a holdout operator, the flag will fail loudly, leaving you with a CPU fallback or accepting nondeterminism for that operator. Failing loudly here is actually the good outcome.

The Real Performance Cost

The commonly cited penalty is "2–3× slower," typically invoked as a reason not to bother. But the author's measurements on ResNet18 training steps show:

Precision	Default	Deterministic	Overhead
fp32	17.94 ms	20.90 ms	+16.5%
bf16 autocast	12.42 ms	14.37 ms	+15.6%

Just 16%. And part of that isn't even pure determinism overhead — it includes giving up cudnn.benchmark, which the author measured as roughly 3% on this model. The "2–3×" figures probably come from particularly unlucky operators or older software stacks, and maybe your model has one — which is exactly why it's worth spending two minutes to measure yourself. On the author's model, the cost of bit-for-bit reproducibility is roughly equivalent to bumping the batch size up a little.

Where This Flag Belongs Now

The author's default setup remains fast and nondeterministic — 16% is a real cost on training throughput, and ordinary training doesn't require bit-exact precision. But the flag has earned three fixed use cases:

• Any debugging session that requires comparing two runs (which means almost any debugging session);
• Numerically sensitive refactors, when you want git diff-level certainty that outputs haven't changed;
• Small-effect-size experiments where the noise floor would otherwise swallow the results.

The decision rule is a single question: Am I about to compare two runs and interpret the difference between them? If yes, 16% is the cheapest insurance in the entire optimization playbook — and unlike the seed-setting ritual, it actually delivers what it promises.