The FP32 Addition Trap: Why Adding Two Floats Sometimes Changes Nothing

A Counter-Intuitive Phenomenon

In deep learning and numerical computing, we tend to trust floating-point arithmetic without question — after all, computers should be able to add numbers accurately, right? As it turns out, that's not always the case. Here's a surprising and sobering fact: in FP32 (single-precision floating-point), addition can sometimes "do nothing". You execute a + b, and the result is identical to a — as if the addition never happened.

This isn't a bug. It's an inevitable consequence of how floating-point numbers are represented. Understanding this phenomenon is a foundational topic for any engineer working in machine learning training, scientific computing, or high-performance computing.

FP32's Precision Limit: Why Only 7 Significant Digits

FP32 uses 32 bits to represent a floating-point number: 1 sign bit, 8 exponent bits, and 23 mantissa bits. This design means it can only maintain roughly 7 significant digits of precision.

ULP: The Key Concept for Understanding Floating-Point Precision

To understand why addition can fail silently, we need to introduce a critical concept — ULP (Unit in the Last Place). A ULP refers to the smallest interval between two representable floating-point values at a given magnitude.

The crucial point is that the size of a ULP depends on the magnitude of the value itself, not a fixed constant. The larger the number, the wider the "gap" between adjacent representable floats.

Here's a concrete example: when a running sum reaches 1.0, the smallest representable change in FP32 is:

1 ULP = 2⁻²³ ≈ 0.00000012

This means that near 1.0, floating-point values can only step in increments of approximately 0.00000012.

Why Floating-Point Addition Can "Vanish": Absorption Error Explained

Now for the key insight. Under the IEEE 754 rounding rules (round-to-nearest-even), when you add a number to 1.0:

• If the addend is greater than half a ULP (i.e., greater than 0.00000006), the result is rounded to the next representable float — the addition takes effect.
• If the addend is less than half a ULP (i.e., less than 0.00000006), the result is rounded back to 1.0 — the addition has no effect whatsoever.

In other words, adding any value smaller than 0.00000006 to 1.0 still yields 1.0. The addition was technically "executed," but it changed nothing.

# Python example (verified using numpy float32)
import numpy as np
a = np.float32(1.0)
b = np.float32(0.00000005)
print(a + b == a)  # True —— addition had no effect

This is known as absorption error, often described colloquially as the "big number swallowing the small number" effect.

Real-World Impact on Deep Learning Training

What seems like an academic detail can have serious consequences in practice, especially in scenarios involving large-scale accumulation.

Cumulative Distortion in Long Sequence Summation

Consider a common scenario: summing a vector with millions of elements. As the running sum grows larger, subsequent smaller elements may be "swallowed" one by one, causing the final result to deviate significantly from the true value. The longer the array and the more uneven the value distribution, the more pronounced the error.

Hidden Dangers in Gradient Accumulation and Optimizers

When training large models, operations like gradient accumulation, momentum updates, and weight decay are all fundamentally additions. When model weights are large and gradient updates are tiny, some updates may fall below half a ULP and get silently absorbed — meaning some learning signal is quietly discarded and those parameters cannot be effectively updated.

Why Mixed Precision Training Is Necessary

This is one of the core reasons why modern training frameworks widely adopt mixed precision training and loss scaling techniques. By scaling small gradients up into a representable range and then scaling them back down at the appropriate moment, these techniques effectively prevent small values from being swallowed.

Practical Strategies for Mitigating Floating-Point Absorption Errors

The field has developed several proven solutions to address absorption errors:

Kahan Summation Algorithm

Kahan summation introduces a compensation variable that explicitly tracks the low-order bits lost in each addition and feeds them back into the next step. This classic numerical computing technique is straightforward to implement and significantly reduces accumulated error.

Using Higher-Precision Accumulators

For critical accumulation steps, use higher precision — such as FP64 or FP32 accumulators paired with FP16 inputs. Many GPU Tensor Cores internally use FP32 accumulators when multiplying FP16 values during matrix operations, precisely to mitigate absorption errors.

Block Summation to Reduce Risk

Divide large arrays into smaller chunks, sum each chunk separately, then combine the partial results. This prevents any single accumulator from growing too large too quickly, reducing the chance of small values being absorbed. This approach is especially practical in parallel computing contexts.

Takeaway: Floating-Point Numbers Are Not Real Numbers

The phenomenon of "an addition that adds nothing" is a reminder that floating-point numbers are not real numbers. They are finite-precision approximations of real numbers, and their precision changes dynamically with magnitude.

For AI engineers and scientific computing practitioners, understanding ULP, absorption errors, and rounding rules is not optional theory — it's essential knowledge for diagnosing numerical instability, training divergence, and non-reproducible results. When your model exhibits inexplicable behavior, the answer may well be hiding in those tiny numbers quietly being "swallowed" by floating-point arithmetic.