[KongchangAI]
· 1 min read· 966 words

The Cost of FP4 Pretraining: Rare-Fact Learning Delays Hidden by Loss Curves

The Cost of FP4 Pretraining: Rare-Fact Learning Delays Hidden by Loss Curves

Study finds FP4 quantization significantly degrades rare-fact learning even when validation loss barely changes.

Using synthetic biography data and controlled exposure experiments, this study reveals a hidden cost of FP4 quantization (especially NVFP4) in rare-fact acquisition. When validation loss differs by only 0.5% to 2%, NVFP4 requires 1.32–1.52× more exposures than BF16 to reach the same rare-fact recall. Mechanistic analysis shows that quantization noise injected into parameter updates carries about two-thirds of the cost and persists until training ends, and raising the learning rate cannot compensate. The study also proposes using gradient statistics from BF16 checkpoints to predict quantization cost, with a ranking correlation of 0.96. The authors recommend evaluating low-precision recipes with rare-fact acquisition measured alongside loss.

Low-precision pretraining is typically judged by validation loss—but does matching validation loss mean a model learns rare facts just as efficiently? A new study using synthetic biography data and controlled exposure experiments reveals a hidden cost of FP4 quantization (especially NVFP4) in rare-fact acquisition: when validation loss differs by only 0.5% to 2%, NVFP4 requires 1.32× to 1.52× more exposures than BF16 to reach the same recall. This finding challenges the convention of evaluating low-precision training recipes solely by loss curves.

FP4 pretraining research screenshot

The Blind Spot Beyond Loss Curves: Rare-Fact Learning Efficiency

The study trains Transformer models from scratch using simulated 4-bit matrix multiplication, controls the number of exposures per fact on synthetic biography data, and tests both standalone training and training mixed with web text. The core comparison is between two precision formats: BF16 and NVFP4. The results show that although validation loss differs only minimally—0.5% and 2%, respectively—NVFP4 requires significantly more exposures for rare-fact recall: 1.32× in standalone training and 1.52× in mixed training. When both models are evaluated using BF16 matrix multiplication, these multipliers drop to 1.26× and 1.30×. The research team notes that under the primary reading, the exposure cost of matching loss is about 1.2×.

This result implies that loss curves may mask the negative impact of low-precision training on rare knowledge acquisition. For applications that rely on models accurately recalling long-tail facts, pursuing loss parity alone may not be enough.

NVFP4 is a 4-bit floating-point quantization format proposed by NVIDIA, a variant within the Microscaling (MX) format family. Unlike conventional FP4, NVFP4 uses block-wise quantization: every 16 elements share an FP8 (E4M3) scaling factor, while elements within a block are represented in E2M1 (1 sign bit, 2 exponent bits, 1 mantissa bit). This two-level scaling structure allows NVFP4 to cover a wide dynamic range while maintaining an extremely low storage bit width, making it a candidate format that balances efficiency and precision in both training and inference. By contrast, BF16 uses an 8-bit exponent and a 7-bit mantissa, with a dynamic range comparable to FP32, and is the mainstream precision for current large-model pretraining. Understanding the representational differences between the two helps grasp the experimental implications of the comparison in this article: NVFP4 has very few mantissa bits, so the rounding error of a single matrix multiplication is significantly larger than that of BF16, but block-wise scaling still allows it to maintain an unbiased gradient estimate in a statistical sense—this is precisely the root cause of why validation loss can match while rare-fact learning efficiency diverges.

Sources of Gradient Noise and Their Mechanisms of Impact

The study further breaks down the pathways through which quantization error acts. Zero-mean gradient noise can reproduce the cost of quantizing only the backward pass, while quantizing only the backward pass has a negligible effect on loss. In a controlled experiment where NVFP4 error is doubled, noise injected only into Adam's second-moment estimate behaves like a reduced learning rate, and its cost fades gradually over training; noise injected into the update instead lowers the signal-to-noise ratio and carries about two-thirds of the cost, persisting until the end of training.

Raising the learning rate to compensate for the smaller average step size fails to recover the cost, whether for this type of noise or for NVFP4 itself. This indicates that simply adjusting the learning rate cannot offset the rare-fact learning delay caused by quantization.

The Adam optimizer's second-moment estimate is an exponential moving average of squared gradients, denoted v_t, used to normalize the adaptive learning rate for each parameter. When backpropagation is quantized, zero-mean noise introduced into the gradients enters both the first-moment (momentum) and second-moment estimates. The key difference is that noise in the second-moment estimate systematically inflates v_t, suppressing the effective learning rate—manifesting as a "shrunk learning rate" effect—whereas noise injected into parameter updates directly lowers the signal-to-noise ratio of the update signal, drowning out the gradient direction on scarce samples. This explains the different decay behaviors of the two noise costs described in the article: the former fades as training progresses and v_t gradually stabilizes, while the latter persists throughout training because it continuously affects the direction of each update. Signal-to-noise ratio (SNR) here refers to the ratio between the true gradient signal and the variance of quantization noise, and is a core quantity for measuring the impact of low-precision training on learning dynamics.

Predicting Quantization Cost from Gradient Statistics

The study also proposes a fitting-free prediction method: gradient statistics from BF16 checkpoints can rank the costs of different formats, partial quantization, and noise control, achieving a Spearman correlation coefficient of 0.96. This means that by analyzing gradient statistics, one can predict the impact of a quantization recipe on rare-fact learning without retraining.

The authors recommend that when evaluating low-precision training recipes, rare-fact acquisition should be measured alongside loss. This view has cautionary significance for large-model pretraining practices that pursue extreme efficiency: when the loss curve appears normal, the model's grasp of rare knowledge may already be quietly lagging.

Implications for Low-Precision Training Practice

As low-precision formats such as FP4 gain attention for accelerating training and reducing memory, this study reminds practitioners to pay attention to the differentiated impact of quantization on knowledge acquisition. As a single metric, validation loss may be insufficient to reflect a model's performance on long-tail facts. Future evaluation of low-precision recipes may need to introduce recall tests for rare facts or use gradient statistics for early diagnosis.

The research team's methodology—synthetic-data experiments with controlled exposure counts and gradient noise analysis—provides a reusable framework for understanding how quantization error affects learning dynamics. Although the experiments are based on simulated 4-bit matrix multiplication, their conclusions have reference value for real FP4 hardware deployments, especially in scenarios that require models to accurately recall specific facts.

#FP4 pretraining#NVFP4#quantization#rare-fact learning#low-precision training#gradient noise
Share:

Related articles