Format-Aware Fusion: The Engineering Practice That Doubles FP4 Pretraining Throughput

FP4 (four-bit floating-point) Tensor Cores promise compelling speedups for matrix multiplication, but in practice, the overhead from scaling factor computation, operand packing, layout construction, and backward-pass state preservation often erases every gain the hardware delivers. A newly published arXiv paper (arXiv:2610.00053) introduces a method called format-aware fusion that fundamentally redesigns the quantization pipeline at the engineering level, aiming to translate FP4's theoretical advantages into real end-to-end training throughput.

Why FP4 Speedups Often Remain Theoretical

FP4 Tensor Cores can execute matrix multiplications at extremely low bit-widths, which in theory dramatically increases compute density. But in large-scale model training, the auxiliary operations surrounding matrix multiplication become the bottleneck. Quantization is not free — you need to compute scaling factors, pack operands according to hardware requirements, construct specific memory layouts, and save intermediate states for backpropagation.

If these steps are poorly designed, their combined time and bandwidth cost can completely negate FP4's intended speedup. The paper's central argument is precisely this: FP4's success or failure is not determined by numerical precision alone — it is a systemic engineering problem jointly governed by the scaling contract, operands, and execution path.

FP4 (4-bit floating-point) Tensor Cores are a feature introduced with NVIDIA's Blackwell architecture (GB200/B100 series). Unlike INT4, FP4 retains an exponent bit, allowing it to more naturally represent the dynamic range distribution of neural network weights and activations. However, with only 4 bits — 1 sign, 2 exponent, 1 mantissa (E2M1 format), or 1 sign, 1 exponent, 2 mantissa (E1M2 format) — the representable range is extremely limited. This means scaling factors must be computed precisely before quantization, establishing a "scaling contract" that maps original bfloat16/float32 values into FP4's representable range; any error leads directly to overflow or precision loss. The two dominant scaling schemes are NVIDIA's nvfp (block-level scaling, where a block of values shares one scaling factor) and the MX standard's mxfp (finer-grained microscaling, jointly defined by Intel, AMD, Microsoft, and others). They differ significantly in scaling granularity and memory overhead, each presenting distinct engineering trade-offs.

The Design Philosophy of Format-Aware Fusion

The method proposed in the paper co-designs the quantization pipeline for different formats. It binds each quantization producer together with its corresponding scale domain and consumer layout, adapting to three distinct scenarios: native mxfp, global nvfp, and cooperative thread array (CTA)-local nvfp.

This co-design philosophy means quantization is no longer treated as an isolated preprocessing step. Instead, where scaling factors are computed, at what granularity, and in what layout data is handed off to Tensor Cores are all considered within a unified design — eliminating the hidden costs of format conversion and layout rearrangement.

Two Key Numerical Techniques

On the mxfp path, the authors introduce two noteworthy numerical processing techniques:

• Row-gradient stochastic rounding: Applying stochastic rounding (rather than deterministic rounding) during gradient quantization helps mitigate bias accumulation caused by low precision.
• Fixed-sign 32-value Hadamard weight-gradient preconditioning: Applying a Hadamard transform to weight gradients to improve their numerical distribution at low bit-widths.

These two techniques are essential for keeping FP4 training numerically acceptable.

Stochastic Rounding is a technique for mitigating systematic bias in low-precision computation. Traditional round-to-nearest introduces biased truncation errors that accumulate during gradient updates — especially problematic at low precision, as long-term accumulation can shift the model's convergence direction. Stochastic rounding instead rounds up or down with probability proportional to the magnitude of the error, maintaining statistical unbiasedness at the cost of controlled random noise. Hadamard preconditioning comes from signal processing: the Hadamard transform is an orthogonal transform that projects data onto a set of orthogonal bases, producing more uniformly distributed coefficients with smaller peak-to-peak ratios. For FP4 quantization, raw weight gradients often exhibit sharp, heavy-tailed distributions where extreme values "waste" the precious dynamic range. After the Hadamard transform, numerical energy is spread more evenly, reducing overall quantization error. "Fixed-sign 32-value" refers to using a 32×32 Hadamard matrix with fixed signs, avoiding the overhead of sign randomization and enabling this preprocessing step to be efficiently fused into the kernel.

Empirical Results: Throughput Doubled, Precision Controlled

The paper evaluates pretraining on the Llama-3 8B model at 160 billion tokens, with the output projection layer in bfloat16 and using compiled cross-entropy. The results from controlled experiments on the same accelerator are striking:

Configuration	Throughput (tokens/s/GPU)	Relative Training Loss
bfloat16 baseline	18.8K	—
Transformer Engine nvfp	27.6K	—
Fastest custom path	37.9K	—
mxfp (stochastic rounding + Hadamard)	37.2K (86.3% MFU)	+2.11% above bfloat16 endpoint
TE recipe (last 4 bfloat16 blocks)	27.1K	+0.87% above bfloat16 endpoint

The fastest custom path pushes throughput from the bfloat16 baseline of 18.8K all the way to 37.9K — nearly doubling it. The mxfp configuration reaches 37.2K tokens/s/GPU, corresponding to 86.3% bfloat16 Model FLOP Utilization (MFU), which is a remarkably high utilization figure, while the training loss endpoint is only 2.11% higher than pure bfloat16.

By contrast, the Transformer Engine recipe — which retains four bfloat16 blocks at the end — achieves a training loss only 0.87% above bfloat16, but throughput stalls at 27.1K. This illustrates the speed-precision trade-off at play.

An Easy-to-Overlook Finding: Downstream Rankings Diverge from Training Loss Rankings

One of the most illuminating findings in the paper is that rankings on downstream tasks do not match rankings by training loss. In other words, the configuration with lower training loss does not necessarily perform better on downstream evaluations.

This observation warns practitioners against judging low-precision schemes solely by their training loss curves. FP4's ultimate effectiveness is jointly shaped by the scaling contract, operand format, and execution path — and single-metric evaluation can be misleading. For teams planning to deploy FP4 in production, thorough validation on real downstream tasks is essential, rather than stopping at the loss level.

The disconnect between training loss and downstream task performance is not unheard of in large model research, but it deserves particular vigilance in low-precision training settings. The fundamental reason is that training loss measures average prediction error on the training distribution, while downstream tasks (such as commonsense reasoning, code generation, and instruction following) depend on specific knowledge representations and generalization capabilities that are more sensitive to the noise patterns introduced by quantization. FP4 quantization perturbs different layers and parameters unevenly — for example, attention key-value matrices and feed-forward weights differ in their tolerance for quantization error — and this non-uniform perturbation may appear as a minor difference on the loss curve while causing substantial degradation on specific downstream capabilities. For engineering practice, this finding means that FP4 validation must be grounded in a diverse downstream evaluation benchmark; "converges to a lower loss" cannot be the sole criterion for selecting a quantization scheme.

Implications for Low-Precision Training

The value of this work goes beyond the specific speedup numbers. It shifts the conversation around FP4 training away from "can the model converge at this precision?" and back to "how should the system be engineered?" As Tensor Core support for lower bit-width formats matures, the design of quantization data flows will become a critical battleground for extracting hardware performance.

The format-aware fusion approach — co-designing quantization producers, scale domains, and consumer layouts as a unified whole — provides a clear framework for future research. For large-model teams pursuing maximum training throughput, this kind of engineering-level optimization may offer better returns than simply adding more compute.