Mixed Precision Training
A deep learning training optimization technique that uses half-precision floating point (FP16) during forward propagation and gradient computation while retaining full-precision (FP32) master weight copies for parameter updates. This approach significantly reduces GPU memory consumption and accelerates training throughput.
Timeline (last 90 days)
Mixed-precision training uses both low-precision (FP16, BF16 or FP8) and high-precision (such as FP32) numerical formats during training, using low precision for matrix operations to save memory and bandwidth, while retaining high precision for sensitive steps such as parameter updates to avoid overflow.
Pretraining's footprint on VRAM (or NPU on-chip memory) is usually several times that of inference, and it is extremely sensitive to numerical precision; a single overflow or underflow during mixed-precision training can cause the loss to diverge.
Different hardware architectures, different CUDA versions, and different operator scheduling orders can all cause subtle differences in floating-point rounding results.
In production environments, engineers need to handle complex issues such as mixed-precision training (FP16/BF16) and distributed parallelism strategies (data parallelism, pipeline parallelism, and tensor parallelism).
Gradient checkpointing and mixed-precision training (FP16/BF16) are common memory-reduction measures.
Modern training frameworks use mixed-precision training and gradient scaling (loss scaling) techniques to prevent small values from being swallowed by amplifying small gradients into the representable range.
Mixed-precision training with the Adam optimizer requires approximately 14 bytes per parameter, meaning full-parameter training of a 70-billion-parameter model may require over 1TB of memory.
在有限硬件条件下可使用混合精度训练、梯度累积等技巧来最大化资源利用效率
All Facts (9)
混合精度训练在不显著损失模型精度的前提下，可将训练速度提升约2倍，显存占用降低近一半
75%UnverifiedMixed-precision training uses both low-precision (FP16, BF16 or FP8) and high-precision (such as FP32) numerical formats during training, using low precision for matrix operations to save memory and bandwidth, while retaining high precision for sensitive steps such as parameter updates to avoid overflow.
50%UnverifiedPretraining's footprint on VRAM (or NPU on-chip memory) is usually several times that of inference, and it is extremely sensitive to numerical precision; a single overflow or underflow during mixed-precision training can cause the loss to diverge.
50%UnverifiedDifferent hardware architectures, different CUDA versions, and different operator scheduling orders can all cause subtle differences in floating-point rounding results.
50%UnverifiedIn production environments, engineers need to handle complex issues such as mixed-precision training (FP16/BF16) and distributed parallelism strategies (data parallelism, pipeline parallelism, and tensor parallelism).
50%UnverifiedGradient checkpointing and mixed-precision training (FP16/BF16) are common memory-reduction measures.
50%UnverifiedModern training frameworks use mixed-precision training and gradient scaling (loss scaling) techniques to prevent small values from being swallowed by amplifying small gradients into the representable range.
50%UnverifiedMixed-precision training with the Adam optimizer requires approximately 14 bytes per parameter, meaning full-parameter training of a 70-billion-parameter model may require over 1TB of memory.
50%Unverified在有限硬件条件下可使用混合精度训练、梯度累积等技巧来最大化资源利用效率
50%