Mixed Precision
A numerical strategy that uses FP16/BF16 and FP32 at different stages of computation, with throughput-sensitive operators run at low precision while accumulation and reduction are kept in FP32, balancing performance and numerical stability.
Timeline (last 90 days)
In MLIP, descriptor construction and final energy summation often require higher precision, while the matrix multiplications of intermediate hidden layers can be executed in mixed precision.
Typical scenarios of abstraction leakage include distributed training getting stuck at all-reduce, mixed precision causing gradient overflow, and custom operators not being supported.
The paper "The Price of Precision" proposes a calibrated cost model that first determines the true bottleneck under a given batch size and context length, and then uses dynamic programming to select a mixed-precision configuration, rather than relying solely on byte width.
All Facts (3)
In MLIP, descriptor construction and final energy summation often require higher precision, while the matrix multiplications of intermediate hidden layers can be executed in mixed precision.
50%UnverifiedTypical scenarios of abstraction leakage include distributed training getting stuck at all-reduce, mixed precision causing gradient overflow, and custom operators not being supported.
50%UnverifiedThe paper "The Price of Precision" proposes a calibrated cost model that first determines the true bottleneck under a given batch size and context length, and then uses dynamic programming to select a mixed-precision configuration, rather than relying solely on byte width.
50%