ConceptLow-Precision Training
Low-Precision Training
A technique for training neural networks using low-precision numerical formats such as FP16 and INT8. It relies on methods such as loss scaling, FP32 master weight copies, or stochastic rounding to maintain gradient numerical stability.
Timeline (last 90 days)
Oct 14
In backpropagation, gradient values are often extremely small, and under low precision they easily underflow or are drowned out by rounding noise, requiring techniques such as loss scaling, an FP32 master weight copy or stochastic rounding to maintain numerical stability.
Unverified50%