Conceptwarmup / 学习率预热
Learning Rate Warmup
A strategy that gradually raises the learning rate from a very small value to a peak during the early stage of training and then decays it, used to avoid excessively large gradient updates at the start of training that cause loss explosion, thereby improving the training stability of Transformer-type models.
Timeline (last 90 days)
Oct 20
Learning rate warmup gradually raises the learning rate from a very small value to a peak early in training and then decays it, which can significantly improve the training stability of Transformer-type models.
Unverified50%