Nesterov Momentum
An improved momentum optimization method that first performs a look-ahead update in the momentum direction and then computes the gradient, allowing faster response to changes in gradient direction and reducing oscillation.
Timeline (last 90 days)
The video builds the optimizer step by step in the order of plain GD, Nesterov, AdaGrad, RMSProp, and Adam.
Nesterov momentum introduces a momentum term to accelerate convergence and reduce oscillation, while adjusting direction by "looking one step ahead."
The difference between Nesterov momentum and classical momentum lies in where the gradient is computed: classical momentum computes the gradient at the current parameters, whereas Nesterov momentum first performs a lookahead update of the parameters based on the momentum term and computes the gradient at the predicted position.
Nesterov momentum can achieve a stronger convergence rate in convex optimization.
All Facts (4)
The video builds the optimizer step by step in the order of plain GD, Nesterov, AdaGrad, RMSProp, and Adam.
50%UnverifiedNesterov momentum introduces a momentum term to accelerate convergence and reduce oscillation, while adjusting direction by "looking one step ahead."
50%UnverifiedThe difference between Nesterov momentum and classical momentum lies in where the gradient is computed: classical momentum computes the gradient at the current parameters, whereas Nesterov momentum first performs a lookahead update of the parameters based on the momentum term and computes the gradient at the predicted position.
50%UnverifiedNesterov momentum can achieve a stronger convergence rate in convex optimization.
50%