Conceptpreconditioner
Preconditioner
In adaptive optimization algorithms, a diagonal preconditioner formed from the inverse square root of the second moment, which scales the gradient step size for each coordinate.
Timeline (last 90 days)
Oct 13
In AdamW, the parameter update is approximately equal to the first moment divided by (the square root of the second moment plus ε), and the reciprocal of the square root of the second moment acts as a preconditioner, adaptively scaling the gradient for each coordinate.
Unverified50%