Natural Gradient Descent
An optimization method proposed by Amari in 1998. Its core idea is to precondition the gradient with the Fisher information matrix in parameter space, making the update direction independent of arbitrary reparameterizations of the parameters. Updates proceed along the direction of steepest descent on the Riemannian manifold, endowing the method with parameterization invariance.
Timeline (last 90 days)
A study published on arXiv (arXiv:2610.00004v1) decomposes Adam's complete update rule (including the momentum term) into a diagonal empirical Fisher approximation.
Natural Gradient Descent (NGD) was proposed by Amari in 1998.
Adam can be viewed as a simplified NGD approximation that uses the exponential moving average of squared gradients to estimate the diagonal elements of the Fisher matrix.
In non-convex small neural networks, the directional misalignment between Adam and NGD reaches an order of magnitude of approximately 10^3.
All Facts (4)
A study published on arXiv (arXiv:2610.00004v1) decomposes Adam's complete update rule (including the momentum term) into a diagonal empirical Fisher approximation.
50%UnverifiedNatural Gradient Descent (NGD) was proposed by Amari in 1998.
50%UnverifiedAdam can be viewed as a simplified NGD approximation that uses the exponential moving average of squared gradients to estimate the diagonal elements of the Fisher matrix.
50%UnverifiedIn non-convex small neural networks, the directional misalignment between Adam and NGD reaches an order of magnitude of approximately 10^3.
50%