How Far Is Adam from Natural Gradient Descent, Geometrically?

A Long-Overlooked Theoretical Question

Adam has become the default optimizer for deep learning training — you can find it in nearly every mainstream model's training script. Yet a fundamental question has remained unanswered for a long time: in a geometric sense, how far is Adam from the theoretically more elegant Natural Gradient Descent (NGD)?

A new study posted on arXiv (arXiv:2610.00004v1) tackles this question head-on. The researchers decompose Adam's full update rule — momentum included — as a form of diagonal empirical Fisher approximation, and show that in practice it is constrained by three factors: diagonal truncation, empirical label substitution, and temporal lag. These three factors jointly determine how much Adam deviates from true NGD.

Natural Gradient Descent (NGD), introduced by Amari in 1998, preconditions gradients using the Fisher Information Matrix (FIM) in parameter space, making update directions invariant to arbitrary reparameterizations of the model. Intuitively, ordinary gradient descent moves along the direction of steepest descent in Euclidean space, while the natural gradient moves along the steepest descent direction in the space of probability distributions (a Riemannian manifold defined by KL divergence). This gives NGD theoretical parameterization invariance — regardless of how you re-express model parameters, the update direction remains consistent. However, computing the full FIM has complexity $O(p^2)$ (where $p$ is the number of parameters), making it completely infeasible for modern deep networks. Adam can be seen as a drastically simplified approximation to NGD: it uses the exponential moving average of squared gradients (i.e., $v_t$) to estimate the diagonal elements of the Fisher matrix, achieving a kind of "adaptive learning rate" effect at very low cost.

Measuring Deviation with a Geometric Metric

The study employs a scale-invariant metric $\gamma(\Delta\theta)$ to measure the geometric deviation of Adam's update direction relative to the true NGD direction. The emphasis on "scale-invariant" is important: since step sizes and parameter magnitudes vary drastically during optimization, only a scale-independent metric can fairly compare directional alignment across different stages and tasks.

The authors conducted systematic experiments on four types of loss surfaces:

• Well-conditioned linear regression
• Ill-conditioned linear regression
• Logistic regression
• A non-convex small neural network

This gradient design — ranging from simple to complex, from convex to non-convex — allows the study to clearly track how geometric deviation evolves as problem difficulty increases.

Deviation Is Highly Dependent on the Problem's Condition Number

The most central finding is that Adam's geometric trajectory is context-dependent — there is no single universal "deviation constant."

Under well-conditioned settings, the directional deviation between Adam and NGD is small, and Adam can be considered a good approximation to the natural gradient. But once the problem becomes ill-conditioned, the deviation rises significantly; and in the non-convex small neural network, this directional misalignment reaches a magnitude of roughly $10^3$. In other words, in the tasks most representative of real-world deep learning, Adam's path diverges substantially from the theoretically optimal natural gradient path.

This finding is somewhat counterintuitive — Adam performs excellently in practice, yet not because it closely follows the "correct direction" of the natural gradient.

Condition number is a measure of how sensitive a problem is to input perturbations, defined as the ratio of the largest to smallest eigenvalue of a matrix: $\kappa = \lambda_{\max}/\lambda_{\min}$. In the context of optimization, a higher condition number for the loss surface means greater disparity in curvature across different parameter directions — the surface is extremely steep in some directions and extremely flat in others. This "canyon-shaped" landscape is the fundamental reason why ordinary gradient descent converges slowly, and why second-order methods (like NGD) can dramatically accelerate convergence. Adam's small deviation on well-conditioned problems (low condition number) is because when curvature is similar across directions, the diagonal approximation is sufficient. But on ill-conditioned problems (high condition number), the off-diagonal elements of the Fisher matrix carry critical directional information, and the error from diagonal truncation is dramatically amplified — causing the directional deviation from true NGD to rise sharply.

Geometric Drift Does Not Equal Performance Loss

The study further reveals a key decoupling: higher geometric drift only slows down early-stage optimization, without harming final objective function minimization.

Regardless of the magnitude of directional deviation, Adam consistently converges to low loss values. This means directional "imprecision" is not fatal — Adam always manages to reach good solutions, just potentially taking a more roundabout path in the early stages. This provides empirical-level explanation for "why a method with such a crude approximation works so well."

Additionally, the study compares two Fisher approximation approaches: the improved empirical Fisher (iEF) tracks a more stable path compared to the standard empirical Fisher (EF). Standard EF frequently exhibited oscillations or even divergence in experiments, while iEF behaved more smoothly and reliably — offering a valuable reference direction for future optimizer design.

The core distinction between the Empirical Fisher (EF) and the improved Empirical Fisher (iEF) lies in the source of labels. Standard EF uses the true labels observed in training data to estimate the Fisher matrix: $\hat{F} = \mathbb{E}_{x,y\sim\mathcal{D}}[\nabla\log p(y|x)\nabla\log p(y|x)^\top]$. The theoretically correct Fisher information matrix, however, should sample labels from the model's own predictive distribution $p(y|x;\theta)$. This substitution may seem subtle, but it introduces bias in gradient estimation — especially in early training or when the model's predictions diverge significantly from the true label distribution. iEF avoids this "empirical label substitution" error by sampling labels from the model's current predictive distribution when computing the Fisher approximation, resulting in an optimization path closer to the true natural gradient trajectory, and exhibiting lower oscillation and more stable convergence in experiments.

Where Does Adam's Real Power Come From?

Synthesizing the experimental evidence, the study offers a thought-provoking conclusion: Adam's practical optimization capability may stem from a certain balance between structural approximation error and momentum smoothing, rather than from accurately tracking the natural gradient path.

This reframes our understanding of Adam. For a long time, people have tended to explain Adam's effectiveness by saying "Adam approximates second-order information." But this study suggests that the smoothing effect of momentum and the error introduced by the diagonal approximation mutually cancel and harmonize, together constituting Adam's robustness. It looks more like an empirically refined engineering trade-off than a faithful replication of the theoretically optimal direction.

Implications for Practitioners

For engineers and researchers who use Adam daily, this paper offers at least three actionable insights:

1. Don't assume Adam's update direction is geometrically accurate. On ill-conditioned or highly non-convex problems, its deviation from the natural gradient may be far larger than you'd expect — but this doesn't prevent eventual convergence.
2. Slow early convergence may be a signal of geometric drift. If training progress is sluggish in the early stages, directional deviation could be the underlying cause, rather than a hyperparameter tuning issue.
3. Improved empirical Fisher (iEF) is worth watching. For researchers attempting to design new second-order or natural gradient methods, iEF provides a more stable path foundation than standard EF.

This work doesn't invalidate Adam — quite the opposite. It uses rigorous geometric analysis to explain why Adam can remain highly effective despite its imprecision, and points toward possible breakthroughs in the design of next-generation optimizers.