Stochastic Dynamics of SGDM: A New Framework via Nonlinear Time Series

Stochastic Gradient Descent with Momentum (SGDM) is a workhorse algorithm in deep learning optimization, yet its theoretical analysis has long lagged behind practice. Most convergence studies treat momentum as an auxiliary variable or simplify it away entirely, ignoring the geometrically decaying memory carried by the momentum state itself. A recent theoretical work reformulates SGDM's coupled recursion as a nonlinear time series, providing a unified framework for understanding its stationary behavior and distributional limits.

From Optimization Algorithm to Time Series: A Shift in Perspective

Traditional SGD analysis typically focuses on the convergence rate of parameters, with momentum discussed only as an acceleration trick. This approach overlooks a key fact: the momentum state $m_t$ is itself a recursive variable that combines the previous momentum with the current stochastic gradient in a weighted manner, forming a geometrically weighted memory of historical gradients. The parameter $\theta_t$ and momentum $m_t$ constitute a coupled dynamical system whose behavior cannot be reduced to a stochastic approximation problem of a single parameter.

The core insight of the paper is to treat the joint process $(\theta_t, m_t)$ as a nonlinear time series. This shift in perspective brings two immediate benefits. It allows the use of mature tools from time series analysis—stationarity, causality, mixing—to characterize the algorithm's long-term behavior; meanwhile, nonlinear terms (arising from the curvature of the loss function or the stochasticity of gradients) are no longer treated as noise but incorporated into the model structure for analysis.

This treatment contrasts with classical stochastic approximation theory. Classical methods rely on the Robbins-Monro conditions to guarantee convergence, but SGDM with a constant step size does not satisfy these conditions, and the parameters will continue to fluctuate around a steady state rather than converging to a point. The time series framework is precisely suited to describing such steady-state fluctuations.

Robbins-Monro conditions are classical sufficient conditions in stochastic approximation theory that guarantee the convergence of iterations, proposed by Herbert Robbins and Sutton Monro in 1951. They require the step-size sequence to satisfy $\sum_t \eta_t = \infty$ and $\sum_t \eta_t^2 < \infty$; the former ensures the algorithm can reach any neighborhood, and the latter ensures that noise accumulation does not diverge. Under a decreasing step size (e.g., $\eta_t = 1/t$), these conditions are usually satisfied, and SGD iterations converge almost surely to a stable point of the objective function. However, the constant step size strategy widely adopted in deep learning practice makes $\sum_t \eta_t^2 = \infty$, directly violating the second Robbins-Monro condition. This means that constant-step SGD (and SGDM) does not converge to a point but continues to fluctuate around a steady state with a variance proportional to the step size. Consequently, the convergence conclusions of classical stochastic approximation cannot be directly applied, and researchers must turn to an analytical framework that describes a "stationary distribution" rather than a "limit point"—this is precisely the fundamental motivation for introducing the time series perspective in this paper.

Initialization Forgetting and Unique Stationary Solution

The first main result of the paper concerns the rate of initialization forgetting. Under finite $p$-th moment conditions ($p>1$), the joint process $(\theta_t, m_t)$ forgets initial conditions at a geometric rate. This means that regardless of the initial parameters and momentum, the process will "forget" its starting point within an exponentially small number of steps and enter a unique stationary causal solution.

The significance of this result is that it provides a sufficient condition for the existence of a unique invariant distribution for constant-step SGDM. For practitioners, this means that the algorithm's performance after a long run is essentially independent of initialization—an intuition that is now rigorously proven for the first time. The forgetting rate itself is also quantifiable—it determines how many steps the algorithm needs to enter a "statistical steady state," which has direct implications for hyperparameter selection.

The mechanism of geometric forgetting stems from the contraction property of the coupled recursion. The joint effect of the momentum decay factor (commonly denoted $\beta$) and the step size determines the contraction strength. A smaller step size or faster momentum decay leads to faster forgetting, but the steady-state variance exhibits a different trade-off.

Invariance Principle and Two Types of Gaussian Limits

Under second-moment conditions, the paper further establishes an invariance principle and a quenched Gaussian limit. These are two different but complementary distributional limit results.

For averaged SGDM, the paper proves that its iteration average follows a Gaussian limit. This echoes the classical results of Polyak averaging in convex optimization, but the paper's framework does not rely on convexity assumptions; instead, it is obtained through mixing conditions of the time series.

For the last iterate, the paper employs a small learning rate analysis and also obtains a Gaussian limit. The Gaussian limit for the last iterate is typically harder to establish because the single-point distribution does not benefit from the variance reduction brought by averaging. The small learning rate analysis here serves to "freeze" nonlinear effects and approximately linearize the dynamics.

The coexistence of the two types of limits illustrates the universal behavior of SGDM under different statistics: whether you look at the averaged path or the endpoint, as long as moment conditions are satisfied, Gaussian fluctuations appear—the only difference lies in the covariance structure.

Polyak averaging, introduced by Boris Polyak in 1990, is a technique that takes the cumulative average of an iteration sequence, $\bar{\theta}t = \frac{1}{t}\sum{s=1}^t \theta_s$. Its core insight is that even if individual iterates fluctuate wildly around a steady state due to noise, their average path can still converge to the optimal solution, often at a rate superior to the last iterate. In convex optimization, the asymptotic normality of Polyak averaging has been rigorously established. This paper extends it to the non-convex, constant-step SGDM setting, proving that the averaged iterates also follow a Gaussian limit under second-moment conditions. The key difference from classical results is that the "center" here is no longer the global optimum but a steady-state mean determined jointly by the algorithm's hyperparameters (step size, momentum decay factor) and the geometry of the loss function. Furthermore, the variance reduction effect of averaging may weaken or even fail in non-convex settings, contrasting with the convex-optimization intuition that averaging always improves variance.

Trade-offs in a Stochastic-Curvature Quadratic Model

The paper conducts a more detailed analysis in a stochastic-curvature quadratic model. This model is simple enough to allow analytical computation while retaining the core nonlinear characteristics of SGDM dynamics.

In this model, the paper studies the relationship among the initialization forgetting rate, steady-state variance, and averaged fluctuations. The results show a substantial trade-off between the forgetting rate and steady-state variance: faster forgetting is typically accompanied by larger steady-state variance, and vice versa. This trade-off has direct implications for algorithm tuning—pursuing fast "forgetting of bad initializations" may come at the cost of larger long-term fluctuations.

The behavior of averaged fluctuations is more subtle. It is influenced by both the forgetting rate and the distribution of stochastic curvature. In certain parameter regimes, averaging does not always reduce fluctuations, which differs from the intuition in convex optimization. The nonlinear time series framework makes these previously difficult-to-characterize interactions amenable to analysis.

The stochastic-curvature quadratic model is a commonly used "minimal viable model" in optimization theory, typically of the form $f(\theta) = \frac{1}{2} H \theta^2$, where the curvature $H$ itself is random, simulating the stochasticity and non-uniformity of the Hessian matrix of the loss function in deep learning. Compared to general non-convex losses, the quadratic model allows exact analytical recursions of the iteration process, yielding explicit expressions for quantities such as the forgetting rate and steady-state variance; meanwhile, the randomness of curvature retains the core nonlinear characteristics of SGDM dynamics. This allows researchers to quantitatively observe trade-offs among hyperparameters in a controlled environment. This paper uses this model to discover that there exists a trade-off akin to an "uncertainty principle" between the initialization forgetting rate and steady-state variance—accelerating forgetting necessarily amplifies long-term fluctuations. Such conclusions are difficult to verify separately in pure numerical experiments, but can be precisely characterized in the quadratic model.

Implications for Theory and Open Questions

The contribution of this work is to provide a unified framework that incorporates the stationarity, mixing, and distributional limits of SGDM into a single time series language. It does not aim to provide tighter convergence rates but to answer the more fundamental question of "what constant-step SGDM looks like in a statistical steady state."

The limitations are also clear. The analysis relies on moment conditions ($p>1$ or second moment), and heavy-tailed gradient distributions in deep learning may not satisfy these conditions. Whether the conclusions of the quadratic model can be extended to non-quadratic losses remains to be tested. Furthermore, the paper deals with a constant step size, while learning rate schedules commonly used in practice (e.g., cosine annealing) are outside the current framework.

From a methodological perspective, the nonlinear time series perspective may inspire more work: incorporating the coupled dynamics of other adaptive optimizers (such as Adam, RMSProp) into similar analyses, or studying the interactions between momentum and components like batch normalization. These directions all point to a common goal—making optimization theory more aligned with the complexity of real algorithms.

One-line Summary

Modeling the coupled recursion of constant-step SGDM as a nonlinear time series, proving initialization forgetting, invariance principle, and Gaussian limits, and revealing quantitative trade-offs among hyperparameters.

Paragraph Summary

This paper proposes a nonlinear time series framework to analyze stochastic gradient descent with momentum (SGDM). Traditional convergence studies treat momentum as an auxiliary variable or simplify it away, ignoring the geometrically decaying memory of the momentum state. The paper views the joint process of parameters and momentum as a coupled recursion and uses tools of stationarity, causality, and mixing to study its long-term behavior. Under finite p-th moment conditions (p>1), it proves that initialization forgetting occurs at a geometric rate and that a unique stationary causal solution exists, thereby providing a sufficient condition for a unique invariant distribution of constant-step SGDM. Under second-moment conditions, it establishes an invariance principle: averaged SGDM follows a quenched Gaussian limit, and a small learning rate analysis yields a Gaussian limit for the last iterate. A stochastic-curvature quadratic model further reveals quantitative trade-offs among the forgetting rate, steady-state variance, and averaged fluctuations, and the framework does not rely on convexity assumptions.