Statistical Foundations of Machine Learning: From MLE to EWMA and Deep Learning Optimizer Principles

Why Statistics Is the Cornerstone of Machine Learning

Among the many mathematical foundations of machine learning, statistics is often the most overlooked by beginners—yet it is arguably the most essential. Recently, a content creator shared the second part of their Statistics for Machine Learning tutorial series on Reddit. Spanning approximately two hours, the tutorial uses whiteboard hand-written derivations to build up core statistical concepts in AI/ML from scratch.

The value of this tutorial lies in the fact that it goes beyond simply stacking formulas. Through detailed mathematical derivations and visual explanations, it helps learners truly understand the principles behind algorithms. For practitioners who want to deeply understand how models work rather than just call APIs, this type of content is especially valuable.

Maximum Likelihood Estimation (MLE): Derivation from Univariate to Multivariate Gaussian

One of the core topics in this installment is the complete derivation of Maximum Likelihood Estimation (MLE). MLE is the most fundamental and important method in parameter estimation, answering a basic question: given observed data, what model parameters are most likely to have produced that data?

Historically, MLE was systematically introduced by the British statistician R.A. Fisher in the 1920s and is the central method of frequentist statistical inference. The basic idea is to treat observed data as fixed and parameters as unknowns, then find the optimal parameters by maximizing the likelihood function. MLE possesses several excellent asymptotic properties—as sample size approaches infinity, the MLE estimator is consistent (converges to the true parameter), asymptotically normal, and asymptotically efficient (achieving the Cramér-Rao lower bound).

It's worth explaining the important concept of the Cramér-Rao Lower Bound (CRLB). The CRLB is a fundamental result in statistical estimation theory that provides a lower bound on the variance of any unbiased estimator. Specifically, for any unbiased estimator of parameter θ, its variance is no less than the inverse of the Fisher information: Var(θ̂) ≥ 1/I(θ). Fisher information I(θ) measures how much information the data carries about the parameter—intuitively, the "sharper" the likelihood function is around the optimal parameter, the larger the Fisher information, meaning the data provides more information about the parameter and the upper limit of estimation precision is higher. When an estimator's variance exactly equals the CRLB, it is called an efficient estimator. The fact that MLE achieves this lower bound for large samples means it is asymptotically optimal—no other unbiased estimator can do better in terms of variance.

These properties make MLE widely adopted in machine learning. From logistic regression to the design of training objective functions in neural networks, almost everything can be traced back to the MLE framework.

The tutorial starts with the univariate Gaussian distribution, deriving the maximum likelihood estimates of its mean and variance. This process is relatively intuitive: by taking the log of the likelihood function, differentiating, and setting the derivative to zero, we obtain the familiar sample mean and sample variance formulas.

It's worth noting that the Gaussian distribution (normal distribution) holds its central position in statistics and machine learning not only because the Central Limit Theorem guarantees that the sum of many independent random variables approaches a normal distribution, but also because it is the maximum entropy distribution—under given mean and variance constraints, the normal distribution is the least informative (most uncertain) distribution. Therefore, when we only know the mean and variance of data and lack other information, choosing the Gaussian distribution as a modeling assumption is the most conservative and reasonable choice. This provides information-theoretic support for its widespread use in machine learning.

Visual Understanding of the Multivariate Gaussian Distribution

The real challenge lies in the MLE derivation for the multivariate Gaussian distribution. Here the tutorial introduces two key concepts—the Scatter Matrix and the Centering Matrix—and uses visualization to aid understanding.

In the multivariate case, estimating the covariance matrix involves matrix differentiation, which is a hurdle for many learners. This requires matrix calculus, the generalization of scalar calculus to matrix spaces. Key tools include: differentiating the trace of a matrix, the log-derivative of a determinant (∂log|A|/∂A = A⁻ᵀ), and using the cyclic permutation property of the trace to simplify derivations. These techniques are ubiquitous in machine learning—from PCA derivations to the backpropagation algorithm in neural networks—all rely on basic rules of matrix differentiation. Mastering these tools is a prerequisite for understanding multivariate statistical derivations.

The scatter matrix is defined as S = Σ(xi - x̄)(xi - x̄)ᵀ. It is the unnormalized version of the covariance matrix and plays a central role in Principal Component Analysis (PCA) and Fisher Linear Discriminant Analysis (LDA) as well. The centering matrix H = I - (1/n)11ᵀ is an idempotent matrix (i.e., H² = H) whose function is to project any data matrix X onto a zero-mean subspace. Using the centering matrix, the scatter matrix can be compactly expressed as S = XᵀHX. This matrix representation not only simplifies derivations but also reveals the geometric essence of data centering—it is an orthogonal projection. Visualizing these concepts makes abstract matrix operations intuitive and tangible.

From Statistics to Linear Regression: The Equivalence of MLE and Least Squares

The tutorial then applies MLE to linear regression and derives the Residual Sum of Squares (RSS). This section reveals a profound connection: under the assumption that errors follow a Gaussian distribution, the maximum likelihood estimate for linear regression is mathematically equivalent to the least squares method.

This equivalence depends on a key assumption: observation noise follows a zero-mean Gaussian distribution. If we change the noise assumption, the loss function changes accordingly. For example, assuming noise follows a Laplace distribution, MLE leads to L1 loss (absolute value loss), and the corresponding regression method is called Least Absolute Deviations (LAD) regression, which is more robust to outliers. Assuming noise follows a Bernoulli distribution (in classification problems), MLE leads to cross-entropy loss. This mapping from "probabilistic assumption → loss function" is the fundamental paradigm of Bayesian machine learning and probabilistic graphical model design.

This statistical perspective is far more profound than simply memorizing least squares formulas. It tells us that many loss functions we use daily actually originate from probabilistic assumptions about the data-generating process. Understanding this makes it clear why different noise assumptions call for different loss functions.

Empirical Risk Minimization and Surrogate Loss Functions

Building on this foundation, the tutorial further introduces the concepts of Empirical Risk Minimization (ERM) and Surrogate Loss Functions.

ERM is the core framework of statistical learning theory, systematically established by Vladimir Vapnik. The true risk (expected risk) R(f) = E[L(f(x), y)] measures a model's performance over the entire data distribution, but since the true distribution is unknown, we can only approximate it with the empirical risk on training data: R̂(f) = (1/n)ΣL(f(xi), yi). The core challenge of ERM is generalization: minimizing empirical risk does not equal minimizing true risk—overfitting is precisely the manifestation of too large a gap between the two.

To quantify generalization ability, statistical learning theory has developed various complexity measures. Among them, the VC Dimension (Vapnik-Chervonenkis Dimension) is the most classic, measuring the expressive power of a model's hypothesis space. It is defined as the maximum number of samples that a model can perfectly classify ("shatter"). For example, a two-dimensional linear classifier has a VC dimension of 3, meaning it can shatter at most 3 points but cannot shatter all label combinations of any 4 points. VC theory provides an upper bound on generalization error, showing that true risk does not exceed empirical risk plus a complexity penalty term that is positively correlated with VC dimension and negatively correlated with sample size. This theory mathematically explains why overly complex models (high VC dimension) lead to overfitting and provides a theoretical foundation for regularization methods. More modern tools like Rademacher complexity further refine these generalization bounds, while regularization methods (such as L1/L2 penalties) narrow the gap between empirical and true risk by constraining hypothesis space complexity.

However, in practice, the loss we truly care about (such as 0-1 loss in classification) is often difficult to optimize—it is non-convex and discontinuous, making direct optimization nearly impossible. Therefore, differentiable, convex surrogate loss functions are used instead. Surrogate loss functions must satisfy several key conditions: first, computational feasibility (usually requiring convexity and differentiability); second, Fisher consistency (meaning the optimal solution of the surrogate loss is also the optimal solution of the 0-1 loss). Common surrogate losses include logistic loss (logistic regression), hinge loss (SVM), and exponential loss (AdaBoost). The classic work of Zhang (2004) and Bartlett et al. (2006) systematically studied consistency conditions for surrogate losses, providing theoretical guidance for loss function design. This idea permeates the design of virtually all supervised learning algorithms.

Method of Moments: A More Computationally Convenient Path to Parameter Estimation

Beyond MLE, the tutorial also introduces the Method of Moments. This is a computationally simpler parameter estimation method, proposed by Karl Pearson in 1894 and one of the earliest systematic parameter estimation methods in history. Its core idea is to establish equations between population moments (such as E[X], E[X²]) and parameters, then solve by replacing population moments with sample moments. For a model with k parameters, the first k moments are typically needed to set up k equations.

The main advantage of the method of moments is computational simplicity, and it doesn't require knowing the complete distributional form of the data. The Generalized Method of Moments (GMM) is its modern extension, proposed by Lars Peter Hansen in 1982 (for which he received the 2013 Nobel Prize in Economics). It is widely applied in econometrics and obtains optimal estimates by minimizing weighted moment conditions when there are more moment conditions than parameters.

Commendably, the tutorial does not shy away from the limitations of the method of moments. Compared to MLE, while the method of moments is computationally simpler, it is generally less statistically efficient—meaning that for the same sample size, moment estimators have larger variance, and in some cases may yield unreasonable estimates (e.g., values outside the parameter's valid range). Honestly discussing a method's limitations is precisely what distinguishes quality educational content from marketing-driven tutorials.

Exponentially-Weighted Moving Average (EWMA): The Key to Understanding Deep Learning Optimizers Like Adam

A major highlight of this installment is the in-depth explanation of the Exponentially-Weighted Moving Average (EWMA). While seemingly simple, this concept is the key foundation for understanding modern deep learning optimizers such as Momentum, RMSprop, and Adam.

The EWMA recursive formula is vt = βvt-1 + (1-β)θt, where β is the decay factor (typically set to 0.9 or 0.99) and θt is the current observation. The larger β is, the longer the "memory window"—the effective window is approximately 1/(1-β) time steps. For example, with β=0.9, the effective window is about 10 steps; with β=0.99, it's about 100 steps.

The tutorial explains in detail why bias arises in EWMA and how memory affects the average. In the Adam (Adaptive Moment Estimation) optimizer, the first moment estimate mt tracks the mean direction of gradients (similar to Momentum), while the second moment estimate vt tracks the mean of squared gradients (similar to the adaptive learning rate in RMSprop). Adam was proposed by Kingma and Ba in 2015, with the parameter update formula θt+1 = θt - α·m̂t/(√v̂t + ε), where α is the learning rate and ε is a small constant to prevent division by zero (typically 10⁻⁸). The intuition behind this update formula is: m̂t in the numerator provides gradient direction and smooths noise (momentum effect), while √v̂t in the denominator adaptively scales the learning rate for each parameter dimension—dimensions with large gradient fluctuations automatically get a reduced learning rate, while dimensions with stable gradients maintain a larger learning rate.

At the beginning of training, due to the initialization m0=v0=0, the moving averages in the first few steps are severely biased toward zero. The bias correction formulas m̂t = mt/(1-β₁ᵗ) and v̂t = vt/(1-β₂ᵗ) are designed to eliminate this initialization bias. This correction term has a significant effect when t is small and naturally approaches 1 as t grows—this is the fundamental reason why optimizers like Adam need to introduce bias correction terms.

Adam's default hyperparameters β₁=0.9, β₂=0.999, ε=10⁻⁸ perform well in most scenarios, which is an important reason why it has become the default optimizer in deep learning. Subsequent variants such as AdamW (which decouples weight decay from L2 regularization, proposed by Loshchilov and Hutter in 2019) and LAMB (optimized for large-batch training) have further extended its applicability. Optimizers widely used in recent large language model training, such as AdaFactor and LION, can also be traced back to the fundamental statistical concept of EWMA.

Connecting EWMA from statistics with deep learning optimizers is the most insightful aspect of this tutorial series. It helps learners realize that seemingly isolated deep learning techniques actually share a unified statistical foundation.

Whiteboard Hand-Written Derivations: Returning to the Essence of Mathematical Learning

The creator specifically emphasizes that the entire content is presented through whiteboard derivations from scratch. In an era where video tutorials often rely on polished slides and rapid editing, whiteboard hand-writing is a deliberately "against-the-current" choice.

The value of this approach is that it preserves the authentic thinking process—how each step of the derivation follows from the previous one, and what each symbol means, are all clearly shown. For math-intensive content, this "slow pace" is precisely what helps learners follow the derivation logic rather than being overwhelmed by conclusions.

Conclusion

This approximately two-hour statistics tutorial covers a complete knowledge chain from maximum likelihood estimation, linear regression, and empirical risk minimization to the method of moments and exponentially-weighted moving averages. Its uniqueness lies not only in the systematic coverage of content but also in establishing clear connections between statistical concepts and deep learning practice.

For learners who want to solidify their mathematical foundations in machine learning—especially practitioners who aren't satisfied with just knowing "what" but want to understand "why"—this kind of rigorous, free learning resource is worth paying attention to. Only by understanding these statistical foundations can one develop genuine judgment when facing new algorithms.