Statistical Foundations of Machine Learning: From MLE to EWMA and Deep Learning Optimizer Principles

Why Statistics Is the Cornerstone of Machine Learning

Among the many mathematical foundations of machine learning, statistics is often the most overlooked by beginners, yet it is arguably the most essential. Recently, a content creator shared the second part of their "Statistics for Machine Learning" tutorial series on Reddit — approximately two hours long, using whiteboard hand-written derivations to build core statistical concepts in AI/ML from scratch.

The value of this tutorial lies in the fact that it doesn't merely stack formulas. Instead, through detailed mathematical derivations and visual explanations, it helps learners truly understand the principles behind algorithms. For practitioners who want to deeply understand how models work rather than simply calling APIs, this type of content is invaluable.

Maximum Likelihood Estimation (MLE): From Univariate to Multivariate Gaussian Derivation

One of the core topics in this tutorial is the complete derivation of Maximum Likelihood Estimation (MLE). MLE is the most fundamental and important method in parameter estimation, answering a basic question: given observed data, what model parameters are most likely to have generated that data?

Historically, MLE was systematically proposed by British statistician R.A. Fisher in the 1920s and is the core method of frequentist statistical inference. The basic idea is: treat the observed data as fixed, treat parameters as unknowns, and find optimal parameters by maximizing the likelihood function. MLE possesses several excellent asymptotic properties — as sample size approaches infinity, the MLE estimator is consistent (converges to the true parameter), asymptotically normal, and asymptotically efficient (achieving the Cramér-Rao lower bound).

It's worth explaining the important concept of the Cramér-Rao Lower Bound (CRLB). The CRLB is a fundamental result in statistical estimation theory that provides a lower limit on the variance of any unbiased estimator. Specifically, for any unbiased estimator of parameter θ, its variance is no less than the reciprocal of the Fisher information: Var(θ̂) ≥ 1/I(θ). Fisher information I(θ) measures how much information the data carries about the parameter — intuitively, the "sharper" the likelihood function is around the optimal parameter, the larger the Fisher information, meaning the data provides more information about the parameter, and the upper limit on estimation precision is higher. When an estimator's variance exactly equals the CRLB, it is called an efficient estimator. MLE achieves this lower bound in large samples, meaning it is asymptotically optimal — no other unbiased estimator can do better in terms of variance.

These properties make MLE widely adopted in machine learning. From logistic regression to the design of training objective functions for neural networks, nearly everything can be traced back to the MLE framework.

The tutorial begins with the univariate Gaussian distribution, deriving the maximum likelihood estimates for its mean and variance. This process is relatively intuitive — by taking the logarithm of the likelihood function, differentiating, and setting the derivative to zero, we obtain the familiar sample mean and sample variance formulas.

It's worth noting that the Gaussian distribution (normal distribution) occupies a central position in statistics and machine learning not only because the Central Limit Theorem guarantees that the sum of many independent random variables approaches a normal distribution, but also because it is the maximum entropy distribution — given constraints on mean and variance, the normal distribution is the least informative (most uncertain) distribution. Therefore, when we only know a dataset's mean and variance and lack other information, choosing the Gaussian distribution as a modeling assumption is the most conservative and reasonable choice, providing information-theoretic justification for its widespread use in machine learning.

Visual Understanding of the Multivariate Gaussian Distribution

The real challenge lies in the MLE derivation for the multivariate Gaussian distribution. Here the tutorial introduces two key concepts — the Scatter Matrix and the Centering Matrix — and uses visualization to aid understanding.

In the multivariate case, estimating the covariance matrix involves matrix differentiation, which is a hurdle for many learners. This requires matrix calculus, the generalization of scalar calculus to matrix spaces. Core tools include: differentiating the trace of a matrix, the log-derivative of the determinant (∂log|A|/∂A = A⁻ᵀ), and using the cyclic permutation property of trace to simplify derivations. These techniques are ubiquitous in machine learning — from deriving PCA to the backpropagation algorithm in neural networks — all rely on basic matrix differentiation rules. Mastering this toolkit is a prerequisite for understanding multivariate statistical derivations.

The scatter matrix is defined as S = Σ(xi - x̄)(xi - x̄)ᵀ, the unnormalized version of the covariance matrix, which also plays a central role in Principal Component Analysis (PCA) and Fisher's Linear Discriminant Analysis (LDA). The centering matrix H = I - (1/n)11ᵀ is an idempotent matrix (i.e., H² = H), whose function is to project any data matrix X onto the zero-mean subspace. Through the centering matrix, the scatter matrix can be compactly expressed as S = XᵀHX. This matrix representation not only simplifies the derivation process but also reveals the geometric essence of the data centering operation — it is an orthogonal projection. Visualizing these concepts makes abstract matrix operations intuitive and tangible.

From Statistics to Linear Regression: The Equivalence of MLE and Least Squares

The tutorial then applies MLE to linear regression, deriving the Residual Sum of Squares (RSS). This section reveals a profound connection: under the assumption that errors follow a Gaussian distribution, the maximum likelihood estimate for linear regression is mathematically equivalent to the least squares method.

This equivalence depends on a key assumption: observation noise follows a zero-mean Gaussian distribution. If the noise assumption changes, the loss function changes accordingly. For example, assuming noise follows a Laplace distribution, MLE derives L1 loss (absolute value loss), corresponding to the regression method called Least Absolute Deviations (LAD) regression, which is more robust to outliers. Assuming noise follows a Bernoulli distribution (in classification problems), MLE derives cross-entropy loss. This "probabilistic assumption → loss function" mapping is the basic paradigm of Bayesian machine learning and probabilistic graphical model design.

This statistical perspective is far more profound than merely memorizing least squares formulas. It tells us that many loss functions we commonly use actually originate from probabilistic assumptions about the data-generating process. Understanding this makes it clear why different loss functions should be chosen under different noise assumptions.

Empirical Risk Minimization and Surrogate Loss Functions

Building on this foundation, the tutorial further introduces the concepts of Empirical Risk Minimization (ERM) and Surrogate Loss Functions.

ERM is the core framework of statistical learning theory, systematically established by Vladimir Vapnik. The true risk (expected risk) R(f) = E[L(f(x), y)] measures model performance over the entire data distribution, but since the true distribution is unknown, we can only approximate it with the empirical risk on training data: R̂(f) = (1/n)ΣL(f(xi), yi). The core challenge of ERM is generalization: minimizing empirical risk does not equal minimizing true risk, and overfitting is precisely the manifestation of an excessive gap between the two.

To quantify generalization ability, statistical learning theory has developed various complexity measures. Among them, the VC Dimension (Vapnik-Chervonenkis Dimension) is the most classic, measuring the expressive power of a model's hypothesis space, defined as the maximum number of samples that the model can perfectly classify ("shatter"). For example, a two-dimensional linear classifier has a VC dimension of 3, meaning it can shatter at most 3 points but cannot shatter all label combinations of any 4 points. VC theory provides upper bounds on generalization error, showing that true risk does not exceed empirical risk plus a complexity penalty term that is positively correlated with VC dimension and negatively correlated with sample size. This theory mathematically explains why overly complex models (high VC dimension) lead to overfitting and provides a theoretical basis for regularization methods. More modern tools such as Rademacher complexity further refine these generalization bounds, while regularization methods (such as L1/L2 penalties) narrow the gap between empirical and true risk by constraining hypothesis space complexity.

However, in practice, the loss we truly care about (such as 0-1 loss in classification) is often difficult to optimize — it is non-convex and discontinuous, making direct optimization nearly impossible. Therefore, differentiable, convex surrogate loss functions are needed as replacements. Surrogate loss functions must satisfy several key conditions: first, computational tractability (usually requiring convexity and differentiability); second, Fisher consistency (meaning the optimal solution of the surrogate loss is also the optimal solution of the 0-1 loss). Common surrogate losses include: logistic loss (logistic regression), hinge loss (SVM), and exponential loss (AdaBoost). The classic works of Zhang (2004) and Bartlett et al. (2006) systematically studied consistency conditions for surrogate losses, providing theoretical guidance for loss function design. This idea permeates the design of virtually all supervised learning algorithms.

Method of Moments: A Computationally Simpler Path to Parameter Estimation

Besides MLE, the tutorial also introduces the Method of Moments. This is a computationally simpler parameter estimation method, proposed by Karl Pearson in 1894, and one of the earliest systematic parameter estimation methods in history. Its core idea is to establish equations between population moments (such as E[X], E[X²]) and parameters, then solve by substituting sample moments for population moments. For a model with k parameters, typically the first k moments are needed to establish k equations.

The main advantage of the method of moments is computational simplicity, and it doesn't require knowledge of the complete distributional form of the data. The Generalized Method of Moments (GMM) is its modern extension, proposed by Lars Peter Hansen in 1982 (for which he received the 2013 Nobel Prize in Economics), widely applied in econometrics. When the number of moment conditions exceeds the number of parameters, it obtains optimal estimates by minimizing weighted moment conditions.

Commendably, the tutorial does not shy away from the limitations of the method of moments. Compared to MLE, while computationally simpler, the method of moments is typically less statistically efficient — with the same sample size, moment estimators have larger variance, and in some cases may give unreasonable estimates (e.g., values outside the parameter range). Honestly discussing the limitations of a method is precisely what distinguishes quality educational content from marketing-style tutorials.

Exponentially-Weighted Moving Average (EWMA): The Key to Understanding Adam and Other Deep Learning Optimizers

A major highlight of this tutorial is its in-depth explanation of the Exponentially-Weighted Moving Average (EWMA). This concept may appear simple, but it is the key foundation for understanding modern deep learning optimizers (such as Momentum, RMSprop, and Adam).

The EWMA recursive formula is vt = βvt-1 + (1-β)θt, where β is the decay coefficient (typically 0.9 or 0.99) and θt is the current observation. The larger β is, the longer the "memory window," with an effective window of approximately 1/(1-β) time steps. For example, when β=0.9, the effective window is about 10 steps; when β=0.99, it's about 100 steps.

The tutorial explains in detail why bias arises in EWMA and how memory affects the average. In the Adam (Adaptive Moment Estimation) optimizer, the first moment estimate mt tracks the mean direction of the gradient (similar to Momentum), while the second moment estimate vt tracks the mean of squared gradients (similar to the adaptive learning rate in RMSprop). Proposed by Kingma and Ba in 2015, Adam's parameter update formula is θt+1 = θt - α·m̂t/(√v̂t + ε), where α is the learning rate and ε is a small constant to prevent division by zero (typically 10⁻⁸). The intuition behind this update formula is: m̂t in the numerator provides gradient direction and smooths noise (momentum effect), while √v̂t in the denominator adaptively scales the learning rate for each parameter dimension — dimensions with highly variable gradients automatically get reduced learning rates, while dimensions with stable gradients maintain larger learning rates.

At the beginning of training, due to the initialization m0=v0=0, the moving averages in the first few steps are severely biased toward zero. The bias correction formulas m̂t = mt/(1-β₁ᵗ) and v̂t = vt/(1-β₂ᵗ) are designed precisely to eliminate this initialization bias. This correction term has a significant effect when t is small and naturally approaches 1 as t grows, causing the correction effect to vanish — this is the fundamental reason why Adam and similar optimizers need bias correction terms.

Adam's default hyperparameters β₁=0.9, β₂=0.999, ε=10⁻⁸ perform well in most scenarios, which is an important reason it has become the default optimizer in deep learning. Subsequent variants like AdamW (which decouples weight decay from L2 regularization, proposed by Loshchilov and Hutter in 2019) and LAMB (optimized for large-batch training) further extend its applicability. Optimizers widely used in recent large language model training, such as AdaFactor and LION, can also be traced back to the fundamental statistical concept of EWMA.

Connecting EWMA from statistics with deep learning optimizers is where this tutorial demonstrates its greatest insight. It makes learners realize that seemingly isolated deep learning tricks actually share a unified statistical foundation.

Whiteboard Hand-Written Derivation: Returning to the Essence of Mathematical Learning

The creator specifically emphasizes that the entire content is presented through whiteboard derivation from scratch. In an era where video tutorials routinely rely on polished slides and rapid editing, whiteboard hand-writing is actually a "going against the current" choice.

The value of this approach is that it restores the authentic thinking process — how each step of the derivation follows from the previous one, what each symbol represents, all clearly revealed. For mathematically intensive content, this "slow pace" actually helps learners follow the derivation logic rather than being overwhelmed by conclusions.

Summary

This approximately two-hour statistics tutorial covers the complete knowledge chain from maximum likelihood estimation, linear regression, and empirical risk minimization to the method of moments and exponentially-weighted moving averages. Its uniqueness lies not only in the systematic coverage of content but also in establishing clear connections between statistical concepts and deep learning practice.

For learners who want to solidify their mathematical foundations in machine learning — especially practitioners who are not satisfied with knowing "what" but want to understand "why" — this kind of rigorous free learning resource deserves attention. Only by understanding these statistical foundations can one develop genuine judgment when facing new algorithms.