Introduction to Mathematical Foundations of Machine Learning: Definitions, Probabilistic Perspective & Core Framework Explained

Introduction: What Are We Really Talking About When We Discuss Machine Learning?

Many beginners jump straight into various algorithms and frameworks when first encountering machine learning, yet lack a clear understanding of its underlying mathematical essence. What does machine learning actually mean? Why are its mathematical foundations so important?

This article starts from the most fundamental definitions and gradually introduces the probabilistic perspective of machine learning, providing a clear introductory path for learners who want to truly understand the mathematical nature of ML.

The Classic Definition of Machine Learning: Tom Mitchell's T-P-E Framework

The term "machine learning," taken literally, means "a machine that learns." But this explanation is obviously too vague—what does "learning" mean? How does a machine "learn"?

Renowned Carnegie Mellon University professor Tom Mitchell provided a widely cited classic definition:

A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.

Tom Mitchell first systematically articulated this definition in his 1997 textbook Machine Learning, which remains a classic text at top universities worldwide. Mitchell was the founding head of CMU's Machine Learning Department—the world's first dedicated academic department for machine learning, established in 2006. CMU's involvement in machine learning can be traced back to the 1980s, when Herbert Simon and Allen Newell laid the symbolic AI foundations there. Mitchell then brought statistical learning methods into the mainstream, making CMU an important bridge connecting traditional AI with modern ML. The profound impact of this T-P-E definition lies in transforming the previously vague concept of "learning" into an operational, measurable engineering problem.

The elegance of this definition lies in its introduction of three core elements: Task T, Performance measure P, and Experience E.

The Universality of the T-P-E Framework

The T-P-E framework appears simple but possesses remarkable universality. By substituting the nature of task T and adjusting how performance P is measured, it can encompass virtually all types of machine learning problems:

In image classification tasks, T is identifying image categories, P is classification accuracy, and E is labeled training images;
In a chess-playing program, T is winning games, P is win rate, and E is self-play or historical game data;
In recommendation systems, T is predicting user preferences, P is click-through rate or conversion rate, and E is users' historical behavioral data.

Notably, this framework also implicitly suggests the three major paradigm divisions of machine learning: when experience E contains explicit input-output labels, it corresponds to supervised learning (e.g., image classification); when E contains only input data without labels, it corresponds to unsupervised learning (e.g., clustering and dimensionality reduction); when E appears in the form of reward signals from environmental feedback, it corresponds to reinforcement learning (e.g., chess-playing programs). This perspective of classifying learning paradigms based on the nature of E demonstrates the high abstraction capability of the T-P-E framework.

Through this unified abstract framework, the vast complexity of machine learning applications is organized under a single linguistic system. This is precisely the value of theoretical definitions—they help us see the essence beyond the surface.

The Probabilistic Perspective: The Best Mathematical Lens for Understanding Machine Learning

Machine learning problems can be most deeply understood when analyzed from a probabilistic perspective. This is a core viewpoint that permeates the entire ML field.

Why Is Probability Theory the Cornerstone of Machine Learning?

The real world is full of uncertainty. Data contains noise, measurements have errors, and future inputs cannot be fully predicted. Under these circumstances, how can we enable machines to make reasonable judgments?

Probability theory provides the mathematical language for handling uncertainty. From a probabilistic perspective, unknown quantities are assigned probability distributions and treated as random variables. The goal of machine learning is thus transformed into: given observed data, infer the distributions of these random variables and make decisions accordingly.

Specifically, this inference process is expressed within the Bayesian framework as follows: given a prior distribution P(θ) (reflecting our initial beliefs about model parameters) and a likelihood function P(D|θ) (the probability of data occurring under specific parameters), Bayes' theorem is used to compute the posterior distribution P(θ|D). This posterior distribution completely encodes all our knowledge about the parameters after observing the data. The frequentist school takes a different path, directly seeking parameter values that maximize the probability of the observed data through maximum likelihood estimation. Each paradigm has its advantages and disadvantages: Bayesian methods naturally quantify uncertainty but have high computational overhead, while frequentist methods are computationally efficient but may lose uncertainty information.

The Essence of Random Variables: A Counter-Intuitive Insight

Here is an intriguing fact:

A Random Variable is neither random nor a variable.

This statement sounds confusing at first but actually reveals the historical limitations of mathematical terminology. In the strict mathematical definition, a random variable is essentially a function mapping from a sample space to real numbers—it is a deterministic mapping, not some "randomly jumping value." The words "random" and "variable" are more products of historical convention than accurate descriptions of their mathematical essence.

Under the rigorous framework of measure theory, a random variable is defined as a measurable function from a probability space (Ω, F, P) to a measurable space (R, B(R)). Here Ω is the sample space (the set of all possible outcomes), F is a σ-algebra (defining which event sets can be assigned probabilities), and P is a probability measure. For an intuitive example: when rolling a die, the sample space Ω = {⚀, ⚁, ⚂, ⚃, ⚄, ⚅}, and the random variable X representing "the number rolled" is simply a function mapping each face to its corresponding integer (X(⚀)=1, X(⚁)=2, ...). The function itself is entirely deterministic; the "randomness" comes from our not knowing which outcome in Ω will be selected. Kolmogorov's axiomatic probability system established in 1933 elevated probability theory from an intuition-dependent empirical discipline to a rigorous branch of mathematics—this is the fundamental reason why modern machine learning theory can be rigorously proven and analyzed.

Understanding this point helps beginners overcome intuitive misunderstandings of terminology and truly grasp the mathematical essence of probability theory.

Optimal Decision-Making Under Uncertainty and Decision Theory

Probabilistic methods are not merely analytical tools—they also represent the optimal approach for making decisions under uncertainty.

This is fully demonstrated in Decision Theory. When facing incomplete information and needing to choose among multiple possible outcomes, the expected utility maximization framework based on probability provides the theoretically optimal decision scheme.

The core ideas of decision theory can be traced back to the expected utility theory proposed by von Neumann and Morgenstern in 1944. In the machine learning context, a complete Bayesian decision process contains three elements: the state space (the true state of the world, e.g., whether an email is spam), the action space (available decisions, e.g., mark as spam or let through), and the loss function (the cost of incorrect decisions, e.g., the cost of misclassifying a legitimate email as spam far exceeds that of missing one spam email). Bayesian decision theory states that when we have a posterior probability distribution over states, minimizing expected loss yields the optimal decision—the Bayes optimal decision.

This directly gives rise to many core concepts in machine learning: the Bayes optimal classifier in classification problems (equivalent to choosing the class with the highest posterior probability under 0-1 loss), conditional expectation in regression problems (the optimal prediction under squared loss), and the Bayesian Information Criterion (BIC) in model selection. It's worth noting that while the Bayes optimal classifier is theoretically perfect, it usually cannot be directly computed in practice—because the true conditional probability distribution is often unknown. Therefore, various approximation methods—from Naive Bayes to variational inference, from Markov Chain Monte Carlo (MCMC) to neural network approximate posteriors—constitute an important branch of modern ML algorithms.

From Bayesian inference to maximum likelihood estimation, from classifier decision boundaries to policy selection in reinforcement learning, probability theory permeates every aspect of machine learning. A solid foundation in probability is the prerequisite for deeply understanding modern machine learning algorithms.

Why Are Mathematical Foundations So Important for Machine Learning?

Avoiding the "API Caller" Trap

In today's world of highly mature deep learning frameworks, a model can be trained with just a few lines of code. But without understanding the underlying mathematical principles, learners easily remain at the "API calling" level:

Unable to understand why a model works or fails;
Lacking the ability to innovate and improve when facing new problems;
Difficulty debugging and optimizing deep-level model issues.

For example, when a deep learning model experiences vanishing gradients, without understanding the chain rule mechanism in backpropagation and the derivative properties of activation functions, one cannot determine whether to use ReLU instead of Sigmoid, or introduce residual connections or batch normalization. Similarly, when a model overfits, understanding the statistical principles of the bias-variance tradeoff enables targeted selection of regularization strategies (L1 sparsification vs. L2 smoothing) rather than blind experimentation.

Linear algebra, probability theory, calculus, and optimization theory—these mathematical cornerstones are precisely the key bridges connecting "knowing what" with "knowing why."

Linear algebra provides the language for data representation and transformation—matrix decompositions (such as Singular Value Decomposition, SVD) are core to dimensionality reduction and recommendation systems, eigenvalue decomposition is the mathematical foundation of Principal Component Analysis (PCA), and tensor operations are the basic operations of deep learning computation graphs. Probability theory and statistics provide tools for modeling uncertainty—from parameter estimation to hypothesis testing, from generative models to discriminative models. Calculus (especially multivariate calculus) is the foundation for understanding optimization algorithms like gradient descent—backpropagation is essentially the systematic application of the chain rule, and automatic differentiation technology engineers this process. Optimization theory formalizes learning problems as extremum-finding for objective functions—convex optimization guarantees the existence and efficient computation of global optima, while non-convex optimization (such as loss surfaces of deep networks) faces challenges of local minima and saddle points, giving rise to modern optimizers like Adam and SGD with momentum. These four pillars do not exist in isolation: for example, maximum likelihood estimation simultaneously involves probabilistic modeling (construction of the likelihood function), calculus (differentiation of the log-likelihood), and optimization (finding parameter extrema), demonstrating the deep interconnections between mathematical branches.

The Importance of a Systematic Learning Path

Starting from fundamental definitions and progressively building a knowledge system is far more efficient than learning various algorithms in a fragmented manner. Understanding the first principles of machine learning makes subsequent learning twice as effective with half the effort.

First principles thinking originates from Aristotle's philosophical tradition, referring to decomposing complex systems into the most basic, irreducible truths, then re-deriving from these basic truths. In the machine learning domain, first principles means deriving algorithms from basic mathematical axioms and statistical assumptions, rather than merely memorizing algorithmic steps. For example, understanding the least squares solution of linear regression can be derived from a probabilistic model (assuming noise follows a Gaussian distribution) via maximum likelihood, understood from a geometric perspective (projection theorem onto the column space), or analyzed from an optimization perspective (KKT conditions of convex quadratic programming). These three perspectives converge to the same result, but each provides different directions for extension to more complex problems: the probabilistic perspective extends to Bayesian linear regression and Gaussian processes, the geometric perspective extends to kernel methods and manifold learning, and the optimization perspective extends to online learning and stochastic optimization. Learners who master first principles can re-derive solutions from basic principles when facing entirely new problems, rather than relying on pattern matching to existing algorithms.

Conclusion: Learning Machine Learning from First Principles

This article has outlined the two core perspectives of machine learning: Tom Mitchell's T-P-E definitional framework and the probabilistic perspective that runs throughout.

For anyone wanting to systematically study machine learning, starting from such first principles is far more valuable than blindly accumulating algorithms. Understanding "what machine learning really is" and why probability theory is the best lens for analyzing ML is an indispensable first step on this learning journey.

If you've ever been confused about "what machine learning really means," then starting fresh from mathematical foundations may bring entirely new cognitive breakthroughs. The recommended learning sequence is: first solidify foundations in probability theory and linear algebra, then understand statistical estimation theory (maximum likelihood, Bayesian inference), followed by optimization methods, and only then specific machine learning algorithms and deep learning architectures. While this path requires greater initial investment, it builds a solid cognitive framework that makes subsequent learning like adding bricks to an existing building, rather than repeatedly rebuilding on sand.