Probabilistic Machine Learning in Practice: Code Verification from Loss Functions to Deep Neural Networks

Bridging the Gap Between Mathematical Theory and Code Practice

In the typical machine learning learning journey, there's often a significant gap between theory and practice. Many learners can confidently call pre-built library functions, yet struggle to truly understand the mathematical principles behind their model's outputs. Recently, a content creator named Aayush released the second lecture's code implementation for his "Probabilistic Machine Learning" series — an effort to bridge exactly that gap. The goal isn't just to teach you how to write code, but to guide you toward understanding the outputs you get and verifying them mathematically.

This teaching philosophy emphasizes understanding not just the what, but the why: when you see firsthand that Negative Log Likelihood (NLL) and Mean Squared Error (MSE) produce identical results under a Gaussian distribution assumption, the elegant harmony between theory and practice becomes immediately tangible.

A Probabilistic Perspective on Loss Functions

Comparing L1 and L2 Loss Curves

The course begins with the most fundamental building blocks — comparing the curve shapes of L1 loss (absolute error) and L2 loss (squared error). These two aren't merely different mathematical choices; they correspond to distinct probabilistic assumptions and robustness characteristics. L2 loss is more sensitive to outliers, while L1 loss is comparatively robust. Grasping this distinction forms the foundation for deeper probabilistic modeling ahead.

Gaussian Output Distributions and Uncertainty Modeling

What truly sets this course apart from typical "just import the library" tutorials is its introduction of Gaussian output distributions for modeling prediction uncertainty. Traditional regression tasks typically produce a single point estimate, but from a probabilistic perspective, the model outputs a full probability distribution — telling you not just what the prediction is, but how confident the model is in that prediction.

This approach to uncertainty modeling has profound real-world implications. In high-stakes domains like autonomous driving and medical diagnosis, a model's ability to "know what it doesn't know" is often more critical than raw predictive accuracy.

Gaussian output distributions represent the most fundamental form of Predictive Uncertainty Quantification (UQ). Under this framework, the model no longer outputs a single scalar $\hat{y}$, but instead outputs distribution parameters — typically a mean $\mu(x)$ and variance $\sigma^2(x)$, both of which can be learned by a neural network. The variance $\sigma^2(x)$ captures a mixture of epistemic uncertainty (the model has seen little data in a certain region and lacks confidence) and aleatoric uncertainty (inherent noise in the data itself). This shares the same underlying motivation as more advanced uncertainty estimation methods such as Bayesian Neural Networks and Deep Ensembles, making it an essential foundational concept on the path toward Trustworthy AI.

Mathematical Equivalence of NLL and MSE

One of the course's central "aha moments" is revealing that, under the assumption of a Gaussian output distribution, minimizing Negative Log Likelihood and minimizing Mean Squared Error are mathematically equivalent.

This equivalence is no coincidence. When we assume prediction errors follow a zero-mean, fixed-variance Gaussian distribution, taking the negative log of the likelihood function and expanding it yields a primary term that is exactly the MSE form. This means the MSE loss we use so routinely actually carries an implicit Gaussian noise assumption. Understanding this empowers us to more consciously choose appropriate loss functions when faced with non-Gaussian data.

Expanding on the mathematical derivation: assume observed data $y_i$ satisfies $y_i = f(x_i) + \epsilon$, where $\epsilon \sim \mathcal{N}(0, \sigma^2)$. The likelihood for each sample is $p(y_i|x_i) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(y_i - f(x_i))^2}{2\sigma^2}\right)$. Taking the negative log-likelihood over the full dataset and simplifying, the constant terms and $\sigma$-related terms are all independent of the model parameters, leaving a minimization objective equivalent to $\sum_i (y_i - f(x_i))^2$ — i.e., MSE. This derivation reveals a profound symmetry: choosing a loss function is essentially choosing a probability distribution model for the data noise. If the noise follows a Laplace distribution, the optimal loss function becomes L1 (absolute error); for heavy-tailed noise, a more robust loss design is needed. Loss function selection is therefore not mere convention — it is a declaration of probabilistic assumptions about the data generating process.

From Linear Regression to Deep Neural Networks

The Convex Bowl Loss Surface and Optimization Guarantees

The course then analyzes linear regression and demonstrates that its loss surface takes the classic convex bowl shape. Convexity guarantees that the optimization process has a unique global minimum, so gradient descent will never get trapped in a local optimum. This is one reason linear models remain widely used today: they are interpretable, easy to optimize, and come with strong theoretical guarantees.

Using Polynomial Regression to Understand Underfitting and Overfitting

To intuitively explain underfitting and overfitting — one of machine learning's central challenges — the course uses polynomial regression as its demonstration vehicle. When the polynomial degree is too low, the model fails to capture the true structure in the data, resulting in underfitting. When the degree is too high, the model over-adapts to the noise in the training data, causing generalization to deteriorate.

By iteratively adjusting model complexity and observing how the loss changes, learners develop a far stronger intuition for the bias-variance tradeoff than any written description could provide.

The Bias-Variance Tradeoff is the core framework for understanding underfitting and overfitting. A model's expected generalization error can be decomposed into three terms: squared bias (the model's systematic deviation from the true pattern), variance (the model's sensitivity to fluctuations in training data), and irreducible noise. Low-degree polynomials exhibit high bias and low variance, corresponding to underfitting; high-degree polynomials exhibit low bias and high variance, corresponding to overfitting. Common techniques for addressing overfitting include regularization (L1/L2 penalty terms), Dropout, early stopping, and increasing training data — all of which apply equally in deep learning, and whose theoretical foundations are most clearly visible in this simple polynomial regression setting.

Deep Neural Networks: From Manual Feature Engineering to Automatic Feature Learning

While polynomial regression can fit complex curves, it relies on manually designed features. As data dimensionality grows and structure becomes more complex, manual feature design quickly becomes inadequate. This is precisely where deep neural networks come in — they can automatically learn hierarchical feature representations from data, without human intervention.

The course concludes by implementing a deep neural network, bringing together all previously accumulated concepts (loss function selection, probabilistic modeling, overfitting diagnosis) within a far more powerful modeling framework. This carefully sequenced progression allows learners to clearly see why we need neural networks and what problems they actually solve.

The Learning Value of Theory and Code Mutually Verifying Each Other

In today's landscape of AI learning resources, quick-start tutorials are everywhere — but content that genuinely dives into the mutual verification of mathematical principles and code implementation is relatively rare. Aayush's series adopts a dual-video structure of theory lectures paired with code implementations, offering a more rigorous learning paradigm: first understand the theoretical derivation, then verify it through code, and ultimately transform abstract formulas into runnable, observable results.

For learners who want to advance from "knowing how to call APIs" to "understanding the underlying principles," this approach of mapping concepts like negative log likelihood, Gaussian distributions, and loss curves one-to-one with actual code outputs can effectively address a common frustration in theoretical learning: knowing a formula without understanding what it means.

The beauty of machine learning lies precisely in the unity of theory and practice. When the code you write by hand verifies the mathematical conclusions in a textbook, the depth of understanding you gain is something that reading alone simply cannot provide. Whether you are a newcomer just starting out or a practitioner looking to solidify your fundamentals, this "theory → code → verification" trinity is an approach well worth trying.

Related Resources:

• Code implementation video: https://youtu.be/6ZTVp70Mf5s
• Theory lecture video: https://youtu.be/iThI5AapBc0