Coding Machine Learning Probability Distributions from Scratch: A Practical Guide to Gaussian, Beta, and Gamma Distributions

Why Code Probability Distributions by Hand?

Probability distributions are the mathematical bedrock of machine learning. From the Gaussian assumptions underpinning linear regression to prior specifications in Bayesian inference, virtually every mainstream algorithm operates on the language of probability distributions. Yet most learners stop at memorizing formulas and rarely implement these equations in actual code.

Recently, a developer shared their experience of "coding ML probability distributions by hand" on Reddit, complete with detailed visualizations. In their own words: "Seeing those equations actually run felt incredibly rewarding." This transformation from paper formulas to runnable code is precisely the best path to understanding the true nature of probability distributions. This article walks through the core distributions covered in this exercise and the insights behind them.

Univariate Gaussian Distribution: Where It All Begins

The author calls the Univariate Gaussian "subjectively the most important distribution" — and that's hardly an exaggeration. The Gaussian distribution is ubiquitous thanks to the Central Limit Theorem: the sum of a large number of independent random variables tends toward a normal distribution. This makes it the default choice for noise modeling, regression error assumptions, and even neural network weight initialization.

Implementing the Gaussian probability density function (PDF) by hand gives you an intuitive feel for how the mean μ controls location and the variance σ² controls spread. When you adjust these two parameters in code and watch the curve change in real time, those abstract symbols finally become perceptible geometric shapes.

The Difference Between Homoscedasticity and Heteroskedasticity

The author specifically contrasts homoscedastic and heteroskedastic scenarios. The core distinction lies in whether the variance depends on the input variable.

• Homoscedastic: The noise variance remains constant regardless of the input value. This is the standard assumption of classical linear regression.
• Heteroskedastic: The variance changes with the input — different regions have different levels of uncertainty.

This distinction has enormous practical significance. Real-world data is often heteroskedastic — for example, in income prediction, the variance among high earners is typically much larger than among low earners. If you continue modeling under a homoscedastic assumption, the resulting confidence intervals will be systematically distorted. By coding and visualizing the difference between the two, you can clearly see how the choice of variance modeling affects prediction reliability.

Heavy-Tailed Distributions: Improving Model Robustness to Outliers

Another key topic the author addresses is heavy-tailed distributions and their robustness to outliers.

The tails of a Gaussian distribution decay extremely fast (exponentially), meaning it "believes" extreme values are virtually impossible. So when outliers actually appear in the data, models based on Gaussian assumptions get severely skewed. Heavy-tailed distributions like the Student's t distribution, on the other hand, have slower-decaying tails and naturally "expect" the occasional extreme observation.

This leads to a core insight: a distribution's robustness fundamentally depends on how we model its tail behavior. Replacing the Gaussian with a heavy-tailed distribution allows the model to remain stable in the face of outlier perturbations — this is the central idea behind Robust Statistics. When you inject outliers into your code and observe how much parameter estimates shift under different distributions, the robustness difference becomes immediately apparent.

Beta Distribution: Two Parameters, Infinite Shapes

The author's enthusiasm for the Beta distribution is palpable: "Just two parameters, yet so versatile — capable of producing such a rich variety of density shapes that can fit countless arbitrary curves!"

This observation captures the essence of the Beta distribution. Defined on the [0, 1] interval, it is controlled by two shape parameters α and β:

• When α = β = 1, it reduces to the uniform distribution;
• When α = β and both are greater than 1, it forms a symmetric bell shape;
• When α and β differ significantly, the distribution skews toward one end;
• When α, β < 1, it even takes on a U-shape.

This flexibility makes the Beta distribution the natural conjugate prior for binomial distribution parameters in Bayesian statistics, widely used in click-through rate modeling, A/B testing, and similar applications. Encoding such rich prior beliefs with just two numbers is truly an elegant feat of probability theory.

Gamma Distribution, Exponential Distribution, and Empirical Distributions

Beyond the above, the author also implemented the Gamma distribution and the exponential distribution. The exponential distribution is commonly used to model waiting times between events, while the Gamma distribution can be viewed as a generalization of multiple exponential processes, playing an important role in queueing theory and reliability analysis.

How Empirical Distributions Approximate the True CDF

The experiment that "fascinated" the author the most involved the empirical distribution: after sampling a set of points from a normal distribution and modeling them as an empirical distribution, the resulting empirical cumulative distribution function (that staircase-shaped CDF) turned out to approximate the true continuous Gaussian CDF with remarkable precision.

The phenomenon behind this is guaranteed by the Glivenko–Cantelli Theorem — as the sample size increases, the empirical distribution function converges uniformly to the true distribution function. This is also the theoretical foundation for why many resampling methods (such as Bootstrap) work. Watching the discrete staircase curve gradually conform to the smooth theoretical curve as sample points increase — that's how you build the bridge between theory and practice with your own hands.

Understanding the Mathematical Essence of Probability Distributions Through Code

Perhaps the greatest takeaway from this exercise is about the learning method itself. Probability distributions should not be mere formulas in a textbook — they should be living objects that can be parameterized, visualized, and stress-tested.

When you implement the Gaussian density function by hand, contrast homoscedastic versus heteroskedastic behavior, observe how heavy-tailed distributions resist outliers, and experience the magic of the Beta distribution's two parameters, those once-abstract mathematical concepts come alive at your fingertips. As the author puts it: "Never stop learning, and always code it up!" For anyone who truly wants to master the foundations of machine learning, this may be the most worthwhile investment of your time.

Original video link: https://youtu.be/mdumfp-mamI