Linear Regression from Scratch: Implementing Gradient Descent with NumPy to Understand Machine Learning

Introduction: Why Bother Implementing Linear Regression by Hand

Recently, a learner shared their experience in the Reddit machine learning community: they implemented a linear regression algorithm from scratch using only NumPy. In their own words, "I know it's not something amazing, but I recently learned it in class and implemented it myself in Python and NumPy." (They used Pandas to load CSV data from Kaggle.)

This kind of "reinventing the wheel" might seem pointless given the mature ecosystem of machine learning tools — after all, scikit-learn can accomplish the same task in a single line. But from a learning perspective, implementing a fundamental algorithm by hand is one of the most effective ways to truly grasp machine learning principles. This article uses that example as a springboard to dive deep into the implementation details and mathematical logic behind linear regression.

Linear Regression: The Cornerstone of Machine Learning

Linear regression is one of the most foundational and important algorithms in supervised learning. It attempts to find a line (or hyperplane) that best fits the relationship between input features and the target variable.

The Mathematical Formulation of Linear Regression

For single-variable linear regression, the model can be expressed as:

y = w * x + b

where w is the weight (slope) and b is the bias (intercept). For the multi-variable case, this extends to:

y = w1*x1 + w2*x2 + ... + wn*xn + b

Matrix notation makes this more concise, which is one of the key advantages of using NumPy:

y = X · W + b

Why NumPy Instead of Pure Python

The learner's choice of "only numpy" was a wise one. NumPy provides efficient vectorized operations that eliminate the need for explicit for loops. Matrix multiplication, transposition, and element-wise operations all execute at near-C speeds. This not only makes the code cleaner but also brings it closer in spirit to production-grade implementations.

Core Implementation: Loss Function and Gradient Descent

Implementing linear regression comes down to two key components: measuring how good the predictions are (the loss function), and optimizing the parameters (gradient descent).

Mean Squared Error (MSE) Loss Function

The most commonly used loss function is Mean Squared Error:

MSE = (1/n) * Σ(y_pred - y_true)²

It measures the average squared difference between predicted and actual values. Our goal is to find a set of W and b that minimizes this error as much as possible.

The Core Logic and Code for Gradient Descent

Gradient descent repeatedly computes the partial derivatives of the loss function with respect to the parameters, then updates the parameters in the opposite direction of the gradient:

import numpy as np

def gradient_descent(X, y, lr=0.01, epochs=1000):
    n, d = X.shape
    W = np.zeros(d)
    b = 0
    for _ in range(epochs):
        y_pred = X @ W + b
        error = y_pred - y
        dW = (2/n) * X.T @ error
        db = (2/n) * np.sum(error)
        W -= lr * dW
        b -= lr * db
    return W, b

This code distills the essence of linear regression: predict, compute error, calculate gradients, update parameters — and repeat until convergence. The learning rate lr controls the step size of each update; too large and it oscillates and diverges, too small and convergence is painfully slow.

The mathematical foundation of gradient descent is worth unpacking a bit. Taking the partial derivative of the MSE loss with respect to W gives ∂L/∂W = (2/n) * Xᵀ(XW - y), and the partial derivative with respect to b is ∂L/∂b = (2/n) * Σ(y_pred - y). These two gradient formulas map directly to dW and db in the code. Gradient descent has several variants: Batch Gradient Descent computes gradients using the entire dataset each time — stable in direction but computationally expensive; Stochastic Gradient Descent (SGD) uses only one sample per update — fast but noisy; Mini-batch Gradient Descent is a compromise between the two and is the most widely used approach in deep learning frameworks. The code above implements Batch Gradient Descent, which is a reasonable choice for small datasets.

Data Handling: The Role of Pandas in the Machine Learning Pipeline

The learner mentioned that they "technically did use Pandas to import data from a Kaggle CSV." That's a perfectly practical choice. In real-world machine learning workflows, data loading and preprocessing often consume a significant portion of the total effort.

Pandas excels at handling structured data — reading CSVs, dealing with missing values, and selecting features are all straightforward:

import pandas as pd

df = pd.read_csv('data.csv')
X = df[['feature1', 'feature2']].values
y = df['target'].values

Don't Forget Feature Normalization

One detail beginners often overlook is feature scaling. When different features have vastly different value ranges, gradient descent becomes inefficient or may even fail to converge. Standardization (subtracting the mean and dividing by the standard deviation) can significantly improve training performance:

X = (X - X.mean(axis=0)) / X.std(axis=0)

Why feature standardization (Z-score Normalization) affects gradient descent efficiency can be understood through the shape of the loss function's contour lines. When two features differ dramatically in scale (e.g., one ranges from 0 to 1 and another from 0 to 10,000), the loss function's contours in parameter space form narrow, elongated ellipses. Gradient descent oscillates back and forth along the steep direction, making extremely slow progress toward the optimum. After standardization, the contours become more circular, the gradient direction points more directly toward the minimum, and convergence speeds up dramatically. Besides Z-score normalization, another common technique is Min-Max normalization (scaling data to the [0, 1] range): X = (X - X.min()) / (X.max() - X.min()). When linear regression has an analytical solution (the normal equation), standardization isn't strictly necessary — but once you're using iterative optimization, it's virtually a required step.

The Real Value of Implementing Linear Regression from Scratch

Some might ask: since scikit-learn's LinearRegression can get the job done in one line, why go through the trouble of implementing it manually?

Understanding the Fundamentals Instead of Treating It as a Black Box

When you call a library function directly, you only see "data in, model out" — without understanding what happens in between. Implementing it yourself forces you to understand how gradients are computed, how the learning rate affects convergence, and why normalization is necessary. This kind of foundational understanding becomes critical when tackling more complex problems.

Building a Solid Foundation for Advanced Algorithms

The gradient descent framework from linear regression transfers almost seamlessly to logistic regression, neural networks, and other more complex models. Once you've understood this simplest case, learning backpropagation in deep learning will click into place much more naturally. As many in the community have commented: "Everyone in machine learning should implement these fundamental algorithms by hand at least once."

Conclusion: From Foundational Practice to Deeper Understanding

The humble attitude of that Reddit user — "I know it's not something amazing" — is actually something to admire. The journey of learning machine learning begins with exactly these seemingly simple foundational exercises. Building a linear regression model from scratch with NumPy may seem like a small step, but it is the first building block toward understanding the entire world of machine learning.

For everyone currently learning machine learning, this offers a valuable lesson: don't settle for calling ready-made APIs. Try implementing the fundamental algorithms yourself, and you'll arrive at a fundamentally different — and much deeper — understanding of the field.