Gradient Descent and Backpropagation Explained: How Neural Networks Learn from Scratch

The most fascinating part of neural networks isn't their architecture — it's how they learn on their own. This article breaks down the core mechanisms behind neural network learning from the ground up: cost functions, gradient descent, backpropagation, and stochastic gradient descent (SGD). Based on Chapter 2 of a widely praised free interactive course, it traces the complete logical chain from "measuring performance" to "efficient training."

What Does It Mean for a Model to "Learn"?

Before designing a learning algorithm, we need a way to measure network performance. The most straightforward approach: feed the network a batch of samples and count how many it gets right. But that's not enough — we don't just want to know right or wrong, we want to know how close it is to the correct answer, so we can "reward" it as it gradually improves.

Consider a point classification task. Blue points have label y_blue = [1, 0]ᵀ and red points have y_red = [0, 1]ᵀ. For the same point known to be blue, two networks predict:

• ŷ₁ = [0.12, 0.92]ᵀ
• ŷ₂ = [0.41, 0.63]ᵀ

By accuracy alone, both predicted red — both are wrong. But intuitively, ŷ₁ is "more wrong" — it was highly confident yet incorrect; ŷ₂ was more uncertain. By computing the distance between predictions and the true label (‖y − ŷ₁‖ ≈ 1.273, ‖y − ŷ₂‖ ≈ 0.863), we can quantify the degree of error. This approach matters because it gives partial credit to models that are close to the right answer, while encouraging more confident correct predictions.

Cost Function: Turning "Degree of Error" into Math

This intuition can be formalized with a cost function:

C(w, b) = 1/(2n) Σᵢ ‖yᵢ − ŷᵢ‖²

Here w is the weights, b is the biases, n is the number of training samples, yᵢ is the true label, and ŷᵢ is the network's prediction. This is known as Mean Squared Error (MSE) — a great starting point for understanding neural network learning.

MSE has two key properties: each squared error term is always non-negative, and the worse the network predicts, the larger the MSE. So the problem of "how to train a neural network" is fundamentally equivalent to how to minimize the cost function.

It's worth noting that MSE is not the only option. Over the history of deep learning, researchers have designed many cost functions for different tasks: Cross-Entropy Loss often converges faster than MSE for classification tasks because it works naturally with softmax outputs and avoids vanishing gradients in saturation regions; Huber loss is more robust to outliers; contrastive loss and triplet loss are designed specifically for metric learning. MSE is the preferred starting point because of its clean mathematical properties and intuitive geometry — the squared term ensures differentiability, and larger errors are penalized more heavily (quadratic growth), naturally discouraging large deviations. As you go deeper into neural networks, choosing the right cost function becomes both an engineering and theoretical art.

Gradient Descent: Finding the Lowest Point in High-Dimensional Space

Anyone with a calculus background might first think to set the derivative to zero for an analytical solution. This might work for a tiny network with a few parameters, but in high-dimensional spaces, analytical solutions are nearly impossible. A minimal point classifier already has 17 parameters; a handwritten digit classifier has 12,175 — in such a high-dimensional, deeply coupled parameter space, finding a closed-form minimum is both tedious and often intractable.

Instead, we turn to a strategy that scales to high dimensions: gradient descent. The classic analogy: imagine you're lost on a mountaintop with no visibility. The most rational move is to step in the direction of steepest descent.

For a two-parameter function C(v₁, v₂), this is expressed as:

ΔC = ∇C · Δv

The gradient points in the direction of steepest ascent, so its negative is the direction of steepest descent:

Δv = −η∇C

Here η (the learning rate) controls step size. Substituting gives ΔC = −η‖∇C‖², and since η > 0 and ‖∇C‖² ≥ 0, we guarantee ΔC ≤ 0 — every update reduces the cost. Repeatedly applying the update rule:

v → v′ = v − η∇C

we converge toward some local minimum. Applied to neural networks, the weight and bias update rules follow the same form:

• wᵢ → wᵢ − η(∂C/∂wᵢ)
• bⱼ → bⱼ − η(∂C/∂bⱼ)

Local Minima and Practical Considerations for Learning Rate

Gradient descent has two important caveats in practice. First, it typically finds a local minimum rather than the global optimum — smaller networks are especially prone to getting stuck in poor local solutions. Interestingly, research has shown that in very large networks, most local minima perform comparably to the global optimum — because large networks have the flexibility to model complex patterns. Second, a learning rate that's too high causes oscillation around the minimum, while one that's too low makes training unnecessarily slow. In practice, the learning rate is often adjusted dynamically during training.

Backpropagation: The Key Algorithm for Efficiently Computing Gradients

So far we've glossed over a central question: how do you actually compute the gradient? Manual differentiation doesn't scale — computing ∇C requires averaging gradients over every training sample, and repeating this at every step. Numerical differentiation and symbolic differentiation both fail to scale to networks with tens of thousands of parameters.

The solution is backpropagation. Its core insight: each layer of a neural network is a function composition, forming a chain of dependencies that's perfectly suited for applying the chain rule layer by layer, working backward from the output.

The chain rule — a fundamental calculus tool established as far back as Leibniz in the 17th century — wasn't systematically applied to neural network training until the 1986 paper by Rumelhart, Hinton, and Williams brought it to wide attention in the deep learning field. Modern deep learning frameworks (PyTorch, TensorFlow) represent neural networks as computational graphs: each operation is a node, data flow is an edge. Forward propagation flows along the graph in one direction, recording all intermediate operations; backpropagation flows in reverse, automatically computing gradients — a mechanism known as automatic differentiation (Autograd). Understanding the manual derivation of backpropagation is the prerequisite for understanding why these autograd frameworks are so powerful.

The algorithm introduces an intermediate quantity — error δˡⱼ = ∂C/∂zˡⱼ — measuring how sensitive the cost is to the weighted input of neuron j in layer l. From this, four fundamental equations can be derived:

1. Output layer error: δᴸ = ∇ₐC ⊙ σ′(zᴸ). For MSE, ∇ₐC = aᴸ − y; ⊙ denotes element-wise multiplication (Hadamard product).
2. Error backpropagation: δˡ = ((wˡ⁺¹)ᵀ δˡ⁺¹) ⊙ σ′(zˡ). The transposed weight matrix propagates error back to the previous layer, passing through the derivative of the activation function.
3. Bias gradient: ∂C/∂bˡ = δˡ. Since biases only add, the chain rule collapses directly to the error itself.
4. Weight gradient: ∂C/∂wˡ = δˡ (aˡ⁻¹)ᵀ. Weights multiply the input activations, so the gradient includes that term.

Why Backpropagation Is Efficient

The full pipeline is: input → forward pass (storing all weighted sums and activations) → compute output error → propagate error backward layer by layer while computing gradients → output complete gradient.

Backpropagation's true efficiency comes not just from the chain rule itself, but from reusing intermediate values and error terms stored during the forward pass, avoiding redundant computation — the core idea of dynamic programming. Symbolic differentiation is inefficient precisely because it can't exploit these intermediate results. A single backward pass yields gradients for all parameters simultaneously, at a computational cost roughly comparable to the forward pass. This is why even modern large-scale models with billions of parameters can be trained in reasonable time.

Stochastic Gradient Descent: Making Training Fast Enough to Be Practical

Backpropagation solves the problem of computing gradients for a single sample, but computing ∇C still requires averaging gradients over all samples — the more samples, the slower each update. Stochastic Gradient Descent (SGD) dramatically alleviates this bottleneck.

The core idea: instead of averaging over all data, split the data into mini-batches of size m to approximate the true gradient:

∇C ≈ 1/m Σⱼ ∇C_Xⱼ

The key advantage is a much higher update frequency. With n samples and batch size m, each epoch allows n/m updates instead of just one. It's like wanting to gauge public opinion on a topic — randomly sampling 100 people gives a reliable estimate without needing to survey everyone, as long as the sample is unbiased. In a classification task with nearly a thousand points, a batch size of 30 can yield roughly a 33× speedup in training.

SGD traces its roots to the Robbins-Monro stochastic approximation algorithm from 1951 and has evolved continuously over decades. The main drawbacks of vanilla SGD are using the same learning rate for all parameters and noisy, erratic update directions. This led to a series of improvements: Momentum smooths the update trajectory by accumulating historical gradient directions; AdaGrad automatically reduces the learning rate for frequently updated parameters; RMSProp fixes AdaGrad's monotonically decaying learning rate problem; and Adam (2014) combines momentum with adaptive learning rates, becoming today's most widely used optimizer. That said, in large language model training, carefully tuned SGD+Momentum can sometimes match Adam — a reminder that deep understanding of fundamentals never goes out of style.

Noise as an Advantage: Escaping Local Minima and Saddle Points

One often underappreciated benefit of SGD is its statistical noise. Because each mini-batch is only an approximation of the true gradient, even at points where the true gradient is zero, the approximate gradient is rarely exactly zero. This noise is enough to "shake" the network out of shallow local minima.

This property is especially important for saddle points. Mathematically, in an n-dimensional parameter space, the number of saddle points among critical points (where the gradient is zero) grows exponentially with n, while true local minima are extremely rare. Research by Dauphin et al. in 2014 showed that most "difficulties" in neural network training come not from local minima but from saddle points and plateau regions (where gradients are very small but non-zero) — at saddle points, the gradient vanishes but curvature exists in some directions, and ordinary gradient descent may get stuck, while SGD's random perturbations help the network escape and continue converging. This finding fundamentally reshaped how researchers understand optimization difficulties, and explains why SGD's noise is a systematic advantage in deep networks rather than a lucky accident.

The Complete Picture of Neural Network Training

At this point, all the core components needed to train a neural network are in place:

• Cost function: measures the gap between predictions and ground truth
• Gradient descent: minimizes the cost by following the steepest downhill direction
• Backpropagation: efficiently computes gradients for all parameters
• Stochastic gradient descent: makes the entire training process fast enough to be practical

These four modules are tightly interlocked — none can stand alone. With this logical chain in hand, the natural next step is to implement SGD and backpropagation from scratch in NumPy, training a handwritten digit classifier — manually tuning the learning rate, watching SGD escape saddle points — to truly internalize these abstract concepts. When you can derive backpropagation's four equations from first principles and then open PyTorch's .backward() source code, you'll find that autograd frameworks are nothing more than an engineering encapsulation of this exact logic. At that moment, the "black box" truly opens.

Key Takeaways