Building a Neural Network from Scratch: A Practical Guide to Backpropagation and Gradient Computation

Why Build a Neural Network from Scratch

In an era of highly mature deep learning frameworks, tools like PyTorch and TensorFlow make building neural networks remarkably easy—a few lines of code can train a working model. But precisely because of this convenience, more and more developers are asking themselves: when we're accustomed to calling nn.Linear and model.fit(), do we truly understand the internal mechanics of neural networks?

Recently, a developer on Reddit launched a representative project initiative: they wanted to build a custom neural network from scratch and were looking for like-minded people to brainstorm and share ideas. This seemingly simple idea actually touches on a core principle in the machine learning learning path—understanding over invocation.

The value of building from scratch isn't about replacing mature frameworks—it's about closing the cognitive loop. When you implement backpropagation by hand, manually derive gradients, and debug numerical instability issues, those previously abstract concepts become concrete and deeply understood.

Core Components You Need to Implement When Building a Neural Network from Scratch

Breaking Down the Core Components

A minimal viable neural network requires the following core modules, each worth understanding independently:

• Forward Pass: Data flows through the network layer by layer, undergoing linear transformations and activation functions to ultimately produce an output. This part is relatively intuitive—it's essentially the stacking of matrix multiplications and nonlinear mappings.
• Activation Functions: ReLU, Sigmoid, Tanh, etc. Understanding their derivative properties is a prerequisite for understanding vanishing/exploding gradient problems. Vanishing and exploding gradients are classic challenges in training deep networks, rooted in the chain multiplication nature of backpropagation: gradients must be multiplied across many layers to propagate back to earlier layers. If each layer's gradient value is slightly less than 1 (for example, the Sigmoid function's maximum derivative is only 0.25), after multiplying across dozens of layers the gradient decays exponentially toward zero, leaving earlier layers virtually unable to update—this is the vanishing gradient problem. Conversely, if each layer's gradient exceeds 1, the gradient grows explosively through multiplication. ReLU became popular precisely because its derivative in the positive region is always 1, effectively mitigating the vanishing gradient problem. However, it introduces a new challenge—"Dead ReLU"—when a neuron's output is consistently negative, its gradient is permanently zero and the neuron becomes permanently inactive. Variants like Leaky ReLU and ELU were designed specifically to address this issue.
• Loss Functions: Mean Squared Error (MSE) for regression, cross-entropy for classification. The loss function determines the direction of network optimization.
• Backpropagation: This is the technical core of the entire project and the part most prone to errors. It is fundamentally the systematic application of the chain rule on a computational graph. The Chain Rule is a fundamental theorem of calculus: the derivative of a composite function equals the product of the derivatives of each constituent function. In the neural network context, this means the gradient of the loss function with respect to any weight can be obtained by multiplying the partial derivatives layer by layer from the output layer to the layer containing that weight. The Computational Graph is a tool for visualizing and systematizing this process—each operation is represented as a node in the graph, and data flow is represented as edges. PyTorch's autograd engine essentially builds a computational graph dynamically at runtime and traverses it in reverse when .backward() is called to automatically compute gradients. Understanding computational graphs is key not only for understanding backpropagation but also for understanding the architectural differences between dynamic graphs (PyTorch) and static graphs (early TensorFlow).
• Optimizer: Start with the most basic Stochastic Gradient Descent (SGD) and progressively understand improvements like momentum and Adam. SGD is the most straightforward optimization method: use one sample or a mini-batch to estimate the gradient, then update parameters in the direction of the negative gradient. However, SGD oscillates and converges slowly when the loss surface has elongated valleys. Momentum borrows the concept of inertia from physics, introducing an exponentially weighted moving average of historical gradients to smooth the update direction and accelerate traversal through flat regions. AdaGrad maintains independent learning rates for each parameter, reducing the learning rate for frequently updated parameters and increasing it for sparsely updated ones. Adam (Adaptive Moment Estimation, proposed in 2014) combines the advantages of both momentum and adaptive learning rates, maintaining first-moment and second-moment estimates of the gradient with bias correction, making it the most widely used default optimizer today.

Recommended Implementation Path

For developers who want hands-on practice, a progressive path is more effective:

1. Start by implementing a single-layer perceptron using pure Python + NumPy to solve linearly separable problems (such as AND/OR logic).
2. Extend to a multi-layer network, manually implement backpropagation, and use it to solve nonlinear problems like XOR—the classic test for verifying backpropagation correctness. The XOR problem is important because it's the simplest linearly inseparable problem: a single-layer perceptron cannot correctly classify XOR's four inputs no matter how the weights are adjusted. This limitation was rigorously proven by Minsky and Papert in their 1969 book Perceptrons, which led to over a decade of stagnation in neural network research. Multi-layer perceptrons solved this problem by introducing hidden layers and nonlinear activation functions, making successful XOR classification the litmus test for verifying the correctness of your multi-layer network and backpropagation implementation.
3. Introduce mini-batch training and different activation functions, and validate on the MNIST handwritten digit dataset. MNIST (Modified National Institute of Standards and Technology) was released by Yann LeCun et al. in 1998, containing 60,000 training images and 10,000 test images, each a 28×28 pixel grayscale image. It has played the role of "Hello World" in machine learning history, with virtually every classification algorithm benchmarked on this dataset. Although its images are too simple—even a simple linear classifier can achieve about 92% accuracy—for the purpose of learning to implement neural networks from scratch, MNIST remains the ideal validation dataset because of its moderate size, clear labels, and easily interpretable results.
4. Finally, compare your implementation with PyTorch's results to understand what the framework automates for you.

Technical Challenges and Solutions When Implementing from Scratch

Verifying Gradient Correctness

The biggest pitfall of implementing from scratch is getting the backpropagation gradients wrong without realizing it. A professional approach is to use Gradient Checking—approximating gradients through numerical differentiation and comparing them with analytical gradients. Specifically, the numerical gradient is computed using the formula [f(θ+ε) - f(θ-ε)] / 2ε (where ε is typically 1e-5), which is the central difference method and is more accurate than one-sided differences. Compare the numerical gradient with the analytical gradient computed by backpropagation element by element, typically using the relative error |g_analytical - g_numerical| / max(|g_analytical|, |g_numerical|) as the metric. If the difference is extremely small (typically on the order of 1e-7), the backpropagation implementation is correct. Note that gradient checking is computationally expensive (requiring two forward passes per parameter), so it should only be used during debugging and disabled during training. This is an extremely important debugging technique that many beginners overlook.

Numerical Stability Issues

When implementing manually, the computation of Softmax and cross-entropy is highly susceptible to numerical overflow. For example, exponentiation can produce extremely large values that cause overflow. The industry-standard approach is to subtract the maximum value in Softmax (i.e., compute exp(x_i - max(x)) instead of exp(x_i), which is mathematically equivalent but avoids overflow), and to combine log-softmax computation (avoiding the -inf issue that arises when computing softmax first and then taking the logarithm when softmax outputs are near 0). Additionally, floating-point precision issues must be considered in practice: Python defaults to 64-bit double precision, while deep learning frameworks commonly use 32-bit single precision or even 16-bit half precision—the lower the precision, the more pronounced numerical instability becomes. These engineering details are hidden within frameworks but are problems you must confront head-on when implementing from scratch.

The Impact of Weight Initialization

The choice of initial weights has a decisive impact on training convergence. All-zero initialization causes a symmetry problem—all neurons in the same layer will compute identical gradients and make identical updates, and no amount of training can break this symmetry, essentially reducing each layer to a single neuron. Excessively large or small initial values trigger gradient explosion or vanishing. Xavier initialization (proposed by Xavier Glorot in 2010) assumes symmetric activation functions like Sigmoid/Tanh and samples weights from a distribution with mean 0 and variance 2/(n_in + n_out), where n_in and n_out are the number of input and output neurons of that layer. Its derivation is based on keeping the variance of each layer's output consistent during both forward and backward propagation. He initialization (proposed by Kaiming He in 2015) is modified for ReLU activation functions—since ReLU sets approximately half of activation values to zero, to compensate for this information loss, He initialization adjusts the variance to 2/n_in. The design principles behind these two initialization methods are something you'll only truly appreciate after encountering training convergence failures firsthand.

The Advantages of Community-Based Collaborative Learning for Deep Learning

This developer's choice to seek partners for brainstorming rather than working alone is itself commendable. Many deep learning concepts have cognitive blind spots, and when studying independently, it's easy to go further and further down a path of incorrect understanding. Community collaboration offers several key benefits:

• Multi-perspective verification: Different people may have complementary perspectives on the same concept. Some excel at mathematical derivation, others at code implementation.
• Collaborative debugging: Backpropagation bugs are often extremely subtle, and a second pair of eyes can dramatically improve debugging efficiency. For example, common errors like missing matrix transposes, misuse of broadcasting mechanisms, and accumulating instead of resetting gradients are often invisible to the person who wrote the code but obvious to an outside observer.
• Sustained motivation: Building from scratch is a lengthy process, and positive feedback from a community helps you see it through. Research shows that collaborative learning projects in open-source communities have significantly higher completion rates than individual solo projects.

Learning Resources and Recommendations

For developers who want to pursue this path, several classic resources are worth recommending: Andrej Karpathy's micrograd project implements an automatic differentiation engine in under a hundred lines of code and is an excellent model for understanding backpropagation. micrograd implements a scalar-level computational graph where each Value object records the operation and parent nodes that produced it, and completes gradient computation by topologically sorting and reverse-traversing the computational graph when backward() is called. Its elegance lies in stripping away all engineering complexity (such as tensor operations, GPU acceleration, and batching), allowing learners to focus on the core mechanism of automatic differentiation. After understanding micrograd, reading PyTorch's autograd source code reveals a high degree of architectural consistency between the two—PyTorch is essentially an industrial-grade extension of micrograd to the tensor dimension, with added CUDA backends, memory management, and extensive operator optimizations.

Karpathy's "Neural Networks: Zero to Hero" video series takes you step by step from scratch, from micrograd to GPT-level language models, progressively covering core topics like automatic differentiation, backpropagation, and language modeling. Additionally, the assignments in Stanford CS231n (Convolutional Neural Networks for Visual Recognition) require students to implement complete backpropagation in NumPy, including fully connected layers, batch normalization, Dropout, and other components, and are widely recognized as high-quality training. The course was created by Fei-Fei Li, Andrej Karpathy, and others, and its lecture notes and assignments remain the gold standard for getting started with deep learning.

It's worth noting that the goal of building from scratch is understanding, not reinventing the wheel. Once you've truly internalized the principles, returning to mature frameworks for engineering practice is the right path forward. Understanding the underlying mechanisms will make you a practitioner who is better at tuning hyperparameters, more adept at debugging, and more capable of innovation. For example, once you understand the forward and backward propagation implementation of batch normalization, you'll be better equipped to judge whether Layer Normalization or Group Normalization should be used as alternatives in specific scenarios. Once you understand Adam optimizer's first-moment and second-moment estimates, you'll understand why SGD + Momentum can sometimes achieve better generalization performance on certain tasks.

Conclusion

Building a neural network from scratch is a journey worth taking for every developer who is serious about machine learning. It's not about performance or efficiency—it's about building a complete cognitive chain from mathematical principles to code implementation. As this Reddit user advocated, if you can find like-minded partners to explore together, this learning journey will become more efficient and more enjoyable. For anyone looking to go deep into the AI field, this is a practice with an exceptionally high return on investment.