Writing Deep Learning from Scratch: Can Anyone Really Do It Without a Framework?

Deep learning frameworks (PyTorch, TensorFlow, JAX) have become the infrastructure of AI development, but a question that sparked heated discussion on Reddit hits a nerve for many: Do you truly understand the math behind deep learning well enough to cast aside all DL libraries and implement it from scratch?

This isn't just a show-off question—it touches on a real divide in today's AI field: are we actually "using" deep learning, or do we "understand" it?

Why This Question Resonates

The original post's phrasing is subtle: "if you had the time and motivation"—it's not asking whether you remember all the formulas, but whether the underlying computational logic has been internalized to the point where you could independently reproduce it.

The implication of this condition is: most practitioners are actually in a state of "knowing the what but not necessarily the why." You can get a model running by calling libraries, but the moment you need to implement custom backpropagation, hand-write an attention mechanism, or optimize parameters without automatic differentiation, you get stuck.

Why Can't Most People Do It?

Frameworks Abstract Away Three Layers of Complexity

Modern DL libraries hide at least three things for you: automatic differentiation (autograd), low-level scheduling of tensor operations (GPU kernels, memory management), and numerical stability handling (softmax overflow, gradient clipping, mixed precision). Remove any one of these three layers, and the difficulty of a hand-written implementation jumps by orders of magnitude.

Automatic differentiation (autograd) is one of the most core abstractions in DL frameworks. It doesn't use symbolic derivation to find gradient formulas (that's what SymPy does), nor does it approximate with finite differences (numerically unstable and computationally linear in the number of parameters). Instead, it leverages the chain rule to record a computation graph during the forward pass, then traverses this graph in reverse to obtain exact, efficient gradients for all parameters. PyTorch uses define-by-run dynamic graphs, TensorFlow was static-graph early on and supports eager mode since 2.x, and JAX performs function transformations on pure functions via grad. The key to understanding autograd is distinguishing "forward values" from "adjoint values": each operator must simultaneously define how it maps gradients coming from upstream back to its own inputs. Backpropagation is essentially a reverse topological traversal of this graph.

Mathematical Understanding ≠ Engineering Implementation

Knowing how to derive the chain rule and efficiently doing batched matrix derivatives with NumPy, handling broadcast dimensions properly, and ensuring gradient numerical stability are two different skill sets. Stanford's CS231n course early on required students to hand-write a two-layer neural network in NumPy, and many people truly understood backpropagation for the first time at this step—not in a formula sense, but in the sense of "which line of code corresponds to which computational step."

CS231n (Stanford's "Convolutional Neural Networks for Visual Recognition") has long been regarded as a classic introductory deep learning course. Its Assignments 1 and 2 require students to implement linear classifiers, two-layer neural networks, batch normalization, and convolutional layers in NumPy—completely without frameworks. The design intent of these assignments isn't to train engineering skills, but to force students to personally experience the full pipeline of "forward pass—loss computation—backward pass—parameter update." Many people get stuck when deriving the gradient of softmax cross-entropy, because it involves the coupling of the Jacobian with probability normalization; and implementing the backward pass of batchnorm is an even greater test of fine-grained mastery of the chain rule. This "line-by-line correspondence" experience is precisely the cognitive turning point that framework users struggle to obtain.

The Practical Value of Hand-Written Implementation

Experienced practitioners generally agree that implementing from scratch at least once pays off:

• A qualitative leap in debugging ability: When loss isn't decreasing, you can determine whether it's a gradient computation error, a learning rate issue, or a data pipeline problem—rather than blindly switching framework versions.
• Confidence in custom layers: No more fear when you need to implement a new operator or modify backpropagation logic.
• Deeper penetration into papers: When reading a paper on a new architecture, you can directly reconstruct the computation graph in your head.

Is It Realistic to Use No Libraries at All?

The original post asks "can you," not "should you." From an engineering perspective, training a production-grade model without any libraries is almost unrealistic—you need CUDA, cuBLAS, distributed communication libraries, and these themselves constitute low-level dependencies.

But narrow the question down to "implementing a trainable Transformer in NumPy on CPU" or "hand-writing an MLP and getting it to run on MNIST"—this is entirely feasible, and many educational projects do exactly this. The difference is: an understanding-oriented implementation pursues correctness and clarity, not performance.

CUDA is NVIDIA's parallel computing platform, and cuBLAS is its underlying basic linear algebra library. Together they form the foundation for GPU computation in virtually all DL frameworks. When you write torch.matmul in PyTorch, what actually executes is often a highly tuned cuBLAS kernel; and distributed training involves NCCL (a collective communication library) responsible for gradient synchronization across multiple GPUs. This means "using no libraries at all" in practice is equivalent to "rewriting the entire computational stack from the assembly or C level"—nearly impossible. In educational contexts, "from-scratch implementation" typically draws the boundary at the NumPy layer: using it only for tensor storage and basic operations, while all automatic differentiation, layer structures, and optimizers are still written by yourself—this preserves the learning value while avoiding reinventing the wheel.

A Tendency Worth Being Wary Of

Treating "can implement from scratch" as the sole measure of ability is unreasonable. The value of deep learning ultimately lies in solving practical problems—rapidly validating ideas with frameworks, tuning hyperparameters, and deploying are equally core skills.

The real question is: Do you know what the abstraction layers you depend on are doing? If the answer is no, then when you encounter edge cases that frameworks can't directly solve, you may find yourself completely stuck.

Rather than agonizing over "can you do it without libraries," set the standard as: at least once, fully implement from scratch a component you use every day—even if it's just a fully connected layer with backpropagation. The cognitive gains from this process are far more important than the final code's benchmark scores.

Abstraction layer is a core concept in software engineering, referring to the design of encapsulating underlying complexity and exposing only a clean interface. Every evolution of deep learning frameworks—from Theano to TensorFlow to PyTorch and JAX—is essentially about adding abstractions at different levels. The benefit of abstraction is productivity; the cost is "abstraction leakage": when system behavior deviates from expectations, users who don't understand the underlying layers are often helpless. Distributed training stuck at all-reduce, mixed precision causing gradient overflow, custom operators not being supported—these are all typical scenarios of abstraction leakage. A healthy engineer both enjoys the dividends of abstraction and retains the ability to penetrate the underlying layers on critical paths—this is what the standard of "can implement from scratch" truly aims to express.

One-line Summary

Discussing whether one can write deep learning from scratch without DL frameworks, weighing understanding against engineering practice.

Paragraph Summary

The article revolves around the hotly debated Reddit question "Can you cast aside frameworks like PyTorch, TensorFlow, and JAX to write deep learning from scratch," pointing out that most people are in a state of "knowing the what but not the why." The reason is that frameworks hide three layers of complexity—automatic differentiation, low-level tensor scheduling, and numerical stability—and mathematical derivation and engineering implementation are two different skills. The author argues that implementing from scratch at least once is valuable for the qualitative leap in debugging ability, confidence in custom layers, and penetration into papers—not as a replacement for frameworks. Training production models without any libraries is unrealistic, but implementing a trainable MLP or small Transformer in NumPy is entirely feasible. Be wary of treating "can implement from scratch" as the sole standard; the key is to understand what the abstraction layers you depend on are doing.