Implementing Tensor Operations and Automatic Differentiation from Scratch in C++: Deconstructing PyTorch's Internals

Why Build a Deep Learning Framework from Scratch in C++

When we're used to writing import torch and calling loss.backward(), few of us ever stop to think about what's really happening under the hood. PyTorch is an open-source deep learning framework developed by Meta (Facebook), released in 2016. Known for its dynamic computation graphs (define-by-run) and Pythonic interface, it quickly became one of the most popular deep learning tools in both academia and industry. However, behind its ease of use lies a complex underlying implementation: a C++ core engine, CUDA acceleration, an automatic differentiation system, and more. When a developer calls loss.backward(), the framework automatically handles computation graph construction, gradient calculation, and parameter updates — a level of encapsulation that makes it difficult to see the internal mechanics.

One developer (GitHub user mechanical-turk) decided to build a tensor operation library and automatic differentiation engine from scratch in C++, with the explicit goal of opening PyTorch's black box — to see exactly what the framework does when you call a single line of code. While this "black box" nature boosts development efficiency, it also leaves many developers with only a surface-level understanding of deep learning fundamentals.

The project, called "Deep Learning All The Way Down," isn't meant to replace mature frameworks — it's a deep learning exercise in the truest sense. The author has open-sourced the complete code with Git checkpoints and recorded the entire process as a video series, currently up to Episode 7. This "build while you explain" approach is incredibly valuable for developers who want to truly understand deep learning internals.

C++ Tensor Implementation: From Flat Storage to Broadcasting

Tensors are the core data structure of every deep learning framework. The author's C++ tensor implementation already supports a fairly complete set of features:

• Flat storage: The underlying data is stored in a one-dimensional contiguous array — the same approach used by PyTorch, NumPy, and other mainstream libraries. Flat storage is the standard practice for high-performance numerical computing libraries. Multi-dimensional arrays may logically be 2D, 3D, or higher-dimensional structures, but in physical memory they must be stored as contiguous 1D sequences. This design has three key advantages: first, contiguous memory layout maximizes CPU cache hit rates, as modern processor prefetching mechanisms load adjacent data together, significantly improving access speed; second, it simplifies memory allocation and deallocation, requiring only a single malloc/free operation; and third, it facilitates integration with low-level math libraries like BLAS and LAPACK, which assume contiguous data storage.

• Multi-dimensional indexing: On top of flat storage, stride-based computation maps multi-dimensional coordinates to linear offsets. By maintaining a stride array that records the step distance for each dimension, operations like multi-dimensional indexing, transposition, and slicing can be performed efficiently — often without copying data at all.

• Elementwise operations: Basic arithmetic like addition, subtraction, multiplication, and division.

• Reductions: Dimension-reducing operations such as sum and mean.

• Broadcasting: A mechanism that automatically aligns tensors of different shapes for arithmetic operations. Broadcasting was a concept introduced by NumPy in 2005 and later adopted by all major deep learning frameworks. It defines a set of rules allowing arrays of different shapes to participate in arithmetic. The core rule: align dimensions from the rightmost side; if a dimension has size 1 or is missing, it is automatically "stretched" to match the other array. For example, when adding an array of shape (3, 1) to one of shape (3, 4), the (3, 1) array is broadcast along the second dimension to (3, 4). This mechanism is ubiquitous in deep learning: subtracting means during batch normalization, mask multiplication in attention mechanisms, loss function computation, and more. Implementing broadcasting requires complex shape inference logic and stride adjustments — it's extremely common in deep learning but quite tedious to implement.

• Rank-two matmul: One of the most fundamental computations in neural networks.

• Mean Squared Error (MSE): A common loss function implementation.

From this feature list, it's clear the author has built a tensor system capable of supporting basic neural network forward computation. These seemingly basic capabilities are the foundation of any deep learning framework.

Scalar Autograd Engine: The Core of Backpropagation

The project's most recent highlight is a standalone scalar reverse-mode automatic differentiation (autograd) engine. Automatic differentiation (AD) comes in two flavors: forward mode and reverse mode. Deep learning universally uses reverse mode because it can compute the gradient of a scalar loss with respect to all parameters in O(n) time, whereas forward mode requires O(n) forward passes. Reverse mode is based on the chain rule of calculus: if z=f(y) and y=g(x), then dz/dx = (dz/dy) × (dy/dx). In a computation graph, each operation node stores its local derivative; during backpropagation, starting from the output node, the upstream gradient is multiplied by the local derivative and passed layer by layer back to the input nodes. The workflow clearly reproduces the core ideas of modern automatic differentiation.

Computation Graph Construction and Backpropagation Flow

1. Building the computation graph during forward pass: Arithmetic operators dynamically record dependencies between operations as they execute, forming a computation graph. The computation graph is the core abstraction of deep learning frameworks: nodes represent operations or variables, and edges represent data dependencies. Static graphs (TensorFlow 1.x) are fully constructed before execution; dynamic graphs (PyTorch) are built on-the-fly during computation.

2. backward() generates a topological sort: Starting from the output (loss) node, the entire graph is topologically sorted. Backpropagation needs to visit nodes in reverse dependency order — exactly what topological sorting provides. Topological sorting is a classic algorithm for directed acyclic graphs (DAGs), commonly implemented with Kahn's algorithm or depth-first search. In automatic differentiation, correct topological ordering guarantees proper gradient computation order: child node gradients must be fully computed before parent nodes.

3. Reverse traversal of the graph: Nodes are visited in reverse topological order.

4. Applying local derivatives: Each operation applies its local gradient according to the chain rule.

5. Gradient accumulation: When a value influences the final loss through multiple paths, gradients from different paths are correctly summed. Gradient accumulation is critical in automatic differentiation: when a variable affects the loss function through multiple computation paths, gradients from each path must be added together. For example, if x both directly participates in the loss computation and indirectly influences loss through an intermediate variable y, then dL/dx = (dL/dx)_direct + (dL/dy) × (dy/dx). The additive form of the chain rule ensures this.

This last point is especially critical — gradient accumulation is at the heart of autograd correctness, and it's the detail beginners most often overlook.

Gradient Computation Verification Example

The author provides a concise example:

Value prediction = w1*x1 + w2*x2 + w3*x3 + bias;
Value residual = prediction - target;
Value loss = residual * residual;
loss.backward();

Given weights [0.5, -1.0, 2.0], inputs [4.0, 3.0, 2.0], bias 0.5, and target 2.5:

• Forward pass yields prediction 3.5, loss 1.0
• Backward pass recovers gradients: dL/db = 2, dL/dw = [8, 6, 4]

We can verify by hand: the residual is 3.5 - 2.5 = 1.0, the derivative of the loss with respect to the residual is 2 × 1.0 = 2.0 (i.e., dL/db), and the derivative with respect to each weight is 2.0 × corresponding input, giving [8, 6, 4]. The results match perfectly. This hand-verifiable design helps learners build intuitive confidence in gradient computation.

Integrating Tensors with Autograd: An Unresolved Design Challenge

Currently, the scalar autograd engine and the tensor implementation are two independent systems. The author's next step is to integrate graph identity, ownership, and gradient information into the tensor, thereby building a complete training loop.

To that end, the author posed a question to the community that reflects deep engineering insight:

For integrating tensors with autograd, should autograd metadata live inside each Tensor handle, or should tensors point to independent shared computation graph nodes?

This touches on a fundamental design decision in deep learning frameworks:

• Metadata embedded in tensors: Straightforward to implement, with tensors tightly coupled to their gradient information. However, managing graph ownership becomes complex when tensors are copied, shared, or viewed.

• Tensors pointing to shared graph nodes: This is closer to PyTorch's actual design. In PyTorch, Tensors are associated with the computation graph through an AutogradMeta structure. AutogradMeta contains pointers to Nodes (graph nodes), gradient Tensors, version counters, and other information. This design decouples tensors from their computation history: multiple Tensors can share the same Node (e.g., views before and after in-place operations), and Node lifetimes are managed by reference counting, independent of Tensor lifetimes. When .detach() is called, the new Tensor's AutogradMeta is empty, severing its connection to the computation graph. This approach handles the case of multiple tensors sharing the same computation history more elegantly, but introduces the complexity of reference counting and lifetime management.

PyTorch itself chose a variant of the latter: Tensors are linked to the computation graph via AutogradMeta, and graph nodes live independently of tensors. For a learning-oriented project, understanding the trade-offs behind this decision is itself an extremely valuable takeaway.

Takeaways for Deep Learning Developers

The greatest value of this project isn't whether it can match PyTorch's performance — it's that it demystifies deep learning frameworks. By implementing flat storage, broadcasting, topological sorting, and gradient accumulation by hand, the author (and those following the video series) build a first-principles understanding of framework internals.

For anyone who truly wants to master deep learning, "building a toy PyTorch" is a proven, repeatedly validated learning path. micrograd is a minimalist autograd engine developed by Andrej Karpathy (founding team member of OpenAI, former head of Tesla AI), with core code under 150 lines of Python. It implements scalar-level backpropagation and supports building simple neural networks. Karpathy uses it as a teaching tool with an accompanying video course "building neural networks from scratch," helping learners understand the essence of autograd. Similar projects include tinygrad (by George Hotz, achieving full GPU acceleration in about 1000 lines). The value of these "toy frameworks" lies in stripping away engineering complexity to expose the mathematical core of deep learning. By reproducing these projects, developers gain deep understanding of dynamic computation graphs, the chain rule, and gradient descent — rather than staying at the level of API calls.

Building in C++ rather than Python adds an extra dimension, forcing developers to confront harder systems issues like memory management, ownership, and performance.

The project code and Git checkpoints are open-sourced on GitHub, and the companion video series continues to be updated. For developers who want to build their own version, this serves both as a runnable reference implementation and a window into the thought process behind "reinventing the wheel."

Key Takeaways

• This project implements tensor operations and automatic differentiation from scratch in C++, aiming to reveal the internal mechanics of frameworks like PyTorch
• The tensor implementation uses flat storage, stride-based indexing, and broadcasting — core design patterns shared by all major frameworks
• The scalar autograd engine demonstrates the complete flow of computation graph construction, topological sorting, and gradient accumulation
• The project's core design challenge — how to integrate tensors with the computation graph — reflects the real engineering trade-offs in production frameworks
• Building first-principles understanding of deep learning through "reinventing the wheel" is a learning path repeatedly validated by experts like Karpathy