17-Year-Old Builds Deep Learning Framework Forge from Scratch in C++, Precisely Reproducing GPT-2

When PyTorch Is the Default, Someone Chose to Start from Zero

In deep learning engineering practice, PyTorch and TensorFlow have long been the default choices. The vast majority of practitioners focus on how to call high-level APIs to train models, while the underlying mechanisms of frameworks—tensor operations, automatic differentiation, memory management—are often treated as black boxes.

Tensor operations are the foundation of deep learning frameworks. Tensors are essentially mathematical abstractions of multidimensional arrays—all data in deep learning, from image pixels to text embeddings to model weights, exists in tensor form. Automatic Differentiation is the core mechanism that enables frameworks to compute gradients automatically. It works by building a computational graph that records every step of the forward pass, then uses the chain rule to backpropagate errors. PyTorch uses dynamic computational graphs (Define-by-Run), constructing the graph structure in real-time during each forward pass. This makes debugging more intuitive but also increases the complexity of the underlying implementation. Implementing an autodiff engine from scratch means defining forward computation and corresponding gradient rules for every operator, while managing the lifecycle of intermediate variables.

This is precisely why a project shared on Reddit by a self-described 17-year-old developer passionate about deep learning and systems-level programming stands out so strikingly—he built a complete deep learning framework called Forge from scratch in C++, with no dependency on any existing deep learning library.

The value of this project isn't about how impressive the performance is, but rather that it fully exposes all the underlying components required to run a modern Transformer model, while achieving an extremely difficult validation standard: after loading real GPT-2 weights, the output matches HuggingFace token-for-token exactly.

Forge Framework Core Implementation: From Tensor Engine to BPE Tokenizer

According to the author, Forge has been in development since January 2026. The math backend primarily relies on Eigen, while the author hand-wrote AVX2-based SIMD kernels for element-wise operations and uses OpenBLAS-backed GEMM for heavy matrix multiplication.

AVX2 (Advanced Vector Extensions 2) is an instruction set extension introduced by Intel in 2013 that allows CPUs to process 256 bits of data in a single instruction—that's 8 single-precision floating-point numbers simultaneously. The core idea of SIMD (Single Instruction Multiple Data) programming is data parallelism: for element-wise addition, activation functions, and similar operations, there are no dependencies between input data elements, making them naturally suited for vectorized processing. GEMM (General Matrix Multiply), on the other hand, is the most fundamental operation in linear algebra—attention computation and fully connected layers in Transformers are essentially matrix multiplications. Efficient GEMM implementations require carefully designed tiling strategies, cache locality optimization, and instruction pipeline scheduling. Libraries like OpenBLAS have accumulated decades of engineering expertise in this area, making it extremely difficult for a single person to surpass their performance.

This division of labor is sensible: element-wise operations are well-suited for SIMD vectorization, while computationally intensive tasks like matrix multiplication are delegated to mature BLAS implementations, achieving near-industrial throughput without reinventing the wheel.

Core Component List

In terms of feature coverage, Forge is far beyond a toy-level experiment:

• Custom tensor engine with its own autodiff engine and memory allocator
• Dense/Linear layers, multiple optimizers (Adam, AdamW, SGD, and SGD with momentum)
• Self-Attention, LayerNorm, Embedding
• Activation functions (sigmoid, softmax, tanh, GELU tanh approximation, ReLU, LeakyReLU)
• Loss functions (cross-entropy with fused log-softmax, binary cross-entropy with fused sigmoid, mean squared error)
• BPE tokenizer built from scratch, using GPT-2-style pre-tokenization and merge rules
• safetensors format save/load pipeline

The Underlying Logic of the BPE Tokenizer

BPE (Byte Pair Encoding) was originally a data compression algorithm, introduced to natural language processing in 2016 for subword segmentation. The BPE variant used by GPT-2 first performs pre-tokenization using regex patterns, splitting input into word-level segments, then iteratively merges the most frequently occurring adjacent character pairs at the byte level. This process is driven by a pre-trained merge rules table, ultimately converting arbitrary text into token sequences from a fixed vocabulary. Implementing BPE from scratch requires precisely reproducing the pre-tokenization regex patterns, merge priority ordering, and special token handling logic—any subtle deviation will produce different token sequences compared to the reference implementation, subsequently affecting model output.

Security Advantages of the safetensors Format

safetensors is a model weight serialization format released by HuggingFace, designed to replace the traditional pickle format. Pickle deserialization executes arbitrary Python code, posing serious security risks (such as remote code execution attacks). safetensors uses a simple binary layout: the file header stores tensor names, data types, shapes, and offset metadata in JSON format, followed immediately by raw data blocks, supporting zero-copy memory-mapped reading. This design is not only secure but also allows selective loading of specific tensors without reading the entire file. Forge's implementation of the safetensors read/write pipeline enables it to directly load pre-trained model weights published in the HuggingFace ecosystem.

Parameter System and C++ Reflection

The parameter system design deserves special mention. The author used reflect-cpp to implement a reflection-based parameter discovery mechanism: models only need to declare their structure, and Forge automatically identifies trainable parameters without manual registration.

C++, as a statically typed language, loses most type information after compilation and doesn't have runtime reflection capabilities like Python or Java. This means that in C++, a framework cannot automatically traverse a class's member variables at runtime to discover which ones are trainable parameters. reflect-cpp is a third-party library that achieves compile-time reflection using C++20 structured bindings and template metaprogramming techniques. In PyTorch, users simply inherit from nn.Module and define layers in __init__, and the framework automatically collects parameters through Python's __dict__; achieving a similar development experience in C++ requires either a reflection library or manual registration macros. Forge's choice of the reflect-cpp approach avoids large amounts of repetitive parameter registration code, demonstrating that the author gave serious thought to developer experience rather than simply piling on features.

The Hardest Part: Token-Level Exact Reproduction of GPT-2

What the author is most proud of is that after loading real GPT-2 small pre-trained weights into the GPT-2 architecture built with Forge, under greedy decoding, the output matches the HuggingFace transformers library token-for-token exactly—not similar, but precisely matching.

The significance of this achievement requires elaboration. A Transformer chains together embedding, multiple attention layers, LayerNorm, final projection, and other stages. Any numerical implementation deviation in any stage—a single incorrect transpose, a masking boundary bug—will cause the output to completely diverge after generating just a few tokens. Achieving token-for-token alignment with a reference implementation means the forward computation at every layer must be numerically correct. This is effectively an extremely rigorous end-to-end verification: it uses an observable final result to retroactively guarantee the correctness of the entire computation pipeline.

In other words, this isn't just "it runs"—it's "it runs correctly," down to bit-reproducible precision. For a framework written entirely from scratch, this speaks to engineering quality more convincingly than any benchmark number ever could.

Performance Bottlenecks and Future Roadmap

The author is quite candid about the project's current state, not shying away from its shortcomings. Currently, Forge is still a pure CPU implementation, supporting only float32 and int32, and is still filling in dtype and SIMD coverage. Performance is also "slower than expected," with the main bottlenecks concentrated in cross-entropy loss function and its gradient computation and softmax, while KV-cache has not yet been implemented.

Are the Bottleneck Assessments Reasonable?

These bottleneck identifications are well-founded. Cross-entropy combined with softmax performs normalization over the vocabulary dimension—for GPT-2, the vocabulary size is approximately 50,000. The computation and memory access overhead at this step is indeed substantial, especially without thorough vectorization.

The lack of KV-cache is the most significant constraint on current inference performance. KV-cache (Key-Value Cache) is the most critical optimization technique in Transformer autoregressive inference. During generative inference, the model generates only one new token at a time, but standard attention mechanisms require computing Keys and Values for all positions in the complete sequence. Since Keys/Values corresponding to already-generated tokens don't change in subsequent steps (a property of causal attention), KV-cache stores them, requiring only the computation of Q/K/V for the new token at each step, then concatenating with the cache. This reduces inference time complexity from O(n²) to O(n) per step. Without KV-cache, generating the 100th token still requires recomputing Keys and Values for all 99 preceding positions—an enormous waste. For GPT-2's 12-layer structure with 12 attention heads per layer, this redundant computation accumulates rapidly as sequence length grows.

The author lists a CUDA backend and these performance fixes as next steps, which is the right direction: ensure correctness first, then address throughput. This priority ordering follows the general pattern of framework development.

What This Project Means for Deep Learning Developers

Setting aside the eye-catching label of the author's age, Forge is more like an empirical demonstration of "deep understanding." Today, the barrier to entry for deep learning has been dramatically lowered by high-level frameworks, but this has also caused many people to plateau at the API-calling level. Hand-writing everything from a tensor engine, autodiff, and SIMD kernels all the way to precisely reproducing GPT-2—this process itself represents the most thorough possible deconstruction of Transformer internals.

For developers who want to go deeper, Forge's codebase provides a moderately-sized, structurally complete reference sample: it contains the key abstractions of modern frameworks (autodiff, parameter management, safetensors) without being so large that it can't be read through entirely. It won't replace PyTorch, but it proves one thing—understanding how these systems work is still something one person can accomplish from scratch.

The project is open-sourced on GitHub with release builds for Windows/Linux. The author has also posted a YouTube demo video and is publicly soliciting feedback. For a personal project still rapidly iterating, this open and self-reflective attitude is commendable in itself.