Reproducing GPT-2: How a Hidden Bug Affected Weight Training

Introduction: When Reproduction Meets a Gap

In the era of large models, reproducing classic models from scratch is an important way for many researchers and engineers to verify their understanding and test their engineering implementations. OpenAI's GPT-2, as a landmark language model with publicly available weights and architecture, has become a practice target for countless learners. However, a seemingly simple question has troubled quite a few people: Why do weights trained with the same architecture consistently underperform compared to OpenAI's officially released version?

GPT-2 was released by OpenAI in February 2019 as an autoregressive language model based on the Transformer decoder architecture. It has 1.5 billion parameters (largest version), was trained on the WebText dataset, and was famously delayed in full release due to "safety concerns" over its generation quality. GPT-2 uses a decoder-only Transformer architecture with 48 layers (1558M version), 25 attention heads per layer, and an embedding dimension of 1600, processing a vocabulary of 50,257 tokens using a Byte Pair Encoding (BPE) tokenizer. Its architectural choices—such as placing LayerNorm before sublayers (Pre-LN) rather than after (Post-LN), and using learned positional embeddings instead of sinusoidal encoding—became design paradigms for subsequent models.

The article titled "Why do OpenAI's GPT-2 weights beat mine? Part two: the bugfix" is an in-depth investigation centered around this question. In this second installment, the author ultimately pinpoints a bug hidden in the training pipeline and provides a fix. While such debugging processes may seem tedious, they often contain the most authentic insights into deep learning engineering practice.

The Challenges of Reproducing GPT-2: Details Determine Success or Failure

Reproducing a published model may appear to simply require copying the architecture and loading the same hyperparameters. But in practice, performance gaps often stem from "tacit knowledge" that never makes it into the paper.

Where Does the Gap Come From?

In large model training, even with an identical architecture, the following factors can all lead to differences in final weight quality:

• Data preprocessing details: Tokenizer implementation, special token handling, and text cleaning strategies. GPT-2's BPE tokenizer is based on byte-level encoding, first converting text to UTF-8 byte sequences and then performing merge operations. This design allows the tokenizer to handle any Unicode text without producing unknown tokens. However, GPT-2's BPE implementation has a specific detail: it uses a regex to pre-split text, preventing cross-word merges. The exact form of this regex, how spaces are handled (GPT-2 adds a space before words as part of the token), and how special tokens like <|endoftext|> are processed can all produce subtle differences when reimplemented. Even minor deviations in tokenization results will accumulate and amplify throughout the entire training process.

• Weight initialization: Different initialization schemes affect the convergence path. GPT-2 uses a special initialization strategy—projection layer weights before residual connections are scaled by $1/\sqrt{N}$ (where N is the number of residual layers) to maintain stable signal propagation early in training.

• Training hyperparameters: Learning rate schedule, warmup steps, gradient clipping threshold.

• Numerical precision: Accumulated precision loss in mixed-precision training. Modern large model training almost universally uses mixed-precision training, performing forward and backward passes in FP16 or BF16 to accelerate computation and save memory while maintaining an FP32 master copy of weights for parameter updates. FP16 has limited dynamic range (minimum positive number ~6×10⁻⁸, maximum ~65504) and is prone to underflow and overflow, requiring loss scaling techniques. BF16, while having the same dynamic range as FP32, has lower precision (only 7 mantissa bits vs. FP32's 23 bits). In numerically sensitive operations like Softmax computation and LayerNorm variance calculation, different precision choices can lead to subtle differences in training dynamics.

• Bugs in implementation details: This is the core focus of this article.

The author had already investigated multiple possibilities in part one, but the performance gap persisted. This drove him into part two—a deeper code-level investigation.

Locating the Bug: From Symptoms to Root Cause

Methodology for Debugging Deep Learning Bugs

Locating bugs in deep learning training is extremely challenging because many issues don't cause program crashes but instead "silently" degrade model performance. These types of bugs typically manifest as:

1. Training loss appears to "decrease normally" but converges to a suboptimal point
2. The model can generate reasonable output, but quality is consistently one notch below
3. Intermediate tensors don't match the reference implementation

The debugging approach the author likely adopted was layer-by-layer comparison of intermediate activations—comparing forward pass results from their own implementation with outputs from OpenAI's official weights on the same input, identifying the first point where divergence occurs. The specific approach is: prepare a fixed input batch, run forward passes through both the reference implementation and your own implementation, record tensor values at each layer's output, then compute the difference between them (typically using maximum absolute error or relative error). In a correct implementation, minor numerical differences (on the order of 1e-6) are normal due to the non-associativity of floating-point operations; but if a significant deviation suddenly appears at a certain layer (such as 1e-2 or larger), it indicates an implementation error at that layer or somewhere before it. This method can also be combined with backpropagation—comparing gradient tensors—to locate bugs that affect training but not inference. PyTorch tools like register_forward_hook can conveniently extract intermediate layer outputs without modifying model code. This "binary search localization" method is the standard approach for debugging neural network implementations.

What the Bug Itself Reveals

The subtitle "the bugfix" itself tells the story: the author ultimately found a specific, fixable implementation error. Common types of such bugs include:

• Incorrect application of attention masks: Direction or position offset in the causal mask. In autoregressive language models, the causal mask ensures that position i can only attend to tokens at positions 0 through i, typically implemented via a lower triangular matrix. If the mask's dimensions, starting position, or boolean logic are off, the model might "peek" at future token information (causing artificially low training loss but degraded inference), or fail to attend to certain legitimate positions (causing information loss).

• Indexing errors in positional encoding: Off-by-one type errors. GPT-2 uses learned positional embeddings—a learnable matrix of shape [max_seq_len, d_model]. If indexing starts from 1 instead of 0, or if positions are shifted by one when handling padding, the model's utilization of positional information will have a systematic bias.

• Misplaced LayerNorm: Confusion between Pre-LN and Post-LN. The original Transformer paper uses Post-LN structure, i.e., output = LayerNorm(x + Sublayer(x)), while GPT-2 uses Pre-LN structure, i.e., output = x + Sublayer(LayerNorm(x)). This seemingly minor difference has a huge impact on training stability—Post-LN is prone to vanishing gradients in deep networks, while Pre-LN keeps gradients stable along the residual path. Additionally, GPT-2 adds an extra LayerNorm after the final Transformer block, a detail easily overlooked in some architecture descriptions.

• Weight transpose or dimension ordering issues: Shape mismatches when loading official weights. Different frameworks (TensorFlow vs PyTorch) have different conventions for storing convolution kernel and fully connected layer weights. GPT-2's original implementation is based on TensorFlow, and transposition relationships must be carefully handled when converting to PyTorch.

Any single one of these details is sufficient to produce a significant gap between the reproduced results and the original.

Why This Type of Debugging Article Is Valuable

"Tacit Knowledge" in Engineering Practice

The significance of this article far exceeds its surface-level technical content. It reveals a widely overlooked reality: there is an enormous chasm between the architectural description in a paper and a reproducible engineering implementation.

Extensive research shows that the reproducibility crisis in deep learning is quite severe. The NeurIPS 2019 reproducibility challenge revealed a concerning reality: even when paper authors provide code, the rate at which independent researchers successfully reproduce claimed results is far lower than expected. A 2020 survey found that approximately 63% of papers in machine learning face reproducibility difficulties. The causes are manifold: the impact of random seeds on results (some methods only work with specific seeds), unreported hyperparameter tuning processes, hardware differences (different GPUs may produce different floating-point results), framework version differences (default behaviors may change across PyTorch versions), and most importantly—those implementation details "too obvious to be worth writing down." This reproducibility crisis has driven the rise of experiment tracking tools like MLflow and Weights & Biases, as well as practices like Dockerized training environments.

Even with open-source code, different runtime environments, library versions, and random seeds can cause result deviations. And when you need to implement from scratch, those implementation tricks tucked away in corners of the original author's codebase are often the true source of performance gaps.

Lessons for Deep Learning Engineers

For engineers who want to deeply understand Transformers and large language models, these "reproduction pitfall" articles are more educational than polished papers:

1. Establish reference baselines: Always use the official reference implementation as ground truth. This means not only comparing final outputs but also ensuring you can load official pretrained weights into your own implementation, verifying that inference results are completely consistent before beginning training.

2. Prioritize intermediate verification: Don't just look at the final loss—verify tensor values layer by layer. You can write automated tests to continuously verify the numerical correctness of each layer in your CI pipeline.

3. Beware of silent failures: Bugs that degrade performance are harder to detect than bugs that crash. A bug that raises loss from 3.0 to 3.1 might lurk for weeks before being discovered, by which time vast computational resources have been wasted.

4. Maintain a skeptical attitude: When results fall short of expectations, suspect your own implementation first. Don't rush to conclude that "the method doesn't work" before ruling out your own bugs.

Conclusion: Learning Through Reproduction

Reproducing a classic model like GPT-2 from scratch is essentially a battle against details. The reason OpenAI's weights are "better" is usually not because of some mysterious black magic, but because every single detail in their implementation was handled correctly.

The debugging journey documented in this article reminds us that in large model engineering, the devil is in the details. A single offset in positional encoding or an error in masking is enough to undermine a carefully trained model. And the process of finding and fixing these bugs is the essential path from "knowing how to use" to "truly understanding."

For every technical practitioner deeply invested in the era of large models, this spirit of digging to the bottom and tracing back to the source may be more valuable than any ready-made model weights. It's worth noting that open-source projects like Andrej Karpathy's nanoGPT and Eleuther AI's GPT-NeoX only achieved performance close to official implementations after going through similar iterative debugging processes. The commit histories of these projects are themselves living "pitfall manuals," documenting countless painful climbs out of subtle bugs.