Training a 200M Parameter LLM from Scratch: A Real-World Engineering War Story

Going Against the Grain

Loading Llama 3 with Unsloth and spending an afternoon fine-tuning it has become completely routine. But one developer decided to go the opposite direction — writing a BPE tokenizer from scratch, implementing RoPE positional encoding by hand, coding up a SwiGLU feed-forward network, and building a complete training loop in pure PyTorch, without relying on any existing frameworks.

The result is EmsyAI (V4), a 196.7 million parameter language model trained on consumer-grade GPUs. This isn't another benchmark-chasing model — it's a valuable engineering field guide that honestly documents the edge cases that textbooks never tell you about, ones that only surface when you scale an architecture from 88M to nearly 200M parameters.

reddit source: Building a 200M parameter LLM from scratch (and what broke at scale)

Model Configuration: A Lean Modern Architecture

Despite its modest size, EmsyAI uses the same technology stack found in today's mainstream large models — small but complete:

Active parameters: 196.7M (~180M in Transformer blocks, ~16k vocabulary embedding layer for the rest)
Training tokens: 1.96 billion
Context window: 4,096 tokens
Hidden dimension: 1,024
Attention mechanism: GQA (16 Query heads / 4 KV heads)
FFN dimension: 2,816

The adoption of GQA (Grouped Query Attention) is particularly interesting — this is the key technique used by mainstream models like Llama 2/3 and Mistral to reduce KV cache memory during inference. Introducing this mechanism at such a small scale signals that the goal here wasn't just "get it running," but to get as close as possible to a genuine industrial-grade architecture.

GQA (Grouped Query Attention) is a middle ground between Multi-Head Attention (MHA) and Multi-Query Attention (MQA). In traditional MHA, every attention head has its own independent Q, K, and V projection matrices. MQA has all heads share a single set of K/V projections, dramatically reducing KV cache memory at the cost of expressive power. GQA divides Query heads into groups, each sharing one K/V head pair — in this model's configuration, 16 Query heads map to 4 KV heads, with every 4 Query heads sharing one K/V group. This design reduces KV cache memory to 1/4 of MHA during inference while retaining stronger modeling capacity than MQA. Llama 2 70B and the Llama 3 series were able to run longer contexts on consumer hardware with limited VRAM precisely because of GQA.

Pre-training: A Psychological Battle Against NaN

The author trained for 15,000 steps using FP16 mixed-precision. He openly admits the pre-training process was "terrifying" — constantly watching the loss curve with anxiety, dreading a sudden spike to NaN that would waste hours of compute.

Fortunately, the curve held steady, and validation perplexity converged to 5.86.

"Watching the raw outputs gradually transform from random gibberish at step 100 into recognizable Python syntax at step 10,000 was a uniquely satisfying experience."

This perhaps captures the most fundamental value of training a model from scratch: you get to witness firsthand how "intelligence" gradually emerges from noise. It's an intuitive experience you simply cannot get from calling a pre-built API.

Perplexity (PPL) is one of the core evaluation metrics for language models, measuring how "surprised" the model is by test text. Intuitively, lower PPL means the model predicts the next token more confidently and accurately. Mathematically, it equals the exponent of the cross-entropy loss on the test set: PPL = exp(loss), so loss=1.77 corresponds to PPL≈5.86. For a 197M parameter model trained on only 1.96B tokens, a validation perplexity of 5.86 is within a reasonable range — GPT-2 (117M parameters, 40GB data) scores around 18–29 on WikiText-103, while modern small models trained on more data can achieve below 3. Perplexity follows a power-law relationship with both model scale and training data volume, which is exactly the core principle described by the Chinchilla scaling laws.

FP16 mixed-precision training means using half-precision floating point (16-bit) for forward passes and gradient computation to save memory and accelerate computation, while keeping a "master copy" of model parameters and optimizer states (such as Adam's first and second moments) in FP32 to avoid precision loss causing training instability. NaN (Not a Number) typically arises from overflow or gradient explosion, and occurs more easily in FP16 because its numerical range is far smaller than FP32 (max ~65,504 vs. ~3.4×10³⁸).

The Deep Insight Behind a HumanEval Score of Zero

After pre-training, the author used a LoRA adapter with only 2.3 million parameters to fine-tune on the CodeAlpaca dataset, then ran OpenAI's HumanEval code benchmark.

The result: 0.0%.

This sounds dismal, but the author's analysis is arguably the most valuable part of the entire write-up. He points out that for a small model trained on only 2 billion tokens, being unable to solve multi-step algorithmic problems is exactly the expected baseline performance.

A Counterintuitive Finding About Data Contamination

More intriguing is what the author found when he audited the training data. Using the standard 13-gram exact match threshold from the GPT-3 paper, he discovered 1,644 sequences in CodeAlpaca that leaked HumanEval test logic.

In other words, the model had actually "seen" some of the answers during fine-tuning. Yet it still scored zero. The author's interpretation is clear-eyed:

"This doesn't prove it's a generalization genius — it just means this tiny 2.3M parameter LoRA adapter simply didn't have the capacity to memorize those sequences verbatim. Data contamination couldn't help it cheat because it couldn't remember the answers in the first place."

This observation carries a warning for the entire industry: the harm caused by data contamination scales directly with the model's memory capacity. Large models can "cheat" on contaminated benchmarks precisely because they have enough parameters to memorize answers; capacity-constrained small models can't "benefit" even when exposed to contaminated data. This is also a reminder that benchmark scores for large models must always be scrutinized for training set leakage.

N-gram exact matching is one of the most commonly used methods for detecting data contamination. The 13-gram threshold used in the GPT-3 paper means: if any continuous sequence of 13 tokens in the training set is an exact match to something in the evaluation set, that sample is flagged as a potential leak. The number 13 is considered long enough to rule out random coincidence, yet short enough to catch genuine text overlap. The limitation of this approach is that it can only detect literal repetition — it cannot detect semantically equivalent but differently worded contamination, such as code with renamed variables. More rigorous contamination detection methods include embedding similarity matching and model perplexity comparison, but these come with significantly higher computational costs.

Three Things That "Broke" When Scaling to 200M Parameters

When the architecture scaled from 88M to 196M parameters, a series of issues that were completely hidden at smaller scales came to the surface. This section contains the highest-value engineering lessons in the entire write-up.

Attention Logit Explosion

At a hidden dimension of 1,024, attention logits would occasionally spike dramatically. The standard architecture does not normalize Query and Key vectors before computing dot products, which becomes a serious stability risk as scale increases.

The author initially thought models like Qwen 2.5 and Gemma 2 had already fixed this problem directly, but later realized he had it wrong — Gemma 2 uses "attention logit soft-capping." What actually established QK-Norm (normalizing Q/K vectors) as a widely recognized training stability technique was OLMo 2, followed by Gemma 3 and Qwen 3 drawing inspiration from Meta's Chameleon.

V5 will introduce QK-Norm to permanently eliminate these spikes. This technical lineage tracing is valuable in itself — it shows how architectural improvements actually propagate through the open-source community.

QK-Norm refers to applying RMSNorm or LayerNorm separately to Query and Key vectors before computing attention weights. Standard Transformer attention scales dot product results by dividing by √d_k (the square root of the head dimension), but as model scale and training depth increase, this static scaling is insufficient to prevent extreme values from appearing in the logit distribution at certain layers or training steps. This causes softmax saturation (outputs approaching one-hot), vanishing gradients, and ultimately loss spikes or NaN. QK-Norm fundamentally limits logit magnitude by dynamically constraining the L2 norm of Q/K vectors, and represents one of the important advances in large model training stability engineering in recent years. Meanwhile, the "soft-capping" strategy used by Gemma 2 applies a tanh function to logits after computation, clipping them to a fixed range — a different engineering path that is not fully equivalent to QK-Norm.

Tokenizer Byte Collision

When writing the BPE tokenizer by hand, the author made an error in the fallback byte mapping: bytes 0–3 overlapped with special control tokens (such as <|endoftext|>). This bug doesn't cause training to crash — it's a silent performance drag. V5 will completely rebuild the tokenizer and double the vocabulary size from 16k to 32k.

Cross-Document Attention Pollution

The current data loader simply concatenates text files to fill the 4,096-token context length. This means the model wastes significant compute attending to documents that happen to land in the same training window but have completely unrelated content.

The fix is to implement document-aware packing with proper attention masking, ensuring attention cannot cross document boundaries. This is also a detail that many developers training LLMs from scratch tend to overlook.

Run a 200M Parameter Model Yourself

The author exported the final weights in GGUF format, so anyone can run it in Ollama and see firsthand how a 196M model "tries (and fails) to write FizzBuzz":

git clone https://github.com/gulding/EmsyAI.git
cd EmsyAI
huggingface-cli download gulding/EmsyAI emsyai-v4-instruct-f32.gguf --local-dir .
ollama create emsyai-v4 -f Modelfile
ollama run emsyai-v4

The complete PyTorch training loop, architecture, and tokenizer code are all open-sourced on GitHub.

Why This Kind of Experiment Matters

In an era where anyone can fine-tune with a single click, writing a large model from scratch might seem like "reinventing the wheel" — but its value lies precisely in exposing all the complexity that frameworks hide from you. The stabilizing effect of QK-Norm, the relationship between data contamination and model capacity, the hidden waste of cross-document attention — these are pain points you only truly internalize when you've built everything from the ground up.

For developers who want to deeply understand LLM internals, there's no substitute for going through these struggles yourself rather than just calling APIs. After all, the tension and satisfaction of watching a loss curve teeter on the edge of NaN and finally hold steady — that's where real understanding of these systems begins.