Training LLaMa from Scratch: A Full Record of 11-Hour Training for a 125M Model

A developer shared on Reddit their experience training a LLaMa model from scratch. They not only fully reproduced LLaMa's core architectural components but also progressively advanced from a tiny character-level dataset to large-scale corpus training, ultimately completing the training of a 125M-parameter model on dual A5000 GPUs. Throughout the entire process, they did not rely on any existing training framework but manually implemented every key module.

Core Architecture: Manually Implementing LLaMa's Four Key Components

LLaMa's important position among open-source models is inseparable from several carefully designed architectural choices. This developer did not call existing libraries but implemented each of these modules one by one:

• RoPE (Rotary Position Embedding): Encodes positional information using rotation matrices. Compared with traditional absolute position encoding, it performs more stably when handling long sequences.
• Grouped Query Attention (GQA): A compromise between multi-head attention and multi-query attention, where multiple query heads share a set of key-value heads. This compresses the KV cache while preserving expressiveness.
• RMSNorm: Replaces LayerNorm with root mean square normalization, resulting in simpler computation and better numerical stability during training.
• SwiGLU: The activation function uses a Swish variant plus a gated linear unit, which typically yields better language modeling performance at the same parameter count.

These components are now standard in LLaMa-family models, but the point of manually reproducing them is this: only by writing the batched inference logic for the KV cache yourself can you truly understand the trade-off between memory usage and throughput during inference.

KV cache is a key optimization technique in the inference phase of large language models. In autoregressive generation, the model needs to recompute attention over all historical tokens each time it generates a new token. Without caching, the computation grows quadratically with sequence length. The KV cache approach stores the Key and Value tensors already computed at each layer, so that subsequent generation only needs to compute the Query for the new token and concatenate it with the cached KV, reducing the per-step computational complexity from O(n²) to O(n). The cost is that memory usage grows linearly with sequence length and number of layers; especially in large-batch, long-context scenarios, the KV cache often becomes a memory bottleneck. GQA is designed precisely to alleviate this pressure—by having multiple query heads share a set of KV heads, it directly compresses the cache size. This is also the key to understanding LLaMa's inference efficiency.

Three-Step Training Strategy: From Validation to Scaling

The developer's training path was designed quite pragmatically. Rather than jumping straight to a large model, they validated things progressively in three stages:

Step 1: tiny-shakespeare character-level training. This is a classic "smoke test" dataset, extremely small in scale and character-level. The goal is to confirm that there are no bugs in the basic components such as forward propagation, backpropagation, and loss decrease. Many beginners skip this step and go straight to large models, only to find that when training fails to converge, they cannot tell whether the problem is in the architecture or the data.

Step 2: TinyStories-v2, a 41M-parameter model. TinyStories is a synthetic story dataset released by Microsoft, designed specifically for small models, with simple grammar but complete semantics. Training a 41M-parameter model on this dataset can verify whether the model has basic language generation ability while keeping training costs manageable.

Step 3: FineWeb, a 125M-parameter model. FineWeb is a large-scale, high-quality web text dataset released by HuggingFace. The final model has 125M parameters, trained on 5.27B tokens, using 2 A5000 GPUs, with a training duration of 11 hours. This configuration is above average among individual developers—a single A5000 has 24GB of memory, and dual cards support only a limited model scale and batch size, but it is sufficient for completing a full pretraining pipeline.

According to the developer, they referred to Umar Jamil's video tutorial on LLaMa during the learning process. That tutorial is known for explaining the architecture implementation line by line and has a good reputation in the community.

From Pretraining to Alignment: The Next Step Is RLHF

The developer mentioned that they have also recently been learning and implementing reinforcement learning algorithms, and their next project is planned to be RLHF (Reinforcement Learning from Human Feedback). However, they admitted that they first need to confirm how large a model their GPU can accommodate—RLHF usually requires simultaneously loading the policy model, reference model, reward model, and value function model, so memory pressure is far greater than in pure pretraining.

This concern is very practical. Pretraining a 125M-parameter model can be completed in 11 hours on dual A5000s, but the RLHF pipeline's memory requirements increase severalfold. If using the same hardware, they may need to consider parameter-efficient fine-tuning, model quantization, or a smaller base model.

What Is the Value of Personal Reproduction?

Today, when open-source model weights can be downloaded at will, what is the point of training a LLaMa from scratch? This developer's practice offers one answer: the training process itself is the best way to learn.

Reproducing architectural components helps one understand the trade-offs behind each design decision; the staged training strategy cultivates engineering intuition—knowing what scale of data matches what scale of model; and personally handling batched inference with the KV cache is a key step from "being able to call APIs" to "understanding inference systems." For developers who want to go deep into the underlying mechanisms of LLMs, this path is time-consuming, but the gains are hard to replace by reading papers and source code.

Of course, the capability of a personally reproduced model cannot be compared with industrial-grade large models. A 125M-parameter model trained on 5.27B tokens from FineWeb produces text that is roughly at the level of "grammatically fluent but logically simple." Its value lies not in producing a usable product, but in validating understanding, accumulating experience, and laying the foundation for more complex subsequent projects (such as RLHF).

Background Notes

Character-level modeling refers to splitting text by individual characters (rather than subword tokens) and building a vocabulary; the tiny-shakespeare dataset's vocabulary usually contains only a few dozen characters. Compared with BPE tokenization, character-level modeling has longer sequences and sparser per-step semantics, but the vocabulary is extremely small, preprocessing is simple, and there is no out-of-vocabulary problem. Its main value lies in serving as a teaching and debugging tool: the model scale can be reduced to very small, and the loss curve can be observed after just a few minutes of training, making it easy to quickly locate implementation errors in forward/backward propagation. Once a character-level model can stably generate coherent Shakespeare-style fragments, it shows that the basic correctness of the entire training pipeline has been verified, and one can then switch to large-scale corpora with subword tokenization with more confidence.

RLHF (Reinforcement Learning from Human Feedback) usually adopts a three-stage process of "reward model + PPO": first collect human preference ranking data on model outputs, and train a reward model to fit human preferences; then use the pretrained model as the policy, optimize the policy with reinforcement learning algorithms such as PPO to maximize reward, while using KL divergence constraints to prevent the policy from deviating too far from the reference model. The root cause of high memory pressure is that training requires keeping four model copies resident simultaneously—the policy model (trainable), reference model (frozen), reward model (frozen), and value function model (trainable)—plus optimizer states and activations, so total memory can reach several times that of pretraining. In practice, parameter-efficient fine-tuning such as LoRA, quantizing frozen models, or reducing the base model scale are commonly used to adapt to limited hardware, consistent with the approach mentioned in the original text.

One-line Summary

The developer hand-wrote LLaMa components from scratch and trained progressively in three stages up to a 125M-parameter model.

Paragraph Summary

A developer shared on Reddit their experience training a LLaMa model from scratch without relying on existing frameworks, manually implementing the four core components RoPE, GQA, RMSNorm, and SwiGLU as well as batched inference with the KV cache. The training used a three-step strategy: first a smoke test with tiny-shakespeare character-level data, then a 41M-parameter model trained on TinyStories-v2 to validate language generation ability, and finally a 125M-parameter model trained on 5.27B tokens from FineWeb, taking 11 hours on dual A5000 GPUs. The developer's next step is to work on RLHF, but they need to assess whether memory can accommodate a multi-model pipeline. The article emphasizes that the value of personal reproduction lies in understanding architectural trade-offs and cultivating engineering intuition, rather than producing a usable product.