Training a Language Model from Scratch on a 4GB GPU: The Failures Are Worth More Than the Success

A laptop GPU with 4GB of VRAM, a small-scale corpus, and a series of breakdowns that turned out to be the most educational part of the whole experience.

Someone trained a 22-million-parameter language model from random weights on a gaming laptop with just 4GB of VRAM. It learned to mimic the surface appearance of technical documentation, fell into endless repetition loops, and never managed to produce a working Fibonacci function. Near the end of training, CUDA crashed once. The final artifact isn't a useful chatbot — but those failures are precisely what make the whole exercise worthwhile.

Pulling a pretrained model via Ollama makes modern AI feel deceptively simple. That illusion dissolves quickly when you try to train even a tiny model from scratch.

Why Bother Looking at What Happens Before Inference

Calling an API or loading a local pretrained model is convenient — but that convenience hides exactly what the author wanted to understand: What does a dataset actually look like? How much does cleaning matter? How does a document become tokens? What is a checkpoint, really? What does loss tell you, and what does it quietly conceal? What does a model's output look like before it becomes useful?

None of those questions can be answered by calling something that already works. The approach was straightforward: build a less polished version by hand and watch it fail in real time. This is deliberately avoiding shortcuts for research purposes — not product development.

4GB of VRAM: The Constraint Is the Teacher

The hardware was a Lenovo IdeaPad gaming laptop — Intel i7-10750H, 16GB RAM, GTX 1650 with 4GB VRAM — running PyTorch and CUDA under WSL2. The author is upfront that this is not a machine suited for training, and never frames it as one.

But the constraint is the point. A GPU with abundant VRAM can hide sloppy decisions behind excess headroom. With only 4GB, you're forced to think clearly about why the context length is what it is, why batches are broken into microbatches, and how much memory the optimizer state actually consumes.

The author settled on a context length of 256, microbatches of 16, paired with gradient accumulation so each parameter update still sees a reasonably large effective batch. Peak memory during the main training run was 1.54 GiB — well within the GPU's limit. The takeaway is restrained: this wasn't the GPU being generous. It was the result of designing the entire pipeline around the constraint from the start, rather than discovering bottlenecks mid-run.

Gradient accumulation is the key technique for expanding effective batch size under constrained VRAM: a large batch is split into smaller microbatches, each processed in a forward pass that accumulates gradients without updating parameters. Once the target number of steps is accumulated, a single parameter update is applied. The model sees the same effective sample count as a large batch, but peak VRAM only needs to hold the activations of a single microbatch. Mixed precision training is another critical memory optimization: forward passes and gradient computation use FP16 (half-precision), while a full FP32 master copy of the weights is retained only for parameter updates — theoretically halving memory usage, but requiring a gradient scaler (GradScaler) to prevent FP16 underflow from causing vanishing gradients. This is also why recovering from a CUDA "sticky error" requires restoring the scaler state alongside everything else.

The Corpus: Downloading Is Easy, Filtering Is the Work

The corpus came from Spanish and English Wikipedia, Python documentation, and several technical documentation collections. Collecting text is trivial — anyone can download a data dump. What actually matters is deciding what gets into the training set, and understanding what that decision will cost you later.

The author chose a controlled corpus over a large unfiltered web dataset, evaluating each source along dimensions like quality, licensing, language balance, deduplication risk, and preprocessing difficulty. Cleaning and filtering is the dividing line between "everything I could download" and "something I intentionally built."

The full pipeline is visible: choose source text → clean and filter → tokenize → train from random weights → save recoverable checkpoints → evaluate on held-out text and inspect generated output. The author emphasizes that these early choices resurface at evaluation time as specific, attributable behaviors.

Falling Loss, Crashed CUDA, and the Checkpoint That Saved Everything

Validation loss started at 9.48. With a vocabulary size of 12,000, uniform random guessing yields a loss of roughly 9.39. After approximately one full pass through the corpus, the model reached 4.27. That's real progress, and it feels satisfying to watch — but it answers a narrow question: whether predictions on held-out text improved. It says nothing about whether the model can generate anything useful.

This is also when checkpoints stopped being an abstract concept. The author tested resuming from a checkpoint with a short training run before the main one, and printed the expected output after recovery:

Resumed from: results/EXP-001/resume-test/latest.pt
Resume step: 10

That turned out to be extremely wise. Shortly after the main training reached step 3,540, CUDA threw an unknown sticky error and training stopped. It wasn't an out-of-memory crash, and loss had been behaving normally. The author resumed from the step-3,500 checkpoint — restoring the optimizer, mixed-precision scaler, and random state together — and ran cleanly to completion.

The more revealing surprises came from generation. Using sampling, later checkpoints produced passages that looked like technical documentation. An early checkpoint blended WordPress, MDN, and Docker content into incoherent noise. Switching to greedy decoding — always selecting the highest-probability next token — caused the final model to loop repeatedly, a failure mode that validation loss never surfaced at all. The reason: during generation, the model must condition on its own output rather than the ground-truth text used during evaluation. A few prompt results:

• "A WordPress plugin can" (sampling, temperature 0.8, final checkpoint) → "A WordPress plugin can be used when the plugin is loaded" / "A plugin contains a PHP file"
• "Barcelona es una ciudad" (greedy decoding) → repeated variations of "de la ciudad"
• "A language model learns" (greedy decoding) → "the language is the language language…"
• "def fibonacci(n):" → no checkpoint produced a valid implementation

Validation loss measures a model's ability to predict the next token on text it hasn't seen, typically expressed as cross-entropy. With a vocabulary of 12,000, random guessing gives an expected loss of ln(12000) ≈ 9.39, so starting at 9.48 means the model is essentially random at initialization. A loss of 4.27 corresponds to a perplexity of roughly 71 — the model is effectively choosing among 71 candidate tokens per step — which indicates it has extracted real statistical patterns from the corpus. However, validation loss is computed under teacher forcing: each step receives the actual ground-truth token as context, not the model's own previous output. This is fundamentally different from autoregressive generation at inference time, where a single bad prediction compounds through context, eventually producing loops or semantic drift — error propagation that is completely invisible in the loss curve.

Evaluation Is More Interesting Than Training

Those generation samples revealed more than the loss curve alone ever could. Rather than relying on a single aggregate score, the author evaluated the final checkpoint separately by corpus segment. Technical documentation showed lower loss than general prose, but with significant variation across sources — MDN and WordPress text was more predictable to this model than Docker, Python, or Git documentation.

The author treats this ranking with appropriate caution: it's one observation from one checkpoint in one training run, and proves nothing about model quality. Its value lies in raising better questions than aggregate loss can ask. Is a source "easy" because it's well-written, or because it's highly repetitive? How much of low loss reflects genuine understanding versus predictable formatting? Is there undetected overlap between domains? Loss alone cannot distinguish any of these.

Greedy decoding selects the highest-probability token at every step — the most deterministic generation strategy, and the one that most directly exposes repetition tendencies, since once a high-frequency fragment enters the context, it tends to reinforce itself with the highest probability. Sampling with a temperature parameter rescales logits by dividing by the temperature before renormalizing: values above 1 flatten the distribution (more diversity), below 1 sharpen it (more concentration). Temperature 0.8 is slightly below 1, meaning the distribution is more peaked than raw output but still probabilistic. The fact that both strategies produce dramatically different behavior on the same model confirms that evaluation strategy is itself an independent variable shaping what you observe.

Small Models Are Brutally Honest

A model with only 22 million parameters trained for roughly one epoch has nowhere to hide. It learned what technical documentation looks like far better than it understood what technical documentation says. It can write plausible-sounding headings and parameter lists for APIs that don't exist, drift from one domain to another within a single paragraph, and fail at every checkpoint to produce a working def fibonacci(n):.

Precisely because it's bad, it's exceptionally useful for learning. Strong models paper over their own internal mechanisms; weak models expose them: capacity, data mix, repetition tendencies, and the gap between "predicting text" and "actually knowing something." This is research, not product development — a distinction the author returns to repeatedly.

What Was Actually Gained

The outcome wasn't a competitive model or a useful question-answering tool. It was a reusable research workflow: reproducible corpus processing tools, a training script that can pause and resume, and evaluation methods broken down by source. More importantly, the author internalized a decision principle — before claiming a model "learned" something, check both held-out loss and actual generation output. That's worth considerably more than a demo that just looks impressive.

Where to Go Next

The author makes no promises about building a larger model. The open questions left by this first training run all center on corpus selection, better evaluation methods, and changing one decision at a time so each change can be clearly attributed. Future plans include incorporating more source code and establishing a fixed evaluation benchmark to compare against this run. In the author's own framing: rather than taking a large step that can only be described, take a small, controlled step that can actually be understood.

For developers looking to understand low-level AI fundamentals, the core value of this experiment isn't the result — it's that it forces into the open all the unglamorous work that pretrained models conceal: data, tokenization, checkpointing, and the limits of evaluation, each made visible through a series of instructive failures.