Stanford CS336 Deep Dive: A Complete Guide to Building Large Language Models from Scratch

Stanford's CS336, Language Models from Scratch, is widely regarded as one of the most rigorous public courses on large language models. Its core philosophy comes down to one idea: you only truly understand how language models work when you build one yourself from the ground up. In an era where anyone can spin up a model with a coding agent in one click, why insist on this "reinvent the wheel" approach to teaching? This article unpacks the course's central ideas and technical roadmap.

Why Build a Large Language Model from Scratch

Course instructor Percy Liang points to a troubling trend: researchers are losing touch with the underlying technology. Ten years ago, every AI researcher implemented and trained their own models. Eight years ago, people downloaded pretrained models like BERT and fine-tuned them. Today, many just prompt a model and call it done.

There's nothing wrong with prompting — raising the level of abstraction is generally a good thing. But as Percy puts it: "Abstractions are leaky." When you want a model to do something and hit a wall, if you're operating purely at the prompt level, you've severely narrowed the design space you can explore. For genuine foundational research, understanding how language models actually work is essential — and the way to understand is to build.

The Limits of Training Small Models

The course is also honest about a real constraint: frontier models have become industrialized. The training cost for GPT-4 was reportedly around $100 million three years ago and may now be in the billions. The GPT-4 paper explicitly states it won't disclose any construction details, citing competitive and safety considerations.

That means students can only train small models — and small models don't necessarily represent what's happening at the frontier. The course offers two illustrative examples: at small scale, MLP layers account for roughly 44% of FLOPs, but that figure climbs to 80% at 175B parameters — optimizations that pay off at small scale may not transfer to large scale. Then there's emergent behavior: many capabilities only appear suddenly once a critical scale threshold is crossed.

Three Types of Transferable Knowledge

Given the limitations of small-scale experiments, what can we actually learn that carries over? The course organizes this into three categories:

• Mechanics: What a Transformer is, how model parallelism works. These transfer completely.
• Mindset: How to approach building a language model, how to squeeze every last drop of performance from hardware, how to take scaling seriously. These also transfer.
• Intuitions: Which data and modeling decisions lead to good performance. These don't necessarily transfer across scales — they need to be developed through real large-scale experiments.

Percy cites Noam Shazeer's famous SwiGLU activation function paper, whose conclusion section honestly states: "We offer no explanation for why these architectures work, and attribute their success to divine benevolence." That's a vivid illustration of how some design decisions are purely empirical.

Rethinking the "Bitter Lesson"

On the widely circulated "Bitter Lesson," the course corrects a common misreading. The wrong interpretation is "scale is everything, algorithms don't matter." The right interpretation is "only algorithms that can scale actually matter." Model accuracy = efficiency × resources, and efficiency becomes even more critical at large scale — if a single training run costs hundreds of millions of dollars, even a 5% efficiency gain is enormously valuable. OpenAI research shows that image model algorithmic efficiency improved 44× between 2012 and 2019.

The "Bitter Lesson" was articulated by Rich Sutton in 2019. Its central claim is that throughout AI history, whenever researchers tried to hard-code human knowledge into systems — handcrafted features, rule-based reasoning — those approaches were eventually surpassed by general methods that leverage large-scale computation. Sutton identified two methods that scale with compute: search and learning. The essay sparked significant debate. Critics argued it encouraged a nihilistic "just throw compute at it" attitude. CS336's reading offers a more constructive perspective: algorithmic efficiency multiplied by resources equals final performance, so efficient algorithm design that holds up at scale isn't less valuable — it's more valuable. OpenAI's 44× efficiency gain data makes exactly this point: algorithmic progress and scale expansion aren't in opposition; they're mutually reinforcing.

The Open-Source Ecosystem: What Makes This Course Possible

The course emphasizes the critical importance of the open-source ecosystem, because without open models, this course simply couldn't exist. Over the past few years, Meta led the charge with the LLaMA, LLaMA 2, and LLaMA 3 series, Mistral joined the field, and then came a wave of strong models from China including DeepSeek and Qwen.

Open-weight models now approach closed models in capability. Going further, AI2, NVIDIA, and the Marin project (which Percy is involved with) provide not just weights but also papers, code, and data — giving us a much clearer view of how these models are built. It's the papers still being published that let us peek inside frontier MoE and RL systems, even if key details like data mixing ratios remain missing.

Five Core Modules and Hands-On Assignments

The course is organized around five sections, each with a corresponding assignment.

Foundations: Training Your First Language Model

The goal of the first two weeks is to train a language model from scratch, covering tokenization, architecture, optimizers, and training. Assignment 1 asks students to implement a BPE tokenizer, Transformer, loss function, optimizer, and complete training pipeline, along with detailed resource accounting. Students then train on datasets like tiny stories and open web text and compete on a leaderboard by perplexity.

The course notes that while tokenization, modeling, and training are presented as separate modules, the real skill is trading off between all three: the model needs enough expressiveness to capture the complexity of the data, training needs to stay stable (keeping parameter and gradient norms in the "Goldilocks zone" — not exploding, not vanishing), and everything needs to be efficient (running fast on hardware).

Systems Optimization: Squeezing Hardware Performance

Assignment 2 goes deep into systems: kernels, multi-GPU parallelism, and inference. The core principle is minimizing data movement — because memory and compute are physically separated, data transfer is often the actual bottleneck. The course covers operator fusion, tiling, roofline analysis, and has students write custom kernels in Triton. The inference section covers the prefill and decode phases, speculative decoding, quantization distillation, and other acceleration techniques. The course also recommends Google's How to Scale Your Model.

Scaling Laws: Scientifically Predicting Large Model Performance

Assignment 3 focuses on scaling laws. Imagine you have a budget of 1E25 FLOPs (tens of millions of dollars) — what model should you train? You can't do a standard hyperparameter search at that scale, because you only get one shot.

The key mental shift is: don't think about individual models — think about a "scaling recipe" — a mapping from a FLOPs budget to a hyperparameter configuration. By fitting scaling laws from small-scale experiments, you can predict loss at large scale, letting you forecast performance before spending real money (or even using it to raise funding).

Percy emphasizes that scaling laws are not laws of nature — you have to carefully engineer a recipe to realize them. This requires principled model parameterization to achieve "hyperparameter transfer" — optimal hyperparameters at small scale should predict the right values at large scale. Predictability is therefore at least as important as optimality. The classic Chinchilla scaling law gives a useful rule of thumb: train on roughly 20 tokens per parameter.

The Chinchilla scaling law comes from DeepMind's 2022 paper Training Compute-Optimal Large Language Models. Through systematic experiments, the researchers found that most models at the time — including GPT-3 — were undertrained. Under a fixed FLOPs budget, model parameter count and training data volume should scale proportionally, with the optimal ratio being roughly 20 training tokens per parameter. This upended the prevailing practice of "make the model as large as possible and train on a fixed dataset." Chinchilla (70B parameters, 1.4T tokens) outperformed the much larger Gopher (280B) on multiple benchmarks at a fraction of the inference cost. That said, this isn't an ironclad rule: subsequent work (like the LLaMA series) showed that when inference efficiency is the priority, training a relatively smaller model on far more data than Chinchilla-optimal is often more practical in deployment — illustrating that scaling law conclusions are highly sensitive to how you define the optimization objective.

Data Engineering: The Key Determinant of Model Ceiling

Assignment 4 tackles data. Data doesn't fall from the sky — it has to be actively curated. Data quality largely determines model quality, and it also defines what you want the model to do. The course starts with evaluation, distinguishing between internal development metrics (like perplexity, which emphasizes smoothness across scales) and external-facing metrics (which emphasize ecological validity).

Data processing includes transformation, filtering, deduplication, mixing, and synthetic data generation. Assignment 4 has students start from raw web crawl data and work through all the "dirty" work — cleaning, deduplication, the whole pipeline. That's precisely the full experience of building from scratch.

Model Alignment: Improving from Weak Supervision

Assignment 5 focuses on alignment. Everything up to this point has been fully supervised next-token prediction. Now the course moves to improving models with weak supervision — because it's often easier to critique than to generate. Methods include RL algorithms like PPO and GRPO, as well as the simpler DPO. The course is candid that RL algorithms are unstable and hard to tune, and large-scale RL introduces system challenges around coordinating inference and training servers — "a beautiful mess."

DPO (Direct Preference Optimization) and PPO (Proximal Policy Optimization) represent two dominant paradigms for alignment training. PPO is an online RL algorithm that first trains a separate reward model to score outputs, then uses that reward signal to update the language model via policy gradients. The full pipeline requires running four models simultaneously (reference policy, current policy, reward model, value model), making it engineering-intensive and hyperparameter-sensitive. DPO is an offline method that reframes preference learning as a direct classification loss, bypassing explicit reward model training. It only needs preference pair data (chosen/rejected pairs), is simpler to implement, but being offline, it can't iteratively improve by sampling from the model's own new generations. GRPO (Group Relative Policy Optimization), introduced by DeepSeek, replaces absolute rewards with within-group relative rankings, reducing dependence on a value network. It has shown strong results on mathematical reasoning tasks and is one of the core training methods behind DeepSeek-R1.

Tokenization: A Technical Detail You Can't Skip

The course's first technical unit is tokenization. Why is it necessary? From an efficiency standpoint, tokenization compresses long byte streams into fewer tokens and enables adaptive computation — common patterns map to a single token, rare ones get split into multiple.

The course works through the drawbacks of character-level, byte-level, and word-level tokenization before landing on BPE (Byte Pair Encoding). BPE starts from byte sequences and repeatedly merges the most frequent adjacent token pair, so common sequences become a single token while rare ones split into multiple units — neatly avoiding the problem of <unk> tokens. Assignment 1 asks students to optimize a naive but extremely slow BPE implementation into something practically usable, with the option to rewrite it in Rust or C.

It's worth noting that Percy reportedly wishes every year he didn't have to teach tokenization, hoping for end-to-end approaches that operate directly on bytes (like HNet). But as long as frontier models still use tokenizers, this content remains relevant. Any replacement must satisfy the same two properties: abstracting over sequences and supporting variable-length adaptive computation.

BPE (Byte Pair Encoding) was originally a data compression algorithm proposed in 1994. It was introduced into NLP in 2016 for subword segmentation in neural machine translation, and has since become the standard tokenization approach for major models like GPT and LLaMA. The core training process: starting from a byte- or character-level initial vocabulary, count the frequency of all adjacent token pairs in the corpus, merge the most frequent pair into a new token, and repeat until the vocabulary reaches the target size. At inference time, greedily encode input text according to the merge rule priority order. The main advantage is a controllable vocabulary size (typically 32K–128K) with natural robustness to unseen words — any text can fall back to byte-level representation without producing <unk> tokens. End-to-end byte-level models like HNet attempt to bypass tokenization by modeling raw byte sequences directly, but face challenges with computational efficiency due to the much longer sequence lengths involved, and remain an active area of research.

Closing: Efficiency as the Unifying Thread

Looking across the entire course — tokenization, architecture, systems, data, scaling — the underlying logic unifies around one word: efficiency. You have fixed data, compute, memory, and communication bandwidth. The goal is to build a model that performs best on your evaluation criteria given those resources. That mindset of "always think about the efficiency of your methods" is the core way of thinking CS336 aims to instill in every student.