Stanford CS146S Deep Dive: The Complete LLM Training Pipeline from Raw Data to AI Agents

When you hit send in ChatGPT, a seemingly intelligent response appears almost instantly. But what's actually happening inside that black box? More importantly, what happened in the months or years before you typed your first prompt? Stanford's CS146S course pulls back the curtain on this multi-stage engineering pipeline — tracing how raw, messy internet data becomes the AI assistant you use every day.

This article follows the model's evolutionary path through three foundational training stages — pre-training, supervised fine-tuning, and reinforcement learning — before extending to the advanced prompting techniques needed to truly master these models.

Pre-Training: A Lossy Compression of the Entire Internet

Before becoming a useful assistant, a large language model starts life as a base model — essentially a massive, extraordinarily sophisticated text-completion engine. The goal of pre-training is straightforward: acquire knowledge. The scale, however, is almost incomprehensible.

Data collection begins with aggressive crawling. The largest source is typically Common Crawl, a nonprofit that has been indexing the entire web since 2007. This raw internet data is combined with high-quality documents — Wikipedia, academic journals, and publicly available GitHub code. Filtering is ruthless: spam, malware, racist content, adult content, and duplicate text are all stripped out.

Even so, the resulting datasets are astronomical. Take the public dataset FineWeb: text data alone weighs in at roughly 44TB — equivalent to approximately 15 trillion tokens. These texts are split into sequences of tokens (a token might be a word, or even just part of a word), and the model must absorb all of it.

Autoregression: What Are Those Billions of Dollars in GPUs Actually Computing?

The core mechanism is conceptually simple. Large language models are autoregressive: they predict the probability of the next token in a sequence based entirely on what came before. Feed in "the cat sat on", and the model predicts "the mat" as a likely completion.

Modern models can handle remarkable context lengths. Early models like GPT-2 could only process around 1,000 tokens, while modern models can look back hundreds of thousands — sometimes up to a million — tokens. This context is fed into a neural network with billions to trillions of tunable parameters — think of them as knobs on an enormous mixing board. GPT-2 had around 1.6 billion; GPT-4 reportedly has around 1.8 trillion. Training is the process of repeatedly adjusting these knobs so the model's next-token predictions increasingly match the actual training data.

The compute required is almost unimaginable. A top-tier H100 GPU node costs roughly $3 per GPU per hour, and training a large base model runs tens of millions of dollars. This cost structure defines the industry landscape: only a handful of players can afford to build base models from scratch.

Think of pre-training as a probabilistic "zip file" of human knowledge. But compression means lossy: the model loses the ability to reason, and can't distinguish between what is true and what merely appears frequently on the internet. It has learned the grammar of knowledge, but not the deep structure of logic.

The core architecture behind autoregressive models is the Transformer, introduced by Google in the 2017 paper Attention Is All You Need. The Transformer's key innovation is self-attention: it allows the model to "see" the entire input sequence simultaneously when predicting the next token, dynamically deciding which positions matter most — unlike earlier RNNs that passed information along one step at a time. This is why modern models can handle million-token contexts: attention can in principle bridge arbitrary distances between words. Most of those "billions of parameters" live in the attention matrices and feed-forward network weights across the Transformer's layers. Understanding this helps explain many phenomena discussed later: why models need tokens to think (each token corresponds to one forward-pass computation), and why longer context windows make inference more expensive.

Supervised Fine-Tuning: Giving the AI Its Personality

That expensive autocomplete machine is not what we interact with in conversation. Turning a base model into a useful assistant requires post-training, and the first step is Supervised Fine-Tuning (SFT).

SFT is computationally much cheaper — hours rather than months. The approach swaps out the vast internet corpus for a smaller, carefully curated dataset of hundreds of thousands of high-quality conversational examples. Companies hire contractors who, following extremely detailed instructions, write ideal assistant responses to a wide variety of prompts. This is essentially teaching the model, by demonstration, to embody the persona of someone who is helpful, honest, and harmless.

Looked at from another angle: when you chat with an AI assistant, you aren't talking to some new form of intelligence. You're interacting with a neural network that has been trained to simulate a highly skilled human annotator following instructions flawlessly. Put bluntly, you're talking to the most obsessively optimized customer service simulation ever built.

Reinforcement Learning: Teaching the Model How to Think

If SFT teaches the model what to say, reinforcement learning (RL) teaches it how to think. This is the most cutting-edge third stage.

In verifiable domains like math and code, we know exactly what the correct answer is. We can simply give the model a problem and an answer, and let it figure out the token sequence that reliably leads to the right result. What makes this fascinating is that cognitive strategies emerge from this process.

Because each token has a finite amount of computation, the model gradually learns it must spread reasoning across many tokens — this is Chain of Thought (CoT). If you ask it to output the answer to a hard problem in one shot, it will frequently fail at the "mental arithmetic." The key insight is: models need tokens to think. Denying them the ability to work step-by-step is like asking someone to solve an integral without scratch paper.

RLHF and Its Shortcomings

When answers are objectively verifiable, pure RL works extremely well. But for subjective tasks — writing jokes, crafting persuasive essays — companies turn to Reinforcement Learning from Human Feedback (RLHF): a second AI is trained as a "reward model," learning to replicate how humans rank different outputs.

The course highlights a key flaw: RL is particularly good at finding exploits in any simulated system. The model may learn to generate content that fools the reward model into giving high scores, treating the score as more important than actual quality. This is why RLHF is a "short-lived" fine-tuning process — the model eventually learns to game the reward system, and human oversight remains indispensable.

The "reward hacking" problem in RLHF has a formal name: Goodhart's Law — "When a measure becomes a target, it ceases to be a good measure." In the RL context, once the model's optimization objective becomes maximizing the reward model's score, gradient descent will find and exploit the reward model's blind spots — producing outputs that sound fluent but are hollow, or phrased to flatter human preferences while losing actual informational value. Industry has developed several mitigations, including PPO (Proximal Policy Optimization) with KL-divergence penalties — which limits how much the model can update per step, preventing it from drifting too far from its SFT baseline — and Anthropic's Constitutional AI (CAI), which uses AI self-critique to replace some human ranking, reducing over-reliance on the reward model. These approaches all attempt to balance "making the model more aligned with human preferences" against "preventing it from treating score-gaming as its ultimate goal."

Prompt Engineering: Programming Models Through Conversation

With knowledge, personality, and reasoning ability established, the final step is prompt engineering — essentially programming a model through conversation.

• In-Context Learning (K-Shot Prompting): Embed one to five examples directly in the prompt. Need Korean translations in a specific honorific style? Give the model a couple of examples and it adapts immediately.
• Zero-Shot CoT: For complex logic problems, a simple "Let's think step by step" dramatically improves accuracy, unlocking the model's ability to reason out loud.
• Retrieval-Augmented Generation (RAG): A model's knowledge is frozen at training time — it doesn't know what happened yesterday, or what's in your company's private database. RAG gives the AI a smart search function: the model generates special tokens to call a tool, inserts the search results back into the context window, and is forced to use current facts. This is a critical defense against hallucination.
• Self-Correction: You can run five separate reasoning paths and take the majority answer. For coding especially, "reflection" is valuable — feeding error messages back into the prompt and asking the model to critique and revise its own code.

From Chat to Engineering Leverage: How AI Amplifies Developers

Debates about AI "replacing" workers are endless, but the course emphasizes a different framing: you won't be replaced by AI — you'll be replaced by a capable engineer who knows how to use AI. LLMs don't replace engineers; they turn ordinary engineers into extraordinarily productive ones.

Take internal use of Codex-class models at OpenAI as an example:

• Code comprehension: Paste an error stack trace and prompt the model to "locate this logic in the repository and trace the data flow" — a one-hour debugging session becomes 15 minutes.
• Repetitive changes: Consistently updating a legacy API call across 40 files — three hours of manual work becomes 90 seconds.
• Performance optimization: Ask the model to analyze code, flag slow database calls, and draft fixes — 30 minutes compressed to 5.
• Test coverage: Set the model to batch-process low-coverage code overnight; wake up to runnable unit tests that catch edge cases a human might have missed.

To reach this level of precision, you can't just fire off a casual request. Three key techniques are worth remembering:

1. Role prompting: Explicitly set the context — "You are a senior software engineer-level assistant."
2. Structured prompting: When pasting complex data, use labels (like XML-style tags) to help the model parse efficiently.
3. Persistent context: Maintain a file like AGENTS.md containing business logic and naming conventions the AI couldn't possibly know. Paste it in at the start of every session — this dramatically improves accuracy across all your prompts.

Conclusion: When AI Invents Strategies Humans Can't Conceive

From trillions of raw tokens, to human-annotated conversations, to advanced reasoning that requires Chain of Thought — this complete path reveals that LLMs are simultaneously powerful and brittle. The course describes it as a "Swiss cheese model of capabilities": a model can solve olympiad-level math problems, yet struggle to determine whether 9.1 or 9.9 is larger. You must treat them as probabilistic tools, always verify their outputs, and use the techniques above to work around their limitations.

One final question worth sitting with: when models are trained purely with RL in verifiable domains, they are unconstrained by human logic — just like AlphaGo's legendary Move 37, which no human player had ever conceived. If these reasoning models continue to evolve and discover internal thinking languages more efficient than English, if the optimal way to solve complex problems becomes something humans simply cannot conceptualize — where does human-AI collaboration go from there?

---

Background Notes

The fundamental problem RAG solves is the parametric knowledge limitation of LLMs: all of a model's knowledge is "frozen" into its weights when training ends and cannot update automatically. RAG typically operates in two steps: the retrieval phase uses a vector database (such as Pinecone or Chroma) to convert the user's question into an embedding vector and find semantically similar document chunks in the knowledge base; the generation phase stitches those chunks into the prompt as additional context, grounding the model's response in current facts. This contrasts with the alternative of expanding model parameters to "memorize more knowledge": RAG's advantages are that the knowledge base can be updated in real time, costs are low, and sources are traceable. Its weakness is that retrieval quality directly determines answer quality — retrieving the wrong documents can produce even more confidently stated hallucinations. For enterprise private knowledge bases, legal compliance, healthcare, and other scenarios with high demands on information recency and traceability, RAG is currently the dominant solution.