The Three Stages of AI Training: Pre-training, Post-training, and Alignment Explained

Have you ever encountered this scenario: you ask AI to find some recent news, and it confidently lists headlines, dates, and links that look perfectly legitimate — only to discover when you click through that they're all fabricated.

Why does AI "lie"? To answer this question, we need to understand how AI is actually trained. The entire process is divided into three stages: pre-training, post-training, and alignment. Once you understand these three steps, you'll know where AI's capabilities end, and you'll be able to use it more intelligently.

An Intuitive Analogy: Training a New Employee

AI training isn't about memorizing everything on the internet, nor is it about programmers writing rules line by line like "apologize first when a customer yells at you." It's more like how we train a new employee:

• Step one: Dump a pile of materials on the new hire — ten years of company documents, emails, contracts, product manuals — and let them get a sense of what the industry looks like.
• Step two: A senior employee walks them through standard cases hand-by-hand, showing them how to respond in specific situations.
• Step three: A manager repeatedly reviews their output, scoring it, explaining what's good here, what's bad there, which phrases are off-limits, and which clients you never talk back to.

The three stages of AI training roughly correspond to these three steps.

Stage One: Pre-training — Learning Language First

During pre-training, engineers feed all the text they can find — web pages, encyclopedias, forums, emails, novels, code, news — into a blank model. For something at GPT-4's scale, the training data amounts to roughly trillions of words.

A few more words about the scale of this data. Taking GPT-4 as an example, its parameter count is estimated to exceed 1 trillion, with training data sourced from a mixture of Common Crawl (web-scraped data), Wikipedia, GitHub code repositories, academic papers, and more. Before being fed to the model, this data undergoes rigorous deduplication, filtering, and quality screening — low-quality content, duplicate text, and harmful material are removed. The training process itself requires thousands of high-end GPUs running in parallel for months, typically costing tens of millions to hundreds of millions of dollars. It's this scale of training that gives the model a deep understanding of statistical patterns in language, but it also inherits the biases, errors, and timeliness limitations present in the training data.

The model does only one thing at this stage: learn what language looks like. How well does it learn? Well enough that if you give it the first half of a sentence, it can predict what the second half most likely is.

For example, if you say "The weather today is really..." it will complete with "nice," not "bicycle." This isn't because it understands weather — it's because it has seen the phrase "weather is really nice" hundreds of millions of times.

After this stage, can it write an email if you ask? Yes. But something will feel off — the opening is "Dear Valued Customer," the middle suddenly shifts to a forum-style rant, and the ending reads like podcast copy. That's because it learned "how the entire internet writes," not "how customer emails should be written."

In one sentence: After pre-training, AI can speak, but it doesn't really know how to do things.

Stage Two: Post-training — Teaching It to Understand What You Actually Want

This is where AI annotators enter the picture. They prepare tens of thousands to hundreds of thousands of paired examples: "user request + standard response."

For example:

User request: Write me a customer email about a delivery delay. Professional tone, include an apology, and provide a new delivery date.

Standard response: A well-structured, appropriately-toned sample email.

The model learns from these examples pair by pair, until it understands the full intent behind words like "delivery delay email" — apologize, explain the reason, give a new timeline, maintain a professional tone.

At this point, when you ask it to write an email, the structure, tone, and paragraphing are all correct. But problems remain:

• Over-promising: "I guarantee this will never happen again" — this kind of statement is actually a big no-no in business contexts.
• Over-verbose: A single apology stretches into five paragraphs.
• Poor judgment on nuance: If a customer is rude and abusive in their email, the AI either pushes back aggressively or apologizes with excessive humility.

In one sentence: After post-training, AI can do things, but it doesn't understand nuance.

Stage Three: Alignment — Teaching It What "Doing Well" Means

This step is the most interesting. Engineers stop providing standard answers and switch to a different approach: for the same request, the model generates two responses — version A and version B — and annotators choose which one is better. The annotators' preferences are trained into the model, and after hundreds of thousands to millions of iterations, the model gradually learns one thing: what a human-preferred response looks like.

This method is known in the industry as RLHF (Reinforcement Learning from Human Feedback), and it's the core alignment technology behind mainstream large language models like ChatGPT and Claude.

The complete RLHF workflow is worth understanding in depth. This technique was systematically proposed by OpenAI in their 2022 InstructGPT paper and applied to ChatGPT. The full process has three steps: first, train a "Reward Model" specifically designed to predict human preference scores for given responses; then use reinforcement learning algorithms (typically PPO, Proximal Policy Optimization) to continuously adjust the language model's parameters so its generated responses earn higher reward scores; finally, apply KL divergence constraints to prevent the model from drifting too far from its original language capabilities in pursuit of high scores. The key challenge in this process: human annotators themselves have biases, and different annotators have inconsistent judgment criteria. This means the direction of "alignment" essentially reflects the values of the participating annotator population, rather than any objective standard.

What it learns isn't any single correct answer, but a "feeling" — a sense of propriety about what should and shouldn't be said:

• Apologies should be sincere but not excessive
• Don't promise the customer things you can't deliver
• When a customer is emotionally charged, empathize first, then address the issue
• Don't reveal internal supply chain problems in emails

None of these are hard-coded rules. They're a sense of propriety that's been "trained in."

Additionally, the alignment stage continuously reinforces boundaries: don't write anything offensive to customers, don't fabricate product shipping dates, don't leak internal pricing, don't help someone write a scam email. These boundaries are explicitly labeled during training, and the model will never directly output such content.

Pre-training, Post-training, and Alignment Aren't Three Versions — They're Three Layers of Capability

Many people mistakenly think these three stages are three versions of AI — first version 1.0, then an upgrade to 2.0, and finally 3.0. That's not the case. They're three different layers of capability, stacked within the same model:

Stage	Core Capability	Specific Manifestation
Pre-training	Basic knowledge of language and the world	Knows Cao Xueqin wrote Dream of the Red Chamber, water boils at 100°C, emails often start with "Dear"
Post-training	Understanding the format and intent of human instructions	When you say "write me" it writes, "summarize" means condense, "give three examples" gets you three items
Alignment	Selecting the safer, more preferred answer among multiple reasonable options	Doesn't take sides on controversial topics, calmly presents both perspectives

When GPT tells you "Dream of the Red Chamber was written by Cao Xueqin, with the last forty chapters generally attributed to Gao E" — that's pre-training at work. When you say "organize what I just said into a three-column table" and it gives you a proper table — that's post-training at work. When you ask about a controversial topic and it doesn't take sides but calmly presents both viewpoints — that's alignment at work.

Back to the Original Question: Why Does AI "Make Things Up"?

Now we can answer the question from the beginning. When AI fabricates three nonexistent news articles, it's fundamentally a problem rooted in the pre-training stage.

Pre-training taught it "what language looks like," not "what facts are." It knows a news article should have a headline, a date, and a source, so when you ask for news, it can write something that looks convincing. But it doesn't actually know whether that news story exists.

This is what we commonly call AI hallucination — the model generates content that appears reasonable but doesn't actually exist. It's not "lying to you" — it fundamentally lacks the ability to distinguish between "looks real" and "is real."