Post-Training Techniques Explained: Principles and Selection Guide for SFT, PPO, DPO, and GRPO

Introduction: The Paradigm Shift from Pretraining to Post-Training

Recently, a new community dedicated to AI post-training appeared on Reddit — r/posttrain — focusing on topics like fine-tuning, SFT, RLHF, DPO, preference data, evaluation, and practical experiments. The emergence of this community reflects an important trend in the evolution of large model technology: the differentiation of model capabilities increasingly depends on the work done after pretraining.

If pretraining gives a model broad world knowledge and language abilities, then post-training determines whether the model is "useful," "aligned," and "specialized." This article provides a systematic overview of the core technology stack in post-training — from supervised fine-tuning (SFT) to reinforcement learning methods like PPO, DPO, and the recently spotlighted GRPO — helping readers build a complete technical understanding.

What Is Post-Training?

Post-training refers to the process of further adjusting a model's behavior through a series of supervised or reinforcement learning techniques after the base model has completed large-scale unsupervised pretraining. To understand the significance of post-training, we first need to understand what pretraining produces: during the pretraining phase, the model learns through self-supervised methods — typically the "Next Token Prediction" task — trained on massive text corpora (usually on the scale of trillions of tokens). This process requires thousands of GPUs running for weeks or even months, costing tens of millions of dollars. After pretraining, the model has acquired broad language understanding and world knowledge, but it is still essentially a text completion engine and cannot directly serve as a conversational assistant. This is the fundamental reason post-training exists.

The core objectives of post-training typically include three aspects:

• Instruction following: Enabling the model to understand and execute human instructions, rather than merely continuing text;
• Alignment: Making the model's outputs conform to human values, reducing harmful, biased, or inaccurate content;
• Capability enhancement: Improving the model's performance in specific domains (such as math, code, and reasoning).

A typical post-training pipeline usually follows: Pretraining → Supervised Fine-Tuning (SFT) → Preference Optimization (RLHF / DPO / GRPO). These three stages build upon each other progressively, collectively refining a "raw" language model into a usable AI assistant.

SFT: Supervised Fine-Tuning Is Where Everything Begins

Core Principles of SFT

Supervised Fine-Tuning (SFT) is the first step in post-training. It uses high-quality "instruction-response" paired data to train the model using standard cross-entropy loss. Cross-entropy loss is the most fundamental training objective function in natural language processing — it measures the difference between the model's predicted probability distribution and the true labels. In the context of SFT, the model is asked to predict the next token of the target response one token at a time, and the cross-entropy loss penalizes cases where the model assigns low probability to the correct token. Notably, during SFT training, the loss is typically computed only on the response portion, not on the user instruction portion — a practice known as "computing loss on completions only" — to ensure the model learns how to answer rather than repeating the user's question.

Essentially, SFT teaches the model: "When a user asks this, you should respond like that." The effectiveness of SFT is highly dependent on data quality. The industry consensus is: a small amount of high-quality data often outperforms massive volumes of low-quality data. This is why many teams invest significant effort in data cleaning, annotation, and construction.

Limitations of SFT

While SFT can teach a model the "format" and "style" of answering questions, it can only imitate behaviors that appear in the demonstration data. It cannot tell the model "which answer is better," nor can it easily handle subtle preference differences in open-ended questions. This is precisely why reinforcement learning methods enter the picture.

RLHF and PPO: The Classic Alignment Paradigm

The Three-Stage RLHF Process

Reinforcement Learning from Human Feedback (RLHF) is one of the key technologies behind ChatGPT's success. It typically involves three steps:

1. Obtain an initial policy model through SFT;
2. Train a Reward Model (RM) that learns human preference rankings for different responses;
3. Use a reinforcement learning algorithm (such as PPO) to optimize the policy model so that its outputs receive higher reward scores.

Training the Reward Model (RM) itself is a significant engineering challenge. It is essentially a regression model adapted from a language model, trained on preference judgments from human annotators: for multiple answers to the same question, annotators label which one is better. The reward model learns these preferences through a Bradley-Terry ranking model and can ultimately output a scalar score for any "question-response" pair, where higher scores indicate better response quality. However, reward models are prone to "reward hacking" — where the policy model learns to exploit vulnerabilities in the reward model to obtain high scores without genuinely improving response quality. This is a risk that requires particular vigilance in RLHF practice.

PPO's Characteristics and Costs

Proximal Policy Optimization (PPO) is the most classic reinforcement learning algorithm used in RLHF, proposed by OpenAI in 2017. It is a policy gradient-based reinforcement learning algorithm whose core idea is to limit the magnitude of each policy update through a "clipping" mechanism: when the probability ratio between the new and old policies exceeds a certain range (typically set to 1±0.2), the gradient is clipped to prevent drastic policy changes, thereby ensuring training stability.

However, PPO's implementation is quite complex:

• It requires simultaneously maintaining the policy model (Actor), a reference model (typically a snapshot of the SFT model, used to provide KL divergence constraints), the reward model, and a value model (Critic), resulting in enormous memory overhead — for a 70B parameter model, this means loading approximately four times the model parameters simultaneously, placing extremely demanding requirements on GPU clusters;
• It is sensitive to hyperparameters, and the training process is prone to instability;
• The engineering implementation barrier is high, with significant debugging costs.

These pain points have given rise to a series of simpler alternatives.

DPO: A Simplified Path That Bypasses the Reward Model

Direct Preference Optimization (DPO) is an important recent breakthrough. Its core insight is: there's no need to explicitly train a reward model or perform reinforcement learning — you can directly optimize the policy model using preference data.

DPO's theoretical foundation comes from an elegant mathematical transformation of the RLHF objective function. In standard RLHF, the goal is to maximize reward while constraining the policy from deviating too far from the reference model (via KL divergence penalty). The authors of DPO (Rafailov et al., 2023) discovered that this constrained optimization problem has a closed-form solution — the optimal policy can be directly expressed in terms of the reward function and the reference policy. Conversely, the reward function can also be expressed in terms of the optimal policy and the reference policy. Substituting this relationship into the Bradley-Terry preference model yields a loss function involving only the policy model and the reference model, completely bypassing the reward model. This derivation is remarkably elegant — it transforms a complex reinforcement learning problem into a concise classification problem.

In practice, given a pair of "better response" and "worse response," DPO directly adjusts the model to increase the probability of the good response and decrease the probability of the bad one.

Compared to PPO, DPO's advantages are very clear:

• Simple implementation: No reward model or Critic needed; the training pipeline is close to SFT;
• High stability: Avoids many of the instability factors inherent in reinforcement learning;
• Resource-friendly: Significantly reduced memory and compute overhead.

These advantages have quickly made DPO one of the most popular alignment methods in the open-source community. Of course, DPO also has its limitations: it has high requirements for preference data quality, and its performance ceiling on certain complex reasoning tasks may not match a well-tuned PPO. Additionally, DPO uses offline preference data (i.e., data collected before training) rather than data generated online by the model, which limits the model's ability to explore better strategies during training.

GRPO: A New Reinforcement Learning Paradigm for Reasoning

A Method That Rose with DeepSeek

Group Relative Policy Optimization (GRPO) gained widespread attention with the success of the DeepSeek model series, particularly excelling in improving models' mathematical reasoning and code generation capabilities.

GRPO's core innovation is eliminating the value model (Critic) from PPO. Instead of training a separate network to estimate state values, it samples a group of responses for the same question and uses the group's average reward as a baseline to compute each response's relative advantage.

Specifically, for each question in the training set, GRPO has the current policy model generate G independent responses (G is typically set between 8 and 64). Each response then receives a reward score (which can come from a rule-based verifier or a reward model). GRPO uses the group's average reward as the baseline, and each response's advantage value is its reward minus the group mean divided by the group standard deviation. This normalization allows the model to distinguish good from bad responses for questions of similar difficulty, avoiding the biases introduced by inaccurate Critic model estimates.

Core Advantages of GRPO

This design brings several key benefits:

• Eliminates the Critic model, significantly reducing memory and compute costs;
• Naturally suited for verifiable tasks: In scenarios with clear right/wrong answers like math and code, rule-based rewards (such as whether the answer is correct or whether the code passes test cases) can replace complex reward models, providing precise binary reward signals;
• Drives significant improvements in reasoning capabilities, making it one of the mainstream methods for training today's "reasoning models."

GRPO's popularity is closely tied to the rise of "Reasoning Models" in 2024-2025. Represented by OpenAI's o1/o3 series, DeepSeek-R1, and Google's Gemini 2.5, reasoning models significantly improve performance on math competitions, coding, and scientific problems by generating long Chain-of-Thought processes during inference. The core of training such models is using reinforcement learning — particularly GRPO and its variants — to incentivize the model to learn "slow thinking," meaning multi-step reasoning, self-verification, and error correction before generating the final answer.

GRPO's popularity marks a new direction in the post-training field: using reinforcement learning to unlock models' reasoning potential, rather than merely achieving behavioral alignment. This paradigm is considered a critical step for large language models moving from "pattern matching" toward "genuine reasoning," and represents the most cutting-edge application direction in post-training technology.

Technical Selection: How to Choose Among SFT, PPO, DPO, and GRPO

Faced with multiple post-training methods like SFT, PPO, DPO, and GRPO, how should practitioners make their choices? Here are some experience-based recommendations:

• Getting started and basic alignment: Begin with SFT — it's an essential first step that cannot be skipped;
• Simple and efficient preference alignment: Prioritize DPO for its low implementation cost and stable results;
• Maximum alignment quality with sufficient engineering resources: Consider PPO and its variants;
• Improving reasoning capabilities in math, code, etc.: GRPO is currently the top choice.

It's worth emphasizing that these methods are not mutually exclusive — in practice, they are often used in combination — starting with SFT as the foundation, then further optimizing with DPO or GRPO. Many leading labs' complete training pipelines even include multiple iterative rounds: SFT → DPO for initial alignment → GRPO for reasoning capability enhancement → another round of DPO to fine-tune output style, forming a complete training loop.

Conclusion: Post-Training Becomes the New Frontier of LLM Competition

The establishment of the r/posttrain community confirms an important reality: as pretraining techniques increasingly converge and open-source base models become more abundant, post-training is becoming the key differentiating factor that determines model quality. Whether it's data construction, reward design, or algorithm selection, post-training is full of practical space worth exploring.

For developers and researchers, understanding the principles and trade-offs of the SFT, PPO, DPO, and GRPO technology stack has become an essential skill. Communities like r/posttrain that focus on sharing practical experience are invaluable resources for learning and discussing these technologies.