Post-Training

"Post-training" is the alignment and capability enhancement phase following pre-training, and typically includes SFT, RLHF, and reinforcement learning based on process reward models (such as GRPO and DAPO).

Each RL training step requires running inference (generating rollouts) and backpropagation (updating weights) simultaneously, making the compute requirements far higher than ordinary fine-tuning.

后训练阶段使用几万对到几十万对'用户请求+标准回答'的配对数据来训练模型理解人类指令意图

从预训练检查点到可商用产品需经历监督式指令微调（SFT）和基于人类反馈的强化学习（RLHF）两个后训练阶段

A typical post-training pipeline is: pre-training → supervised fine-tuning (SFT) → preference optimization (RLHF / DPO / GRPO).

AI的训练过程分为三个阶段：预训练、后训练和对齐

The post-training objectives of web-native models include parsing DOM structures, locating interactive elements, determining where to click or type next, and falling back and retrying when the page state is abnormal.

Post-training is a critical phase between the completion of large language model pretraining and actual usability, encompassing technical approaches such as supervised fine-tuning (SFT), instruction alignment, and RLHF/DPO.

One of the fundamental reasons the industry has actively explored alternative paths such as post-training, reinforcement learning, and inference-time scaling in recent years is the diminishing marginal returns of pre-training scaling laws.

A model with fewer parameters but fully trained on more tokens, though it may appear over-trained during training, can significantly reduce costs at inference time, since inference overhead is primarily determined by parameter count.

Collecting unique data for specific tasks and then post-training on it can often yield higher accuracy at lower cost.

Common approaches to compressing thinking length include constructing short-thinking–correct-answer paired data for SFT, or designing length-penalty reward functions for reinforcement learning training.

SFT (supervised fine-tuning) using causal language modeling in the post-training stage takes the form of user tasks with chat templates applied, available tool sets, and tool response payloads from the environment.

Publicly disclosing post-training metrics involves a trade-off: transparency helps build technical credibility, but training recipes and data composition are core competitive barriers for vendors, and excessive disclosure may erode one's own advantage.

Achieving a significant improvement in programming capabilities through post-training on the same base model provides a new model iteration approach for teams with limited computing resources.

Reinforcement learning-related tools account for a significant proportion of the newly added projects, reflecting the importance of RL in the post-training phase of large models.

Post-training costs only 1–5% of the cost of training a large model from scratch, yet can yield a 20–50% performance improvement on target tasks.

数据规模的增长并不等同于有效学习信号的增长，在模型能力提升到一定阈值后简单堆砌样本的收益会急剧下降

成熟团队往往采取混合策略：用扩量数据打好基础覆盖，用精挑的困难样本进行针对性补短板

少量具有挑战性、配以标准答案和审核的真实世界任务比大量普通样本更有价值