Supervised Fine-Tuning

Modern large language models typically go through three training stages: pre-training, supervised fine-tuning (SFT), and reinforcement learning from human feedback (RLHF).

"Post-training" is the alignment and capability enhancement phase following pre-training, and typically includes SFT, RLHF, and reinforcement learning based on process reward models (such as GRPO and DAPO).

Supervised Fine-Tuning (SFT) trains the model using 'prompt-response' example pairs, teaching the model to respond in a conversational format, thereby making outputs more coherent.

Self-supervised learning is the core paradigm of the pre-training phase, where the model constructs training signals from the data itself through a next-token prediction task.

Models trained with reinforcement learning (RL) outperform models trained solely with SFT on Agent tasks.

业界探索的思考力度校准方向包括基于强化学习训练模型的元认知能力、通过监督微调学习推理深度分布、以及设计动态停止机制

Safety alignment usually uses supervised fine-tuning and reinforcement learning from human feedback to teach models to refuse harmful requests, but such training mostly focuses on the semantic content of requests and rarely explicitly models the social attributes of wording.

In such tasks, reinforcement learning usually means having the model generate multiple candidate answers to the same question, then reinforcing high-scoring outputs and suppressing low-scoring ones based on a reward signal; crude reward design can induce score-gaming behavior.

The research team converted raw PDF files into structured question-answer pairs stored in JSON format for use in supervised fine-tuning of the model.

The 'repetition' phenomenon is a typical failure mode for large models in agent scenarios, usually stemming from reward design flaws during the reinforcement learning phase or a lack of effective termination judgment mechanisms.

Supervised Fine-Tuning (SFT) uses a small dataset of several hundred thousand high-quality conversation examples and takes hours rather than months.

Supervised learning can only imitate the local plausibility of each step and cannot guarantee the global consistency of the entire conversation trajectory.

TRACER employs a two-stage training process: the first stage performs supervised fine-tuning (SFT) on real user conversations, and the second stage introduces multi-turn reinforcement learning (RL).

Research directions for improving model calibration include introducing uncertainty tokens through supervised fine-tuning, performing post-calibration on probability distributions using temperature sampling, and training models to proactively output the ability to refuse to answer.

Behavioral constraints imposed through supervised fine-tuning (SFT) can sometimes be overridden at the weight level with as few as hundreds of adversarial examples.

The base model is Qwen3-14B, from which two versions—KGModel and CGModel—are trained separately via supervised fine-tuning (SFT).

典型的SFT数据集可能只有数万到数十万条样本，远小于预训练数据规模