What Is SFT Actually Training? Breaking Down the Core Logic of Supervised Fine-Tuning

SFT (supervised fine-tuning) is the key step that teaches a pretrained model to "talk properly," but what is it actually training? Based on a technical talk from Bilibili, this article breaks down the definition, training objective, and design logic of SFT, and confronts its inherent limitations.

What SFT Is: Behavior Cloning by Showing the Model Correct Answers

SFT, short for supervised fine-tuning, involves researchers preparing a large number of "instruction–ideal response" data pairs. For example, if the instruction is "Please explain photosynthesis in a way a primary school student can understand," the ideal response avoids textbook jargon like "chloroplast" and instead uses an analogy like "sunlight is energy," while covering the inputs (light, water, carbon dioxide) and outputs (nutrients, oxygen), keeping the length restrained and skipping unnecessary preamble.

A good piece of SFT data teaches several things at once: how to understand constraints, how to organize language, how to control length, and not to make things up. In essence, SFT is behavior cloning—it doesn't reason with the model but shows it tens of thousands of correct answers and lets it figure things out on its own. It's like teaching a child to use chopsticks: you don't explain the lever principle, you demonstrate how to use them.

The corresponding data format is an (X, Y) pair: X is the instruction, Y is the reference answer. The training objective is, upon seeing instruction X, to increase the probability of generating reference answer Y.

Training Objective: Same Loss Function as Pretraining, Different Data

SFT's loss function is almost identical to pretraining: for each token of Y, given the instruction and the already-generated portion, compute the probability of the correct token, take the negative log, and sum. What changes is not the principle but the data.

In practice, loss is usually computed only on the response portion, and the tokens of the instruction portion are masked out. Why? Because we want it to learn "how to answer," not "how to write questions." If the instruction counted toward the loss, the model would incidentally practice the skill of asking questions too, and you might then encounter it asking and answering itself—restating the question and then continuing with a new question.

The basic principle of language models predicting the next token is unchanged: the optimizer is the same, the architecture is the same, and the forward pass is the same. The training data is replaced with a clearer structure: the user poses a task, the assistant executes it. So the same continuation machine repeatedly practices a new pattern.

I especially want to emphasize here: SFT does not install a chat module into the model; it performs a large-scale reweighting of the probability distribution. Originally, the probability of "snail" being followed by "choose" is 0.3, and by "answer" is 0.1; after SFT, "answer" becomes 0.9, and other paths still have some probability, just suppressed.

Negative log-likelihood (NLL) loss is the key mathematical tool here. Given the model's predicted probability distribution $p_\theta$ for the next token, for each token $y_t$ in the reference answer, the loss is $-\log p_\theta(y_t \mid X, y_{<t})$, i.e., the negative log of the probability of the correct token. The closer the probability is to 1, the closer the loss is to 0; the more "surprised" the model is, the greater the penalty. Summing or averaging the losses over all tokens gives the loss for the entire response. This is also the concrete form of cross-entropy loss in language modeling—minimizing NLL is equivalent to maximizing the model's likelihood of the reference answer, i.e., moving the model in parameter space so that these reference answers become more probable. Understanding this makes it clear why SFT is "reweighting" rather than "writing a module": it simply points the gradient in the direction of increasing the probability of specific sequences.

Why It's Designed This Way: Teaching the Model the Instruction Protocol and a Dictionary of Verbs

A pretrained model sees all kinds of text without a unified convention. SFT uses a unified data structure to tell the model: this segment is the user's request, and the next segment is the response the assistant should give. More importantly, it incidentally teaches the model a "dictionary of verbs"—summarize means compress information, translate means preserve meaning while changing language, explain means provide a clear explanation, output JSON only means a specific format, answer in three points means control the structure, no more than 100 words means control the length, and continue revising means refer to the preceding conversation.

A base model certainly recognizes the two characters "summarize"; it has seen them millions of times in the corpus. But what it understands is the semantics and contextual distribution of the word "summarize," not "when someone now tells me to summarize, I should perform a compression operation." The difference is something like this: a foreigner who has memorized all the dish names on a menu versus one who can understand the waiter asking "Would you like it spicy?" and respond—these are two different things. What SFT teaches is not vocabulary but the rules for using language in interaction.

A real observation: the ability to follow format instructions like outputting JSON is almost entirely built up by data in the SFT stage—however many formats appear in the data, that's roughly how many formats the model can stably output. The base model can already translate, summarize, write code, extract information, and rewrite text, but it can stably invoke these abilities only when users carefully design prompts. SFT compresses complex templates plus multiple examples into a single natural-language instruction. Research shows that fine-tuning on multiple tasks described by natural-language instructions can significantly improve the model's zero-shot performance on unseen tasks—what the model learns is not these tasks but the meta-skill of "following instructions." Of course, the effect depends on model scale and data composition; some papers point out that instruction fine-tuning may be harmful to small models: their capacity is already tight, and making them spend parameters learning the "etiquette" of instruction formats leaves no room for actual skills—like asking a rookie intern to memorize three days of company conduct rules, which may cause their professional ability to regress.

From a product perspective, the real significance of SFT is to shift the cost of prompt engineering from every user to a one-time training run. Previously, every user had to figure out prompts themselves; now the vendor teaches the model once and for all. This is also the key to ChatGPT's success—ordinary people don't know how to write prompts, and shouldn't be required to.

SFT can also form a stable assistant style: answer the question first and then add explanation, don't arbitrarily continue the user's words, use clear subheadings, avoid irrelevant content, adjust expression to the user's level, ask for clarification when information is missing, and continue the conversation according to multi-turn context. Can these things be written into the loss function? No—"clarity" has no gradient, and "not verbose" has no analytical form. It can only rely on demonstration: if you write thirty thousand clear responses, the model can absorb the statistical features of the "clear" style.

Zero-shot generalization refers to a model's ability to complete a task using only a natural-language instruction, without having seen any training samples for that specific task. This differs from traditional supervised learning: traditional methods train a separate classifier for each task, and changing the task requires relabeling and retraining; instruction fine-tuning, after training on the "instruction–response" format, enables the model to invoke existing abilities when faced with a brand-new task description. Work such as FLAN and T0 systematically verified this—after fine-tuning on a large number of natural-language instruction tasks, the model's zero-shot performance on unseen tasks improved significantly. The mechanism is precisely the "meta-skill" mentioned in the article: what the model learns is not a specific task mapping but the general behavioral pattern of "see an instruction, execute the corresponding operation." This also explains why task diversity in instruction data matters more than sample size for a single task.

The Costs of SFT: Expensive, Ambiguous, No Preferences

Problem one: expensive. For simple tasks, ordinary annotators can write the data, but medicine, law, mathematics, and complex dialectical questions often require experts. Explaining an abnormal ECG requires a doctor; analyzing the risks in contract clauses requires a lawyer. Writing a high-quality answer from scratch takes a long time—for a somewhat difficult math problem, an expert writing a complete, clear step-by-step solution might take more than half an hour. Thirty thousand expert data points at an average of twenty minutes each, calculated at expert hourly rates, is a considerable expense—this is why the entire industry is looking for alternatives.

Problem two: ambiguous answers. A question may have many correct answers. "Help me write a polite leave request email" has no single standard answer; it can be formal or friendly, three lines or three paragraphs. SFT usually provides only one or a few versions. When a question has five equally good answers and you show the model only the second, you are telling it that the second is right and the other four are not—the intent is to provide a sample, but the effect is to give the model a preference. The model's style will quietly become bound to the tastes of a few annotators.

Problem three: good at imitation, bad at expressing subtle preferences. Answer A is complete in content but very verbose; answer B is concise and clear but omits some details—which is better? You may already be hesitating—your hesitation confirms the problem: this is a trade-off, with no right or wrong. Given only one standard answer, the model struggles to understand how to trade off conciseness and completeness. SFT's loss function only says "copy this template"; it does not understand "this is slightly better than that." SFT has only positive samples, no negative samples, and no ranking. It pushes probability toward a single point, but human preference is essentially a comparative relation—A is better than B is an ordering structure, and you cannot express "this is better than that" with the grammar of "copy this."

Data Sources and Quality Gates

There are roughly six paths for SFT data sources: written by human experts, written by ordinary annotators according to specifications, curated from existing text data, generated by stronger models and then filtered, desensitized and organized from real user questions, and converted from task data into natural-language instructions.

Human expert writing is the most expensive—back then OpenAI hired a labeling team of dozens of people and specifically screened the annotators. Generation by stronger models has been the mainstream since 2023; the industry calls it "distillation": throw in a few hundred seed instructions and let a strong model expand them into tens of thousands, cutting costs from hundreds of thousands of dollars to a few hundred. But there are three pitfalls: error inheritance (if the teacher is wrong, the student follows and is wrong more confidently), style convergence (the entire open-source ecosystem increasingly looks like the same person), and compliance risk (many commercial models explicitly prohibit using their outputs to train competing models).

Converting task data into natural-language instructions is the cheapest and most elegant path: decades of annotated datasets accumulated in academia, such as sentiment classification and named entity recognition, were originally for small models, but now adding a template immediately turns them into SFT data.

A high-quality SFT data point must pass these gates: whether it truly answers the user's question, whether it follows the format, whether it is clear and easy to understand, whether it contains fabricated content, whether it is too verbose, whether it meets safety requirements, and whether it matches the target user group. Here is a counterintuitive fact: hallucinations in SFT data will be learned by the model as a role model. If an annotator is too lazy to look up sources when writing an answer and casually makes up a reference, what the model learns is not that specific fake reference but the behavior of confidently making up something that looks plausible in that position—you are teaching a behavioral pattern, not a fact. This is why serious teams' SFT data must undergo fact-checking and why it costs more. The effectiveness of SFT depends to a large extent on data quality rather than data quantity.

Knowledge distillation originally referred to having a small model imitate a large model's output distribution to compress the model, but in the context of SFT, "distillation" is often generalized to using a strong model to generate training data to teach a weaker model (or a model of the same scale). The key difference between the two is that classic distillation transfers soft labels (probability distributions), while SFT distillation transfers hard samples (generated text). Therefore it inherits the teacher model's errors and style but cannot transfer the teacher's uncertainty. This is also the root of "error inheritance" and "style convergence"—the student sees only the answers the teacher "confidently states" and not the part where the teacher is internally "hesitant." Commercial models' terms of use usually prohibit such distillation because the generated data itself is regarded as an extension of the model's capabilities.

Its Place in the Pipeline

Let's run through the whole pipeline: pretraining lets the model learn language knowledge and a large number of text patterns, consuming the vast majority of compute, yielding the base model—an expert full of knowledge but unversed in social niceties. SFT consumes negligible compute compared with pretraining, yet the model's personality changes dramatically: it learns to answer directly, follow formats, and execute tasks, yielding a preliminary chat model—an intern who can basically talk, not yet a reliable senior employee.

A plain analogy: pretraining gives the model a brain, SFT gives the model a mouth, and RLHF gives the model a sense of proportion. SFT cannot express the following: which of two answers is better, how to trade off correctness and conciseness, what style the user prefers, and which behaviors, although common, should not be generated. This leads to the next step: how RLHF uses human preferences to teach the model to choose the better one among multiple answers.

RLHF (reinforcement learning from human feedback) is the alignment stage after SFT. A typical process has three steps: first, collect human preference rankings over multiple responses to the same prompt, and train a reward model to predict "which response is better"; then use reinforcement learning (such as PPO) to optimize the language model so that its generated responses receive higher rewards, while using KL divergence constraints to prevent the model from deviating too far from the post-SFT distribution. Unlike SFT, which provides only positive samples, RLHF's core input is a comparative relation—A is better than B—which precisely makes up for SFT's inability to express "this is better than that." The cost is a more complex process, less stable training, and the reward model itself may be "gamed" (reward hacking). Later methods such as DPO attempt to bypass explicit reinforcement learning and optimize directly on preference data, but the basic idea remains using ranking signals to replace a single demonstration.

One-line Summary

SFT is essentially behavior cloning: it reweights the probability distribution with demonstration data, teaching the model the instruction protocol rather than new knowledge.

Paragraph Summary

This article breaks down the essence of supervised fine-tuning (SFT): it performs behavior cloning on a pretrained model, using a large number of "instruction–ideal response" data pairs to perform large-scale reweighting of the probability distribution, so that the model learns the instruction protocol and assistant style rather than being instilled with new knowledge. The training loss is consistent with pretraining, but loss is computed only on the response portion to prevent the model from learning to ask and answer itself. SFT's real value lies in shifting the cost of prompt engineering from every user to a one-time training run and forming a stable assistant style. Its limitations are also obvious: expert data is expensive, a single standard answer binds the annotator's preferences, and having only positive samples cannot express the comparative relation "A is better than B." Data quality matters more than quantity, and hallucinatory behavior will be learned by the model as a role model. In the pipeline, pretraining installs the brain, SFT installs the mouth, and RLHF installs the sense of proportion.