Microsoft TailSFT: Filtering Fitted Samples to Better Prepare SFT for RL

The Overlooked Gap Between SFT and RL

In the post-training pipeline of large language models, Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has become a near-standard two-stage paradigm: SFT first teaches the model to follow instructions and master task formats, then RL (such as GRPO or PPO) further improves reasoning and generation quality. However, Microsoft's research team posed a sharp question—Does standard SFT actually train a model suitable for RL?

Their answer: No.

While this conclusion seems simple, it touches on a structural issue in post-training pipelines that has long been overlooked. Most teams assume SFT is merely a "warm-up" before RL—the more thorough, the better. But Microsoft's research reveals a counterintuitive fact: overly thorough SFT can actually damage the exploration space needed for subsequent RL.

To understand the weight of this conclusion, we must first clarify the respective mechanisms of SFT and RL. Supervised Fine-Tuning builds on pretrained models by training on high-quality, human-annotated question-answer pairs, teaching the model to generate responses that match expected formats and content according to instructions. The RL stage then uses reward signals to guide the model toward higher-quality outputs—the model samples multiple candidate responses to the same question and updates its policy based on feedback from reward models or rule-based verifiers. GRPO (Group Relative Policy Optimization), mentioned here, is a reinforcement learning algorithm proposed by DeepSeek that samples a group of responses to the same question and calculates group-relative rewards to estimate advantage functions, eliminating the overhead of value networks required in PPO. PPO (Proximal Policy Optimization) is a classic policy gradient algorithm from OpenAI that clips the objective function to limit update magnitudes and ensure training stability. While mechanically different, both rely on the model's ability to sample sufficiently diverse candidate responses.

The Hidden Cost of Standard SFT: Overfitting Compresses Exploration Space

Researchers identified a core flaw in standard SFT: during training, the model continuously consumes gradient updates on sequences it has already fitted well.

When the model has already learned certain types of samples, continued training on them brings no substantial improvement. Instead, it progressively narrows the model's output distribution—making it increasingly "certain" and "confident."

This is fatal for RL. The essence of reinforcement learning is exploring and trial-and-error over a probability distribution, using reward signals to find better generation paths. If SFT has already compressed the model's distribution too tightly, RL lacks sufficient exploration room in subsequent stages. Fewer diverse candidate answers can be sampled, naturally reducing learning efficiency.

From an information-theoretic perspective, this problem becomes even clearer. In reinforcement learning, balancing "exploration" and "exploitation" is a core challenge. The model must trade off between known high-reward paths (exploitation) and insufficiently tried new paths (exploration). When SFT over-converges, the model's output probability distribution trends toward a peaked distribution—entropy drops dramatically, with most probability mass concentrated on a few token sequences. This means during RL sampling, the model will almost exclusively generate very similar responses, unable to effectively explore other potentially superior generation paths. RL algorithms typically require the initial policy to maintain sufficient entropy for effective policy improvement, but excessive SFT destroys precisely this prerequisite.

This creates a paradox: doing SFT "too well" may deprive the model of the exploration potential most needed during the RL stage.

TailSFT Core Mechanism: Dynamic Filtering of Fitted Samples

Addressing this issue, Microsoft proposed a method called TailSFT. Its core idea is remarkably restrained—make only one modification.

Focus on Under-Modeled Tail Data

TailSFT dynamically identifies and filters out sequences the model has already fitted well during training, concentrating learning resources on the "under-modeled tail" of the data distribution.

The "tail" concept here originates from long-tail distribution theory in statistics. In training data, the difficulty distribution of different samples typically exhibits long-tail characteristics: a large number of samples belong to the "head" that the model easily learns, while a small number of difficult samples form the "tail." Traditional SFT treats all samples equally in gradient computation, causing head samples to contribute gradient signals in late training that are essentially noise—their loss is already low, and continued optimization only makes the model over-confident in these patterns. TailSFT dynamically evaluates the fitting degree of each sequence during training (typically based on sequence-level loss or perplexity), excluding samples with sufficiently low loss from gradient computation in the current batch, thus achieving an adaptive curriculum learning effect.

By focusing gradients on these difficult samples not yet fully learned, TailSFT both avoids ineffective repeated training on simple samples and preserves diversity in the model's output distribution, thereby retaining broader exploration space for subsequent RL.

Worth emphasizing: this is the only modification the research team implemented. Compared to introducing complex new architectures or training objectives, this "minimally invasive" design is more easily adopted by existing pipelines and more persuasive—performance improvements can be clearly attributed to this single change.

Experimental Results: Dual Improvement in Coverage and Final Performance

The research validated on the OLMo-3 7B model with compelling results. OLMo (Open Language Model) is a fully open-source large language model series led by the Allen Institute for AI (AI2), with OLMo-3 being its third generation. Choosing this model has methodological significance: its training data, code, and weights are completely open-source, ensuring experimental reproducibility and credibility. The 7B parameter scale is currently the most commonly used in academic research—large enough to demonstrate method effectiveness without requiring extreme computational resources, facilitating verification and follow-up by other teams.

Answer Coverage Improvement in SFT Stage

Using the pass@16 metric to measure the model's answer coverage ability (probability of getting it right at least once in 16 samples):

• Programming tasks: absolute improvement up to 16.8 points
• Math tasks: absolute improvement up to 3.1 points

pass@k is a commonly used evaluation metric in code generation and mathematical reasoning, systematically defined in OpenAI's Codex paper. pass@k measures the probability that among k independent samples for the same problem, at least one is correct. pass@1 reflects single-generation accuracy, while pass@16 reflects the model's ability to cover correct answers across 16 independent samples. The difference between them reveals important distributional properties: if pass@16 is much higher than pass@1, the model's probability distribution contains correct answers but hasn't placed them at the highest probability—this is precisely where RL can play a role, using reward signals to boost the generation probability of correct answers.

The pass@16 improvement means the model retains richer correct answer candidates—the "raw material" needed for RL exploration.

Benefit Propagation in RL Stage

More critically, these checkpoints with higher coverage also showed improved final pass@1 (single-sample accuracy) after GRPO reinforcement learning:

• Final pass@1 improvement up to 3.9 points
• In some settings, early reward climb rate was 2.5x faster than standard SFT baseline

This data completes a full causal chain: TailSFT preserves a broader exploration distribution → RL stage has more exploration space → faster reward convergence and higher final performance.

Implications for Large Model Post-Training Practice

TailSFT's value lies not only in specific performance numbers but in redefining SFT's role positioning in the post-training pipeline.

Current mainstream large model post-training pipelines typically include multiple stages: SFT (instruction tuning) → reward model training (or using rule-based verifiers) → RL alignment (such as RLHF/GRPO) → optional DPO and other direct preference optimization. This paradigm was first established by the InstructGPT paper and has been widely adopted by mainstream models like ChatGPT, Claude, and Gemini. However, coupling effects between stages have long lacked systematic research, with most teams relying on empirical tuning to coordinate training intensity across stages. TailSFT's work fills precisely this theoretical and methodological gap in the critical transition from SFT to RL.

For a long time, the industry has tended to view SFT and RL as two independently optimized stages, each pursuing stage-specific optima. But Microsoft's work reminds us: SFT should not only be responsible for itself but for the entire post-training process. A "perfectly converged" SFT model may not be the best starting point for RL.

For teams building large model post-training pipelines, this brings several practical considerations:

• Reassess SFT training intensity: Over-training may invisibly weaken downstream RL potential.
• Focus on distributional diversity rather than pure fitting accuracy: Coverage metrics like pass@k may better predict RL-stage performance than traditional loss or pass@1.
• Low-cost transformation feasibility: TailSFT only requires sample-level filtering without modifying model structure or RL algorithms, with relatively low engineering implementation barriers.

Conclusion

As the large model capability race intensifies, algorithmic "brute force scaling" is gradually giving way to refined understanding of the training process itself. TailSFT exemplifies this trend—it introduces no dazzling new concepts but finds a leverage point through deep insight into the SFT-RL relationship.

For researchers and engineers following post-training technology evolution, this work provides an important perspective: Before pursuing single-stage optimality, first think clearly about what kind of model the next stage truly needs.

For the paper and details, see the research materials officially released by Microsoft.