Why Does SFT Forget More Than RFT? Style Drift Reveals the Hidden Cost of Supervised Fine-Tuning

A persistent puzzle in the practice of continual training and fine-tuning large language models is that Supervised Fine-Tuning (SFT) tends to cause more severe catastrophic forgetting than Reinforcement Fine-Tuning (RFT). What makes this counterintuitive is that even when teacher demonstrations are semantically correct, SFT still causes the model to lose previously acquired knowledge. A newly released arXiv paper, Learning Style, Forgetting Semantics, offers a rigorous theoretical explanation for this phenomenon.

The Core Question: Why Does Semantic Correctness Still Lead to Forgetting?

The paper focuses on classification tasks as a relatively controlled setting. In this context, different tokens within the same semantic class express the same semantic answer but differ in "style." Multiple tasks share the same underlying semantic rules, with differences only in prompt distribution and teacher style preferences.

This setup elegantly decouples the two dimensions of "semantics" and "style." The central question is: when all teacher demonstrations are semantically correct, where does forgetting come from? The answer lies not at the semantic level, but hidden in the dynamics of style.

Catastrophic Forgetting is a classic challenge in neural networks during sequential multi-task learning: as the model learns a new task, performance on previous tasks degrades sharply. The root cause is that gradient updates overwrite weights critical to earlier tasks. Traditional mitigation strategies include Elastic Weight Consolidation (EWC) and experience replay, but these approaches address the problem at the level of parameter regularization or data, without analyzing the structural differences in update directions between training paradigms (SFT vs. RFT). This paper's contribution lies precisely here: rather than asking "how do we prevent forgetting," it asks "why does SFT's update structure make forgetting structurally more likely" — elevating the question from an engineering patch to a mechanistic analysis.

Theoretical Tools: Exact Decomposition of the Linear-Softmax Policy

To make the problem tractable, the authors adopt a linear-softmax policy model. Based on this, they derive an exact decomposition of parameter updates, explicitly splitting each update into a semantic component and a style component.

This decomposition is the analytical cornerstone of the paper. It allows researchers to directly compare the update directions of SFT and RFT under the same policy and prompt conditions, without relying on vague empirical observations.

The linear-softmax policy is a common tractable approximation model used in research at the intersection of reinforcement learning and NLP: the output layer maps linearly transformed hidden states to a probability distribution via softmax. While simpler than a full Transformer, this setup preserves the core algebraic structure of classification decisions and allows closed-form derivations of gradients. In SFT, the update direction is determined by the log-probability gradient of the teacher token; in RFT (or policy gradient methods), it is determined by the reward-weighted gradient of the model's own sampled outputs. This structural difference is the starting point of the paper's decomposition — the semantic component points in the same direction for both, but RFT's style component cancels out due to the symmetry of the reward signal, while SFT leaves a nonzero residual due to the teacher's nonuniform style preferences.

Key Finding: Aligned Semantic Updates, Divergent Style Dynamics

The paper's central conclusion can be summarized in one sentence: SFT and RFT produce parallel (aligned) semantic updates, but their style dynamics diverge fundamentally.

Specifically:

• RFT preserves symmetry: Starting from a policy with no intra-class style bias, RFT using exact policy gradients preserves this symmetry. In other words, RFT does not introduce style bias from scratch.

• SFT produces off-axis style drift: When the teacher has nonuniform style preferences, SFT generates "off-axis style drift" along a nonzero task mean under population updates. This drift is the root cause of forgetting.

This contrast reveals a profound mechanism: SFT is more prone to forgetting not because it learns the wrong semantics, but because it is forced to imitate the teacher's style preferences, causing unnecessary parameter shifts in the style subspace.

Population updates refer to parameter updates taken in expectation over multiple tasks or data distributions, as opposed to single-sample stochastic gradient updates. Analyzing style drift at the population update level means the conclusions are free from stochastic noise and reflect systematic biases inherent to the training paradigm itself. "Intra-class style symmetry" means that tokens of different styles within the same semantic class appear with uniform frequency, and the model has no prior preference for any of them. The result that RFT preserves this symmetry under this assumption shows that, as long as the initial policy is style-neutral, policy gradient updates do not break this uniformity. SFT, by directly imitating the teacher, "transfers" the teacher's style preferences into the model weights — even when those preferences contribute nothing to semantic correctness.

Theoretical Separation: A Strict Lower Bound on SFT Forgetting

The paper further establishes a "separation" result between the two methods under explicit conditions. For population updates starting from the same perfectly fitted checkpoint:

• SFT forgetting has a strict positive lower bound: Over a finite training interval, the forgetting caused by SFT is bounded below by a value strictly greater than zero — meaning forgetting is mathematically unavoidable.

• RFT maintains zero semantic error: By contrast, RFT can maintain zero semantic error.

This separation result theoretically demonstrates that RFT's advantage in resisting forgetting is not accidental, but is determined by the fundamental differences between the two training paradigms. The authors also validate these theoretical predictions through simulation experiments on task sequences.

Practical Implications

Although the paper is built on a simplified classification task and a linear-softmax model, its insights remain relevant to real-world LLM training.

For continual learning and multi-task fine-tuning scenarios, this work suggests that style-level information — even when semantically correct — can act as a hidden factor that undermines existing capabilities. When teacher data carries style preferences, directly imitating those demonstrations via SFT encodes unnecessary style shifts into model parameters.

This also indirectly explains why reward-signal-based methods such as RLHF and RFT have proven more robust at preserving general model capabilities in recent years — they optimize for correctness of outcomes rather than token-by-token imitation of a specific style, thereby avoiding the side effects of style drift.

RLHF (Reinforcement Learning from Human Feedback) is the core alignment paradigm for mainstream LLMs today: a reward model is first trained on human preference annotations, and the language model is then fine-tuned using algorithms such as PPO guided by that reward model. Unlike SFT's token-by-token imitation, RLHF optimizes the reward score of the overall output sequence, allowing the model to freely explore in style space as long as the final result is approved by the reward model. The theoretical framework in this paper provides a new mathematical mechanistic explanation — grounded in the structure of parameter updates — for why RLHF/RFT better preserves general capabilities: the outcome-oriented nature of reward signals produces a natural cancellation effect in style component updates, rather than hardcoding the teacher's specific style preferences into model weights as SFT does. This perspective also suggests that, when designing continual learning schemes, incorporating reward signals into the fine-tuning pipeline (even rule-based rewards rather than human annotations) may be more favorable for capability retention than pure SFT.

Conclusion

The value of this paper lies in distilling a widely observed empirical phenomenon into a mechanism that can be precisely characterized mathematically. The title "learning style, forgetting semantics" captures the hidden cost of SFT with precision. For researchers and engineers focused on LLM training methodology, understanding how semantics and style decouple in parameter updates can inform the design of fine-tuning strategies that are more resistant to forgetting.