[KongchangAI]
· 2 min read· 1,128 words

Ambient LLMs: Data-Efficient SFT via Temperature Matching

Ambient LLMs: Data-Efficient SFT via Temperature Matching

Ambient SFT mitigates overfitting and forgetting in small-data fine-tuning by matching the pretrained model's high-temperature distribution, achieving an average 21% improvement.

This article introduces a fine-tuning framework called Ambient SFT, designed to address overfitting and catastrophic forgetting caused by small supervised fine-tuning datasets. The method introduces a frozen pretrained model as a noisy teacher, requiring the fine-tuned model to match it on a coarse-grained conditional distribution softened at high temperature, thereby injecting pretrained distribution knowledge as a regularization signal into the fine-tuning process. The temperature hyperparameter T controls the degree of trust in the pretrained model: when T tends to infinity it degenerates to standard SFT, and when T=1 it is equivalent to knowledge distillation. Experiments on domain adaptation, instruction tuning, and reasoning tasks show an average relative improvement of 21%, without relying on additional annotated data. The article also notes from a theoretical perspective that Ambient SFT constructs a continuum between standard SFT and knowledge distillation, offering a flexible and tunable practical solution for data-efficient fine-tuning.

Large-scale, diverse, and high-quality supervised fine-tuning data is hard to obtain, causing SFT to often overfit on small datasets and forget the long-tail knowledge in the pretraining distribution. A recent study proposes the Ambient SFT framework, which alleviates this problem by having the fine-tuned model match the "high-temperature" conditional distribution of the pretrained model at a coarse granularity, achieving significant improvements across multiple tasks.

Ambient LLMs paper title screenshot

Core Problem: Overfitting and Forgetting in Small-Data SFT

Pretraining of autoregressive language models relies on massive data, but subsequent SFT datasets are often much smaller. This scale disparity makes models prone to overfitting to limited samples during fine-tuning and to catastrophically forgetting the tail knowledge of the data distribution learned during pretraining—that is, those infrequent but important language patterns. Traditional SFT only maximizes the likelihood of annotated data and lacks a mechanism to preserve pretrained knowledge, thus limiting performance in small-data scenarios.

Catastrophic Forgetting is a classic problem in continual learning for neural networks: when a model is trained on a new task or new data, its parameter updates overwrite previously learned knowledge, causing a sharp drop in performance on old tasks. In the language model setting, the model learns rich language patterns during pretraining on massive corpora, including low-frequency but important long-tail knowledge; whereas the SFT stage has small data volume and narrow distribution, and gradient updates can easily pull the model toward the local region where the annotated data lies, thereby losing the tail knowledge in the pretraining distribution. This phenomenon is particularly pronounced when the fine-tuning learning rate is large or the data volume is extremely small, and it is one of the core contradictions that data-efficient fine-tuning must focus on alleviating.

Ambient SFT: Constraining the High-Temperature Distribution with a Frozen Teacher

The core idea of Ambient SFT is to use the "imperfect predictions" of the pretrained model to enhance fine-tuning. Specifically, a frozen pretrained model serves as a noisy teacher, requiring the fine-tuned model to match it at a coarse granularity. The object being matched is not the original conditional distribution but its high-temperature version: the high-temperature conditional distribution produced by the fine-tuned model should be consistent with the high-temperature version of the pretrained model.

The temperature hyperparameter T controls the degree of trust in the pretrained model: when T tends to infinity, the method degenerates to standard SFT; when T=1, it becomes knowledge distillation. By adjusting T, one can strike a balance between preserving pretrained knowledge and adapting to new tasks. This design is equivalent to adding a regularization term beyond the SFT loss, encouraging the fine-tuned model not to deviate too far from the pretraining distribution.

Knowledge Distillation was proposed by Hinton et al. Its core idea is to have the student model imitate the soft prediction distribution of the teacher model rather than merely fitting hard labels. Soft predictions contain relative probability information between classes (e.g., the similarity between "cat" and "dog"), and this "dark knowledge" can provide richer supervisory signals than one-hot labels. The temperature parameter T commonly used in distillation softens the softmax distribution: the larger T is, the smoother the distribution, amplifying inter-class differences; T=1 corresponds to the original softmax. Ambient SFT borrows this temperature mechanism, but the teacher is not an independently trained strong model; rather, it is the pretrained version of the fine-tuned model itself, and matching occurs on the coarse-grained distribution softened at high temperature, thereby balancing the preservation of pretrained knowledge and adaptation to downstream tasks.

Experimental Performance: Average Relative Improvement of 21%

The paper evaluates Ambient SFT on domain adaptation, instruction tuning, and reasoning tasks. Results show that the method outperforms standard SFT in all these scenarios, with an average relative improvement of 21%. This means that with the same amount of data, the model can more effectively leverage pretrained knowledge, reducing overfitting and forgetting. Notably, the improvement does not come from additional annotated data but is achieved by fully utilizing the prediction distribution of the frozen model.

This result also suggests that the temperature matching mechanism may be a general data-efficient fine-tuning strategy applicable to various downstream tasks.

Theoretical Perspective: A Continuum Between SFT and Distillation

From the role of the temperature hyperparameter, it can be seen that Ambient SFT constructs a continuum between standard SFT and knowledge distillation. Standard SFT focuses only on hard labels, knowledge distillation relies on the teacher's soft predictions, and Ambient SFT aligns teacher and student at a coarse granularity by softening the distribution at high temperature, avoiding both the overfitting risk of hard labels and the bias that may arise from over-reliance on teacher predictions.

This continuity provides practitioners with a flexible tuning knob: when annotated data is scarce, one can lower T to rely more on pretrained knowledge; when data is relatively abundant, one can raise T to focus more on task-specific signals.

KL Divergence (Kullback-Leibler Divergence) is a commonly used metric for measuring the difference between two probability distributions, defined as (D_{KL}(P|Q)=\sum_x P(x)\log\frac{P(x)}{Q(x)}). In knowledge distillation and Ambient SFT, the loss function usually consists of a task loss and a KL regularization term: the former brings the model output closer to hard labels, and the latter constrains the model distribution not to deviate too far from the teacher distribution. Temperature T plays a key role here—at high temperatures the softmax output tends toward uniformity, and the KL term's constraint on the distribution becomes coarse-grained, requiring only that the model remain consistent with the teacher on larger class groupings, thereby avoiding overfitting to the teacher's specific prediction biases. This coarse-grained matching is precisely the mathematical basis that distinguishes Ambient SFT from standard distillation and enables flexible regularization.

Implications for Data-Efficient Fine-Tuning

The proposal of Ambient SFT echoes a key pain point in the current deployment of large models: high-quality annotated data is expensive and scarce. By using the pretrained model itself as a source of regularization signals, this method improves fine-tuning performance without increasing data collection costs. In the future, this idea may be extended to more modalities and tasks, or combined with parameter-efficient fine-tuning techniques to further lower the barrier to fine-tuning.

For practitioners, Ambient SFT offers a simple and implementable improvement: introducing a temperature matching term into the standard SFT loss yields more robust models in small-data scenarios. The paper's code and implementation details are worth attention.

Conclusion

Ambient SFT injects the distribution knowledge of the pretrained model into the fine-tuning process through temperature matching, effectively alleviating the overfitting and catastrophic forgetting problems of small-data SFT. Experiments show that it outperforms standard SFT on domain adaptation, instruction tuning, and reasoning tasks, with an average relative improvement of 21%. This method provides a new perspective for data-efficient fine-tuning and leaves a flexible design space for tuning the temperature hyperparameter.

#Ambient LLMs#Ambient SFT#temperature matching#supervised fine-tuning#catastrophic forgetting
Share:

Related articles