FSPO: Function-Space Temporal Self-Distillation for More Stable and Efficient Policy Optimization

Reinforcement learning is becoming a key path to improving the reasoning capabilities of large models. Reinforcement Learning with Verifiable Rewards (RLVR) optimizes models based on whether the final answer is correct, but this approach provides overly coarse supervision of the reasoning process—a correct answer may conceal a large number of inefficient or even erroneous intermediate steps. Self-distillation attempts to compensate for this deficiency with finer-grained token-level signals, but it brings a thorny problem: the teacher model evolves in sync with the student, so the target signal is constantly drifting.

A recent study proposes Function-Space Policy Optimization (FSPO), which redesigns the self-distillation framework using a "frozen history" approach. The core idea is to store the next-token distribution output by the policy at some past moment as a fixed target, rather than maintaining a continuously updated teacher model. This article analyzes the practical value and limitations of this work from the perspectives of method design, key technical components, and multi-source evaluation data.

The Granularity Dilemma of RLVR and the Instability Trap of Self-Distillation

RLVR works in a straightforward way: it gives a scalar reward to the complete response generated by the model (usually based on answer correctness), and then updates parameters using policy gradients. The problem is that a 1000-token reasoning chain ultimately corresponds to only a single numerical reward, and the model cannot tell from it "whether the decision at token 37 is good or bad." This coarse-grained supervision can easily lead to inefficient credit assignment in large-scale training.

Self-distillation provides an intuitively reasonable solution: let a teacher model with "extra information" guide the student model's distribution at each token. But where does the teacher's extra information come from? A common approach is to let the teacher access the standard answer or a longer reasoning budget, which is a form of "privileged information."

The contradiction lies here. If the teacher is an exponential moving average (EMA) of the student policy or some synchronous variant, then every update of the student changes the teacher, and the target signal drifts accordingly. The training process becomes chasing a moving target. Existing countermeasures introduce additional regularization terms or maintain a separate teacher model, but this increases memory overhead and tuning complexity.

Credit assignment is a core challenge in reinforcement learning: when a trajectory ultimately receives only a scalar return, how can we determine the contribution of each action in the trajectory to the final outcome? In traditional RL, methods such as temporal-difference (TD) learning and generalized advantage estimation (GAE) are commonly used to assign credit by tracing back step by step. But in large model reasoning scenarios, a 1000-token chain of thought usually has only one signal: whether the final answer is correct, with no direct feedback on intermediate steps. This leads to extremely high variance in policy gradient estimates, making it difficult for the model to learn "which reasoning step is critical" or "which step is redundant." Although RLVR avoids the design of a reward model, it pushes the credit assignment problem to a more extreme form: the entire reasoning chain shares one sparse terminal reward. This is also why researchers turn to token-level self-distillation signals—they naturally provide dense supervision at each step, but at the cost of introducing an unstable dynamic target.

FSPO's Core Design: Temporal Anchors in Function Space

FSPO's solution has considerable engineering elegance. It no longer maintains a "live" teacher. Instead, during training it sparsely stores the next-token distribution of historical policies on specific inputs and uses it as a fixed target (temporal target). These stored distributions act as temporal anchors in function space—the student's current output must align with its past self (the distribution produced with the help of privileged information).

This has two benefits. The first is temporal isolation: the target no longer changes with the current parameters, eliminating the root cause of target instability in self-distillation. The second is memory efficiency: there is no need to load two models simultaneously, and peak memory drops from about 70 GB to 56 GB (on Olmo3-7B-Instruct). For training models at the 7B level, saving 14 GB of memory means running on fewer GPUs or allowing a larger batch size.

The storage strategy is "sparse"—not every training step is recorded, which further controls storage overhead. The specific storage frequency and distribution selection strategy should have detailed ablations in the original paper, but the core idea is to use a small number of key anchors to cover the main stages of the training trajectory.

Function space here is contrasted with parameter space. Traditional policy optimization updates weights in parameter space, whereas FSPO chooses to directly operate in function space on the policy's output distribution, that is, the probability distribution of the next token given the input. The advantage of function space is that it bypasses parameter symmetry and the complexity of optimization paths: two different parameter configurations may produce the same or similar functional behavior, and directly aligning function outputs is more direct and stable than aligning parameters. The concept of temporal anchors draws on momentum encoders in self-supervised learning (such as MoCo) and trajectory modeling ideas in decision transformers: use a policy snapshot from a past moment as a fixed reference to prevent the target from drifting in real time during training. Unlike an EMA teacher, FSPO's anchors are not exponential moving averages in parameter space, but sparse sampled storage in function space. This means that once an anchor is stored, it is completely frozen and does not change with any update of the current policy, thereby mathematically guaranteeing the stationarity of the target distribution.

Three Key Components: Disagreement Correction, Reliability Adaptation, and Memory Quality

FSPO does more than just "store historical distributions." The paper also proposes three supporting mechanisms, each targeting a specific failure mode in the self-distillation setting.

Privileged Disagreement Correction addresses the problem of "local overestimation." When the privileged teacher's distribution diverges significantly from the student's distribution, the student may become overconfident on certain tokens. This mechanism explicitly suppresses these locally overestimated actions and prevents the policy from becoming entrenched in the wrong direction.

Reliability-Adaptive Grounding handles the adjustment of the strength of the reference supervision signal. When evidence generated by the model itself repeatedly fails, the system increases reliance on reference supervision. This is similar to a dynamic trust mechanism: the more uncertain the student is, the more it relies on external anchors.

Quality and Coverage-Aware Temporal Memory determines which historical distributions are worth storing. Not all historical outputs are equally important—verified behaviors are prioritized for retention, while ensuring that the stored distributions cover sufficiently diverse scenarios. This prevents the memory bank from being occupied by low-quality or redundant samples.

Together, these three components form a closed loop: disagreement correction prevents drift, reliability adaptation adjusts supervision weights, and memory management ensures anchor quality.

Empirical Performance and Objective Limitations

On reasoning and tool-use benchmarks, FSPO achieved the highest average accuracy within a 5-hour training budget, outperforming the strongest baseline by about 4 percentage points on average. This magnitude of improvement is significant in RLVR-related work—an average gain of 4 points usually means stable gains across multiple benchmarks rather than a single-point breakthrough.

The reduction in memory from 70 GB to 56 GB is also noteworthy. On the specific model Olmo3-7B-Instruct, this means a considerable reduction in training infrastructure cost.

There are also many areas that need to be viewed calmly. First, the paper abstract does not disclose the specific benchmark names and baseline methods. Whether the 4% improvement is statistically significant and whether it is stable across all tasks requires looking at confidence intervals and variance reports in the main text. Second, the 5-hour training budget is a relatively constrained setting, and FSPO's performance over longer training cycles—especially whether the stored historical distributions become outdated—remains unclear. Third, the method introduces three additional components, each with hyperparameters that need tuning, and the tuning cost in practical use cannot be ignored.

From a broader perspective, FSPO represents a shift in thinking: rather than letting the teacher and student evolve synchronously, it is better to leave "time capsules" in function space. This idea is spiritually similar to experience replay, but it operates on policy distributions rather than state transitions. It provides a structural solution to the stability problem of self-distillation, rather than patching it by adding regularization.

Privileged information in the self-distillation framework refers to information that the teacher model can access but the student model cannot obtain at inference time, such as standard answers, longer reasoning budgets, or additional verification signals. The "Learning Using Privileged Information" (LUPI) framework in learning theory shows that if the teacher can use this extra information to generate better supervision signals, the student can approach the teacher's performance while having only conventional inputs. But the introduction of privileged information also brings a fundamental contradiction: the quality of the teacher distribution depends on the reliability of the privileged information. If the privileged information itself is noisy or incomplete in coverage, the teacher distribution may give incorrect guidance on certain tokens, and the student, lacking a basis for judgment, may blindly imitate these errors. FSPO's three supporting components are precisely intended to alleviate this contradiction: disagreement correction suppresses tokens where the teacher and student deviate excessively, reliability adaptation falls back to reference supervision when teacher signals are unreliable, and memory management filters out low-quality privileged teacher outputs.

One-line Summary

FSPO replaces the evolving teacher with sparse historical distributions, stabilizing self-distillation and saving memory.

Paragraph Summary

To address the target drift caused by the teacher model evolving in sync with the student in self-distillation, FSPO proposes sparsely storing the next-token distributions of historical policies in function space as fixed temporal anchors, achieving temporal isolation and eliminating target instability. The method also introduces three components: Privileged Disagreement Correction, Reliability-Adaptive Grounding, and Quality and Coverage-Aware Temporal Memory, which address local overestimation, supervision weight adjustment, and anchor quality, respectively. On Olmo3-7B-Instruct, peak memory drops from about 70 GB to 56 GB, and average accuracy within a 5-hour training budget is about 4 percentage points higher than the strongest baseline. However, specific benchmark details, statistical significance, whether historical distributions become outdated over long cycles, and the tuning cost brought by additional hyperparameters remain limitations that practical applications need to verify.