FSPO
A policy optimization framework for reinforcement learning of large models. Instead of a continuously evolving teacher model, it sparsely stores the next-token distribution output by historical policies as a fixed temporal anchor target. This addresses the objective drift problem in self-distillation and reduces GPU memory consumption.
Timeline (last 90 days)
FSPO (Function-Space Policy Optimization) proposes sparsely storing the next-token distributions of historical policies on specific inputs during training as fixed temporal anchor targets, rather than maintaining a continuously updated teacher model.
When FSPO is trained on Olmo3-7B-Instruct, peak memory drops from about 70GB to 56GB, saving 14GB.
FSPO contains three key components: Privileged Disagreement Correction, Reliability-Adaptive Grounding, and Quality and Coverage-Aware Temporal Memory.
On reasoning and tool-use benchmarks, FSPO achieved the highest average accuracy within a 5-hour training budget, about 4 percentage points higher on average than the strongest baseline.
FSPO directly operates in function space on the policy's output distribution (the next-token probability distribution given an input), avoiding parameter symmetry and optimization path complexity, and is more direct and stable than alignment in parameter space.
Once FSPO's anchors are stored, they are completely frozen and do not change with updates of the current policy, thereby mathematically guaranteeing the stationarity of the target distribution.
The FSPO paper's abstract does not disclose the specific benchmark names and baseline methods, so it remains unclear whether the 4% improvement is statistically significant and whether it is stable across all tasks.
The FSPO method introduces three additional components, each with hyperparameters that need tuning, and the tuning cost in practical use cannot be ignored.
FSPO's performance under a 5-hour training budget has been verified, but its performance under longer training cycles (especially whether the stored historical distributions become outdated) remains unclear.
The idea of FSPO is spiritually similar to experience replay, but it operates on policy distributions rather than state transitions.
All Facts (10)
The idea of FSPO is spiritually similar to experience replay, but it operates on policy distributions rather than state transitions.
50%UnverifiedWhen FSPO is trained on Olmo3-7B-Instruct, peak memory drops from about 70GB to 56GB, saving 14GB.
50%UnverifiedFSPO contains three key components: Privileged Disagreement Correction, Reliability-Adaptive Grounding, and Quality and Coverage-Aware Temporal Memory.
50%UnverifiedOn reasoning and tool-use benchmarks, FSPO achieved the highest average accuracy within a 5-hour training budget, about 4 percentage points higher on average than the strongest baseline.
50%UnverifiedFSPO directly operates in function space on the policy's output distribution (the next-token probability distribution given an input), avoiding parameter symmetry and optimization path complexity, and is more direct and stable than alignment in parameter space.
50%UnverifiedOnce FSPO's anchors are stored, they are completely frozen and do not change with updates of the current policy, thereby mathematically guaranteeing the stationarity of the target distribution.
50%UnverifiedThe FSPO paper's abstract does not disclose the specific benchmark names and baseline methods, so it remains unclear whether the 4% improvement is statistically significant and whether it is stable across all tasks.
50%UnverifiedThe FSPO method introduces three additional components, each with hyperparameters that need tuning, and the tuning cost in practical use cannot be ignored.
50%UnverifiedFSPO (Function-Space Policy Optimization) proposes sparsely storing the next-token distributions of historical policies on specific inputs during training as fixed temporal anchor targets, rather than maintaining a continuously updated teacher model.
50%UnverifiedFSPO's performance under a 5-hour training budget has been verified, but its performance under longer training cycles (especially whether the stored historical distributions become outdated) remains unclear.
50%