Score Centering: A Numerical Solution for Stabilizing Off-Policy Reinforcement Learning

An Overlooked Problem: LLM Reinforcement Learning Is Extremely Sensitive to Numeric Issues

Reinforcement learning (RL) for large language models has become a cornerstone technique for alignment and reasoning improvement in recent years. Yet one long-underestimated fact remains: LLM RL training is extremely sensitive to numerical issues. In their paper Score Centering Stabilizes Off-policy Reinforcement Learning, researchers Marek and Ryabinin from Together.ai turn their focus to the stability challenges inherent in off-policy reinforcement learning.

In practice, policy updates often don't rely strictly on data sampled from the current policy — instead, they reuse samples generated by older policies collected in earlier stages. This is the hallmark of off-policy learning. While data reuse improves efficiency, it also introduces distribution shift and high-variance gradient estimates, making training prone to collapse or divergence.

The Core Idea Behind Score Centering

The method proposed in the paper is called Score Centering. Its fundamental motivation is that, in off-policy settings, the bias and variance carried by the score function used to estimate policy gradients are a key source of training instability.

"Centering" refers to mean-subtracting the score term so that it fluctuates around a more sensible baseline, thereby reducing the variance of gradient estimates. This is conceptually similar to the classic baseline trick in reinforcement learning, but the authors explicitly frame it around the specific problem of off-policy stability and tailor it to LLM training pipelines.

For LLMs, sequence-level reward signals are often sparse and vary widely in scale. Any uncontrolled numerical amplification can accumulate across long sequences into catastrophic gradient explosions. Score Centering controls the numerical range of the score term, providing additional stability headroom during training.

The score function is the theoretical cornerstone of policy gradient methods. In algorithms like REINFORCE, the policy gradient is expressed as the expectation of the product of rewards and the log-policy gradient, where the gradient of the log-policy with respect to parameters is called the score function. The core challenge is variance: even when the expected direction is correct, a single-sample gradient estimate can deviate wildly from the true value. The classic baseline variance-reduction technique — using a state-value function V(s) as a reference — exploits the property that subtracting any quantity independent of actions does not change the gradient expectation. Score Centering follows the same logic, but places the baseline at the level of the importance-sampling-corrected score term rather than simply adjusting the reward side. In off-policy settings, importance sampling ratios (the ratio of current-policy to behavior-policy probabilities) must also be incorporated; extreme ratio values can massively amplify variance. Centering at the score level therefore suppresses this amplification at the source.

Why Numerical Stability Matters So Much for LLM RL

The post's title explicitly highlights: "LLM RL is extremely sensitive to numeric issues." This is not an exaggeration.

Unlike traditional small-scale RL tasks, LLMs involve enormous parameter counts, long training sequences, and noisy reward models. Under these conditions, factors such as floating-point precision, gradient scale, and extreme importance sampling weights can all cause training to diverge suddenly at a single step. In practice, researchers often have to layer in gradient clipping, KL penalties, and importance weight clipping just to keep training from falling apart.

The value of Score Centering lies in providing a principled numerical intervention point, rather than relying purely on empirical hyperparameter tuning. For teams aiming to conduct LLM RL training in a scalable and reproducible manner, this has real practical significance.

Relationship to Mainstream Methods

Most mainstream LLM RL approaches (such as PPO and its variants) rely on on-policy or approximately on-policy data. While off-policy methods are appealing for their sample efficiency, stability has always been the key obstacle. Marek and Ryabinin's work can be seen as advancing the viability of off-policy training — if off-policy learning can be stabilized, data reuse efficiency in RL training would improve significantly, thereby reducing overall training costs.

It's worth noting that this article is based on a single Reddit discussion thread; the full experimental data, comparison baselines, and ablation analyses should be verified against the original paper. Based on publicly available information, the core contribution of this research lies in identifying and specifically addressing numerical stability in off-policy settings.

PPO (Proximal Policy Optimization) is currently the most widely used algorithm for LLM alignment training (e.g., RLHF). Its core design constrains policy updates to a neighborhood of the old policy by clipping importance sampling ratios, thereby approximating the stability guarantees of on-policy training. This design is fundamentally a conservative trade-off between sample efficiency and stability — each batch of data can typically only be reused for a few mini-epochs before clipping loses its effect. In contrast, purely off-policy methods (such as Q-learning with experience replay) can reuse historical data arbitrarily, offering theoretically higher sample efficiency, but distribution shift causes importance weights to grow dramatically, making gradient estimates unstable. Score Centering aims to suppress this numerical instability without sacrificing the flexibility of data reuse. If experimentally validated, it would provide important support for off-policy LLM training paradigms that go beyond PPO.

Implications for Practitioners

For engineers and researchers currently engaged in — or planning to undertake — LLM reinforcement learning training, this work offers at least two key takeaways:

• Prioritize numerical diagnostics: When RL training collapses inexplicably, the problem likely lies not in algorithmic logic but in numerical scale and variance control.
• Off-policy is not off-limits: With stabilization techniques like Score Centering emerging, off-policy RL has the potential to become a viable option that balances both efficiency and stability.

As organizations like Together.ai continue investing in training infrastructure and algorithmic research, LLM reinforcement learning is gradually moving from "getting it to run" toward a mature phase characterized by stability, efficiency, and reproducibility.