Score Centering
A numerical technique for stabilizing off-policy reinforcement learning training. By mean-centering the score function, it reduces variance in policy gradient estimates and thereby mitigates gradient instability caused by extreme importance-sampling weights that arise in off-policy settings.
Timeline (last 90 days)
The paper 'Score Centering Stabilizes Off-policy Reinforcement Learning' was authored by researchers Marek and Ryabinin from Together.ai.
Score Centering reduces the variance of gradient estimates by applying mean subtraction to the score function.
Score Centering places the effect of the baseline on the off-policy-corrected score term, rather than treating only the reward side.
If Score Centering is experimentally validated as effective, it will provide support for off-policy LLM training approaches that go beyond the PPO paradigm.
All Facts (4)
The paper 'Score Centering Stabilizes Off-policy Reinforcement Learning' was authored by researchers Marek and Ryabinin from Together.ai.
50%UnverifiedScore Centering reduces the variance of gradient estimates by applying mean subtraction to the score function.
50%UnverifiedScore Centering places the effect of the baseline on the off-policy-corrected score term, rather than treating only the reward side.
50%UnverifiedIf Score Centering is experimentally validated as effective, it will provide support for off-policy LLM training approaches that go beyond the PPO paradigm.
50%