WAS-PO
A policy optimization method for multi-reward reinforcement learning. It applies ZCA whitening to advantages before advantage-weighted summation, aligning weight semantics with the optimization objective and resolving weight distortion caused by reward correlation.
Timeline (last 90 days)
WAS-PO (Whitened Advantage Summation Policy Optimization) is a multi-reward reinforcement learning method that, after normalizing advantages and before the weighted sum, first whitens the advantages with Σ^{-1/2}, so that the weights act on the decorrelated components.
The theoretical analysis of WAS-PO shows that even if all reward weights are positive, the effective weight of the whitened aggregate objective on certain original rewards may still be negative.
The paper gives an exact two-reward condition under which the reward with the lighter weight is inverted in the equivalent objective.
The paper gives a loose upper bound on the error introduced by estimating the reward correlation matrix Σ during training.
WAS-PO replaces GDPO's per-prompt normalization with batch-level normalization, but this change and the whitening operation were not separated in the experiments.
In experiments on a tool-calling task using Qwen2.5-1.5B-Instruct, under the three-reward setting WAS-PO achieved a higher average training reward across all seven random seeds, by 3.8%.
Under the four-reward setting, WAS-PO achieved a higher average training reward across all three random seeds, by 3.9%.
Under three rewards, WAS-PO's held-out accuracy is slightly lower than GDPO's (not significant); under four rewards it is absolutely lower by 0.03 (only two seeds were evaluated).
Under the two-reward setting, WAS-PO's training reward is lower than GDPO's.
In a math task setting where two rewards are nearly identical by construction, WAS-PO suppressed the reward with the lighter weight, as predicted by the sign condition.
All Facts (10)
WAS-PO (Whitened Advantage Summation Policy Optimization) is a multi-reward reinforcement learning method that, after normalizing advantages and before the weighted sum, first whitens the advantages with Σ^{-1/2}, so that the weights act on the decorrelated components.
50%UnverifiedThe theoretical analysis of WAS-PO shows that even if all reward weights are positive, the effective weight of the whitened aggregate objective on certain original rewards may still be negative.
50%UnverifiedThe paper gives an exact two-reward condition under which the reward with the lighter weight is inverted in the equivalent objective.
50%UnverifiedThe paper gives a loose upper bound on the error introduced by estimating the reward correlation matrix Σ during training.
50%UnverifiedWAS-PO replaces GDPO's per-prompt normalization with batch-level normalization, but this change and the whitening operation were not separated in the experiments.
50%UnverifiedIn experiments on a tool-calling task using Qwen2.5-1.5B-Instruct, under the three-reward setting WAS-PO achieved a higher average training reward across all seven random seeds, by 3.8%.
50%UnverifiedUnder the four-reward setting, WAS-PO achieved a higher average training reward across all three random seeds, by 3.9%.
50%UnverifiedUnder three rewards, WAS-PO's held-out accuracy is slightly lower than GDPO's (not significant); under four rewards it is absolutely lower by 0.03 (only two seeds were evaluated).
50%UnverifiedUnder the two-reward setting, WAS-PO's training reward is lower than GDPO's.
50%UnverifiedIn a math task setting where two rewards are nearly identical by construction, WAS-PO suppressed the reward with the lighter weight, as predicted by the sign condition.
50%