PORT
PORT is a token-level reweighting framework for preference optimization that uses the empirical cumulative distribution function (CDF) of token log-probabilities as a forward proxy for a gradient-norm penalty, achieving fine-grained robust alignment under noisy preference data.
Timeline (last 90 days)
The researchers propose PORT (Preference Optimization via Robust Token-level Reweighting), a framework for fine-grained robust alignment under noisy preferences.
PORT's core insight is that the gradient bias caused by noisy preference labels can be decomposed onto each token.
PORT uses the empirical cumulative distribution function (CDF) of token log-probabilities to perform token-level reweighting.
PORT's CDF weighting design constitutes an efficient forward-pass surrogate for gradient-norm penalization, requiring no additional backpropagation computation.
PORT reduces the gradient bias caused by noisy preference labels by minimizing an upper bound on the bias term.
Extensive experiments under various noise settings show that PORT consistently improves robustness and outperforms existing sequence-level baselines.
PORT's code is open-sourced on GitHub (osu-srml/PORT).
PORT's forward-pass-only nature means it does not significantly increase training cost, giving it good deployment potential for industrial-scale preference optimization tasks.
All Facts (8)
The researchers propose PORT (Preference Optimization via Robust Token-level Reweighting), a framework for fine-grained robust alignment under noisy preferences.
50%UnverifiedPORT's core insight is that the gradient bias caused by noisy preference labels can be decomposed onto each token.
50%UnverifiedPORT uses the empirical cumulative distribution function (CDF) of token log-probabilities to perform token-level reweighting.
50%UnverifiedPORT's CDF weighting design constitutes an efficient forward-pass surrogate for gradient-norm penalization, requiring no additional backpropagation computation.
50%UnverifiedPORT reduces the gradient bias caused by noisy preference labels by minimizing an upper bound on the bias term.
50%UnverifiedExtensive experiments under various noise settings show that PORT consistently improves robustness and outperforms existing sequence-level baselines.
50%UnverifiedPORT's code is open-sourced on GitHub (osu-srml/PORT).
50%UnverifiedPORT's forward-pass-only nature means it does not significantly increase training cost, giving it good deployment potential for industrial-scale preference optimization tasks.
50%