[KongchangAI]
ConceptPreference Optimization via Robust Token-level Reweighting

PORT

PORT is a token-level reweighting framework for preference optimization that uses the empirical cumulative distribution function (CDF) of token log-probabilities as a forward proxy for a gradient-norm penalty, achieving fine-grained robust alignment under noisy preference data.

Timeline (last 90 days)

Nov 10

The researchers propose PORT (Preference Optimization via Robust Token-level Reweighting), a framework for fine-grained robust alignment under noisy preferences.

Unverified50%
Nov 10

PORT's core insight is that the gradient bias caused by noisy preference labels can be decomposed onto each token.

Unverified50%
Nov 10

PORT uses the empirical cumulative distribution function (CDF) of token log-probabilities to perform token-level reweighting.

Unverified50%
Nov 10

PORT's CDF weighting design constitutes an efficient forward-pass surrogate for gradient-norm penalization, requiring no additional backpropagation computation.

Unverified50%
Nov 10

PORT reduces the gradient bias caused by noisy preference labels by minimizing an upper bound on the bias term.

Unverified50%
Nov 10

Extensive experiments under various noise settings show that PORT consistently improves robustness and outperforms existing sequence-level baselines.

Unverified50%
Nov 10

PORT's code is open-sourced on GitHub (osu-srml/PORT).

Unverified50%
Nov 10

PORT's forward-pass-only nature means it does not significantly increase training cost, giving it good deployment potential for industrial-scale preference optimization tasks.

Unverified50%

All Facts (8)

Source Articles