[KongchangAI]
· 2 min read· 1,267 words

PORT: Token-Level Reweighting Improves Preference Optimization Robustness

PORT: Token-Level Reweighting Improves Preference Optimization Robustness

PORT uses token-level reweighting instead of sequence-level processing to achieve finer-grained robust alignment on noisy preference data.

The article introduces PORT, a robust alignment method for noisy preference data. Existing robust preference optimization operates at the sequence level, treating entire responses as uniformly correct or incorrect. PORT's key insight is that gradient bias from noisy labels can be decomposed to each token, so it uses the empirical CDF of token log-probabilities for token-level reweighting as an efficient forward-pass proxy for gradient norm penalization without extra backprop. Experiments show PORT consistently outperforms sequence-level baselines with controllable overhead, and the code is open-sourced.

Title

PORT: Token-Level Reweighting Improves Preference Optimization Robustness

Meta Description

PORT proposes a token-level reweighting framework that leverages the CDF distribution of token log-probabilities to suppress suspicious tokens in noisy preference labels, reducing gradient bias without additional backpropagation and outperforming sequence-level baselines under various noise settings.

Article Content

Preference optimization is a core technique for aligning large language models with human values, but its effectiveness depends heavily on the quality of preference annotations. In practice, annotation errors and semantic ambiguity fill preference data with noise. Existing robust preference optimization methods mostly operate at the sequence level, implicitly treating an entire response as uniformly correct or incorrect—yet a rejected response may contain valuable reasoning steps, and a preferred response may harbor subtle errors. To address this contradiction, researchers propose PORT (Preference Optimization via Robust Token-level Reweighting), a framework for fine-grained robust alignment under noisy preferences.

PORT paper source screenshot

Limitations of Sequence-Level Methods and Token-Level Insights

Traditional robust preference optimization treats each response as an indivisible whole, with the loss function computed at the sequence level. This approach ignores a key fact: the correctness of a response is not uniformly distributed. A response labeled as "rejected" may be mostly reasonable, erring only at a critical step; a response labeled as "preferred" may contain factual errors in the details. Sequence-level methods cannot distinguish these cases, causing the model during training to either over-penalize useful content or inadvertently reinforce errors.

The core insight of PORT is that the gradient bias introduced by noisy preference labels can be decomposed down to each token. Rather than crudely correcting at the sequence level, it is better to identify and suppress those "suspicious" tokens at the token level, allowing the optimization process to focus more precisely on genuinely trustworthy signals.

Preference Optimization is a class of methods that directly train language models using human preference comparison data, represented by reward modeling in RLHF and DPO (Direct Preference Optimization). In standard preference data, each sample consists of a prompt, a preferred response, and a rejected response, and the model aligns with human intent by learning to increase the relative probability of the preferred response. However, this learning signal rests on the assumption that labels are correct: once the preference labels themselves are wrong, the model is guided to decrease the probability of correct responses or increase the probability of incorrect ones, producing so-called gradient bias. Sequence-level robust methods typically mitigate this problem through sample reweighting or loss truncation, but they can only judge trustworthiness at the granularity of "the entire response" and cannot handle cases where a response is partially correct and partially wrong. PORT's token-level perspective is a direct response to this granularity deficiency.

How PORT Works: CDF-Based Token Reweighting

PORT uses the empirical cumulative distribution function (CDF) of token log-probabilities to perform token-level reweighting. Specifically, for each token, it computes the position of its log-probability within the overall distribution and assigns a weight accordingly. The cleverness of this design lies in that it constitutes an efficient forward-pass proxy for gradient norm penalization—selectively suppressing noise-contaminated tokens without additional backpropagation computation.

Theoretically, PORT reduces the gradient bias caused by noisy preference labels by minimizing an upper bound on the bias term. This forward-pass-only proxy design allows PORT to achieve finer-grained robustness than sequence-level methods while maintaining computational efficiency.

Empirical CDF is a statistical tool for describing the distributional position of a set of observations: given a sample set, the empirical CDF of a value equals the proportion of samples in the set that are less than or equal to that value. PORT treats each token's log-probability as an observation and computes its quantile position within the distribution of all token log-probabilities. Intuitively, tokens with abnormally low log-probabilities (i.e., the model is very unconfident) often correspond to noise or erroneous content, and their CDF values fall at the low end of the distribution, thus receiving smaller weights; tokens in the middle of the distribution receive relatively normal weights. This ranking-based rather than absolute-threshold-based weighting approach adapts well to different models and data distributions, and avoids the fragility of manually setting probability thresholds.

Gradient norm penalization is a common robust training technique: when a sample or token produces an excessively large gradient, it indicates that the model is being strongly pushed by that signal, and if that signal comes from a noisy label, it will cause harmful updates. The classic approach computes the gradient norm and penalizes it, but this requires additional backpropagation and is costly. PORT's core trick is to use the CDF value of token log-probabilities as a forward proxy for the gradient norm: tokens with lower log-probabilities typically correspond to larger gradient contributions, so down-weighting these tokens based on the CDF is equivalent to approximately implementing gradient norm penalization without extra backward computation. This "forward-pass-only" design is an important reason why PORT is engineering-feasible.

Experimental Validation: Stable Improvements Under Diverse Noise Settings

Researchers conducted extensive experiments under various noise settings, and the results show that PORT consistently improves robustness and outperforms existing sequence-level baseline methods. This means that whether in scenarios with high annotation error rates or significant semantic ambiguity, the token-level reweighting strategy can more effectively resist noise interference.

The code is open-sourced on GitHub (osu-srml/PORT), providing a reproducible foundation for future research.

Noisy preference settings are typically divided into two categories: label-flipping noise (swapping the labels of preferred and rejected responses) and semantic ambiguity noise (where the two responses are of similar quality and the annotation itself involves reasonable disagreement). The former simulates random annotator errors, while the latter simulates situations in real scenarios where superiority is difficult to determine. Robust preference optimization research generally evaluates methods under different proportions of these two noise types, observing whether the model's preference accuracy on a clean test set degrades significantly as training noise increases. PORT reports stable improvements across multiple noise types and noise rates, indicating that token-level reweighting is not effective only against a single noise pattern but is a general mitigation of gradient bias.

Implications for Preference Optimization Practice

The value of PORT lies not only in proposing a new method but also in revealing a long-neglected dimension in preference optimization: token-level non-uniformity. When we train alignment models on noisy data, coarse-grained sequence-level processing may mask a great deal of useful information. PORT's token-level reweighting approach provides a new direction for building more robust and fine-grained alignment pipelines.

For practical applications, PORT's forward-pass-only property means it does not significantly increase training costs, giving it good potential for deployment in industrial-scale preference optimization tasks. As the scale of preference data grows and noise problems become the norm, the value of such fine-grained robust methods will become even more prominent.

One-line Summary

PORT replaces sequence-level processing with token-level reweighting to achieve finer-grained robust alignment under noisy preference data.

Paragraph Summary

This article introduces PORT, a robust alignment method for noisy preference data. Existing robust preference optimization mostly operates at the sequence level, treating an entire response as uniformly correct or incorrect, ignoring potentially valuable content in rejected responses and subtle errors in preferred responses. PORT's core insight is that the gradient bias from noisy labels can be decomposed down to each token, so it uses the empirical cumulative distribution function of token log-probabilities for token-level reweighting as an efficient forward-pass proxy for gradient norm penalization, without additional backpropagation. Theoretical analysis shows the method reduces gradient bias by minimizing an upper bound on the bias term. Experiments under various noise settings show PORT consistently outperforms sequence-level baselines with controllable computational overhead, and the code is open-sourced.

#preference optimization#token-level reweighting#robust alignment#noisy labels#gradient bias#large language models
Share:

Related articles