Selective Tie Learning: A New Solution to Spurious Correlations in Preference Optimization

Preference optimization (such as reward modeling in RLHF) is prone to learning spurious correlations, where the model relies on features irrelevant to the task when making predictions. One mitigation approach is to inject "ties" (pairs of responses that are equally preferred) into the preference data, but different ties vary greatly in their effectiveness at reducing spurious correlations. A recent study proposes "Selective Tie Learning," which uses theoretical analysis to筛选 out the most effective ties, thereby achieving debiasing effects comparable to or even better than uniform selection with fewer ties.

Where Do Spurious Correlations Come From?

Preference optimization models typically learn human preferences from pairwise comparisons. If spurious associations exist in the training data (e.g., response length correlated with preference labels), the model may rely on these features rather than true quality. Such spurious correlations lead to performance degradation on out-of-distribution data.

Injecting tie data (i.e., two responses labeled as equally preferred) is thought to weaken spurious correlations because ties provide no discriminative signal, forcing the model to ignore irrelevant features. However, ties are not equivalent: some ties can more effectively correct the model's erroneous dependencies, while others may have little effect.

Spurious correlation in preference optimization specifically refers to non-causal associations between input features and preference labels learned by the model. For example, if longer responses happen to be selected more often in the training data, the model may mistake "length" for a signal of "quality." This association may improve fitting within the training distribution, but on out-of-distribution (OOD) data, when length and quality decouple, model performance drops significantly. The root cause of spurious correlations lies in the observational bias of preference data: human annotators' choices are influenced by many factors that are not necessarily aligned with the task objective. Therefore, the goal of debiasing is to make the model rely on features truly relevant to the task rather than these incidentally correlated surface features.

Theoretical Analysis: The Impact of Ties Can Be Quantified

For log-linear policies, the researchers derived the impact of each tie on spurious correlation dependence. They found that within subsets of the same size, selecting the tie with the largest predicted reduction is "first-order optimal." However, computing this impact requires knowledge of the spurious features and the curvature of the training objective, which is often unavailable or computationally expensive in practice.

Therefore, the researchers explored using the implicit reward margin magnitude of ties as a proxy for impact prediction and characterized the conditions under which it serves as a useful proxy. Theory shows that under certain conditions, reward margin magnitude is positively correlated with impact size, providing a convenient signal for practical selection.

Implicit reward margin refers to the difference in reward scores implicitly assigned by a preference optimization model (such as the Bradley-Terry model) to a pair of responses. Specifically, if the model's implicit reward for response A is r(A) and for response B is r(B), then the margin magnitude is |r(A)-r(B)|. In tie data, two responses are labeled as equally preferred, and ideally the model should assign them similar rewards, i.e., the margin magnitude should approach zero. However, before training, the model may assign a large margin magnitude to this tie pair, reflecting the model's degree of discrimination between the two responses. Theoretical analysis shows that ties with larger margin magnitudes often correspond to the model's currently more "confident" erroneous distinctions, so correcting them may be more effective at reducing spurious correlation dependence. This proxy metric requires no knowledge of the true spurious features and can be computed solely from model outputs, providing convenience for practical selection.

From Theory to Practice: Validation on Neural Models

To verify the transferability of the theory to neural network models, the researchers employed a counterfactual proxy to estimate the impact on spurious correlation dependence and used the Generalized Gauss-Newton approximation to estimate the curvature of language models. This allowed them to approximately compute the impact of each tie and select accordingly.

Experiments covered log-linear policies, neural network models, and large language models. Results show that compared with uniform tie selection, the proposed selection rule consistently achieves comparable effectiveness in reducing spurious correlation dependence but requires fewer ties. This means that in scenarios with high annotation costs, Selective Tie Learning can significantly reduce data requirements.

Generalized Gauss-Newton (GGN) is a method for approximating the curvature (Hessian matrix) of neural network loss functions. In preference optimization, curvature reflects the sensitivity of model parameters to the training objective and is a key component in computing the impact of each tie. Exactly computing the Hessian matrix is extremely computationally expensive for large language models (with parameter counts reaching billions), so the researchers adopted the GGN approximation, using the outer product of Jacobian matrices to approximate the Hessian, thereby estimating curvature at acceptable computational cost. GGN significantly reduces computational complexity while maintaining a certain level of accuracy, making impact estimation feasible on LLMs.

Multi-Source Cross-Validation and Consensus

The study integrates theoretical derivations with empirical experiments, and the conclusions are consistent across multiple model architectures, enhancing credibility. Although effect sizes may vary across models, the core conclusion that "selective is better than uniform" is supported by multiple sources.

Implications for Preference Optimization Practice

The significance of this work lies in providing a data-efficient debiasing strategy. In actual RLHF pipelines, collecting tie data is cheaper than pairwise preference data, but there has been a lack of theoretical guidance on how to select ties. Selective Tie Learning provides an actionable criterion: prioritize ties with large implicit reward margin magnitudes.

Of course, this method relies on curvature approximation and may introduce additional computational overhead on extremely large-scale models. Future work could explore more efficient approximation methods or extend the selection strategy to other preference optimization algorithms (such as DPO).

In summary, Selective Tie Learning provides a new theoretical perspective and practical tool for mitigating spurious correlations in preference optimization, promising to advance the development of more robust and data-efficient preference learning systems.

Direct Preference Optimization (DPO) is a preference optimization algorithm that requires no explicit reward model; it merges reward modeling and policy optimization into a simple classification loss through reparameterization. DPO directly updates language models using preference pair data, avoiding the complex reinforcement learning pipeline in RLHF, and is therefore popular in practical applications. However, DPO is also prone to learning spurious correlations, and its loss function handles tie data differently from reward-model-based methods. The core idea of Selective Tie Learning—prioritizing ties with large implicit reward margin magnitudes—can in principle be transferred to DPO, but impact estimation needs to be re-derived for its loss surface. Future work could explore designing efficient selection criteria within the DPO framework to further expand the applicability of this method.