When Scores and Preferences Conflict: Diagnosis and Adaptive Correction of DPO Constraint Drift

In alignment training based on human preferences, Direct Preference Optimization (DPO) has become one of the mainstream methods. But in real-world scenarios, pairwise preferences often coexist with pointwise criteria such as length, correctness, and safety. When the two are inconsistent—that is, when the preferred output actually scores worse on some criterion—standard DPO still treats it the same as consistent samples. This treatment triggers a systematic failure mode: constraint drift. A recent study formalizes this phenomenon and proposes a practical diagnosis and correction scheme.

Constraint Drift: How Conflicting Samples Drag Down the Alignment Objective

The study divides preference pairs into two categories: conflict pairs, where the preferred output scores worse on pointwise criteria; and matching pairs, where the preference direction agrees with the criterion scores. Under the assumption that parameter updates are sufficiently small, conflict pairs tend to increase constraint cost while matching pairs tend to repair it, and the net change depends on the balance between the two in the dataset.

This explains why some datasets exhibit degradation in safety or correctness during DPO training while others self-correct. Drift is not random noise but a predictable phenomenon determined by the structure of the data distribution.

DPO (Direct Preference Optimization) is a preference alignment method that requires no explicit reward model; it converts human preference data into a classification loss through a closed-form derivation and directly optimizes the policy model. Its core assumption is that preferred outputs should receive higher probability. However, the DPO loss only uses pairwise comparison signals and is unaware of pointwise criteria such as length, correctness, and safety. When the preference direction conflicts with these criteria, DPO still updates parameters according to the preference signal, potentially harming the model's performance on specific criteria. Understanding this mechanism helps grasp the root of constraint drift: DPO is not a design flaw but a structural mismatch between its objective function and the multi-criteria reality.

Closed-Form Diagnostic Metric ρ*_eq: Predicting Risk Before Training

One of the paper's core contributions is a closed-form diagnostic quantity ρ*_eq that can be computed directly from dataset statistics. This metric measures the relative strength of conflict pairs versus matching pairs and provides a clear threshold: when ρ*_eq indicates that the dataset lies in a conflict-dominated regime, constraint drift will accumulate significantly without intervention; otherwise, the dataset has self-correcting capacity.

The value of ρ*_eq lies in pre-alignment diagnosis—without running full training, data statistics alone suffice to determine whether intervention is needed. Experiments show that in conflict-dominated domains, drift grows monotonically with conflict exposure; and the intervention decisions given by ρ*_eq are consistent with the observed drift in every domain, including one domain it judged to be self-correcting. This means the diagnostic metric is not only correlational but also has an actionable decision boundary.

The small-update assumption is a theoretical simplification commonly used when analyzing DPO training dynamics: it assumes that each parameter update step is small enough that loss changes can be approximated by a first-order Taylor expansion. Under this assumption, the alignment contributions of conflict pairs and matching pairs can be linearly superposed, allowing the net drift direction to be inferred directly from data statistics. This assumption approximately holds under standard training configurations (moderate learning rate, limited epochs), but when the learning rate is too large or training epochs are too many, parameter updates are no longer small, higher-order terms cannot be ignored, the linear approximation fails, and the accuracy of the diagnostic metric ρ*_eq declines accordingly. Thus, this assumption is both the cornerstone of the theoretical derivation and the boundary of the method's applicability.

CoDPO: Downweighting Conflict Pairs by Diagnostic Strength and Adaptively Adjusting the Margin

Based on ρ*_eq, the authors propose Conflict-aware DPO (CoDPO). Its core mechanism consists of two parts:

1. Conflict pair downweighting: apply downweighting to conflict pairs, with the weight strength derived directly from ρ*_eq rather than relying on manual tuning. This eliminates the burden of repeatedly searching hyperparameters for different datasets.
2. Adaptive margin shift: dynamically adjust the preference margin during training to offset residual drift. This term compensates for the time-accumulation effect that static downweighting cannot fully eliminate.

Across four main comparison domains, CoDPO consistently reduces constraint drift while largely preserving the original DPO's preference learning capability. Compared with hard filtering (directly discarding conflict pairs) and recent constrained optimization and multi-objective optimization baselines, CoDPO achieves a better trade-off between drift and preference. Although hard filtering is simple, it loses preference signals in the data; CoDPO's downweighting strategy retains this information and only weakens its adversarial influence.

The preference margin in DPO refers to the target gap that the log-probability difference between preferred and non-preferred outputs should reach. Standard DPO implies a fixed margin, pushing the probability of preferred outputs to be significantly higher than that of non-preferred outputs. When conflict pairs exist in the data, a fixed margin may force the model to overfit conflict signals, exacerbating constraint drift. Adaptive margin shifting dynamically adjusts this target gap during training, lowering the margin requirement for conflict samples and thereby mitigating their adversarial influence. This mechanism is complementary to conflict pair downweighting: downweighting reduces the gradient contribution of conflict samples, while margin shifting adjusts the strictness of the optimization objective; together they suppress drift accumulation.

Method Positioning from a Multi-Source Cross Perspective

According to an independent analysis of the paper, CoDPO's contribution is not in proposing a new family of loss functions but in binding data diagnosis to intervention strength. This line of thinking echoes recent discussions on how preference data quality affects alignment stability: a dataset is not neutral training fuel; its internal consistency structure determines the optimization trajectory.

The paper's derivation of ρ*_eq relies on the small-update assumption; under large learning rates or long training cycles, the diagnostic metric's accuracy may decline—this is an explicit boundary of the method's applicability. The training configuration used in the authors' experiments does not exceed this assumption's scope, but attention is needed in real-world deployment.

Practical Implications: Diagnose First, Then Train

For researchers and engineers working on alignment training, this work provides a low-cost pre-check step: before starting DPO, compute the dataset's conflict pair ratio and ρ*_eq to determine whether intervention is needed. If ρ*_eq indicates conflict dominance, CoDPO's downweighting and margin shifting can serve as a plug-and-play correction module.

More generally, the study suggests that criterion conflicts in preference learning are not dirty data to be avoided but distribution features that need to be explicitly modeled. Incorporating conflict structure into the optimization objective rather than simply filtering it out may be a more sustainable path to constrained alignment.

One-line Summary

Research reveals that conflicting samples in DPO cause constraint drift and proposes the closed-form diagnostic metric ρ*_eq and the CoDPO correction method.

Paragraph Summary

This study points out that in Direct Preference Optimization (DPO), when preference pairs are inconsistent with pointwise criteria such as length, correctness, and safety, systematic constraint drift occurs. The authors divide samples into conflict pairs and matching pairs, find that net drift depends on the balance between them, and propose a closed-form diagnostic quantity ρ*_eq computable directly from data statistics, enabling pre-training risk prediction. Based on this metric, they propose CoDPO, which downweights conflict pairs according to diagnostic strength and adds adaptive margin shifting, consistently reducing drift across four domains while largely preserving preference learning capability, outperforming hard filtering and constrained/multi-objective baselines. The method relies on the small-update assumption, and diagnostic accuracy may decline under large learning rates or long training cycles.