GAP-DPO
An iterative algorithm for personalized preference optimization that addresses distribution shift and selection bias in DPO under personalized settings. It employs utility-aware geometric alignment for preference pair selection and a round-by-round regeneration mechanism.
Timeline (last 90 days)
The paper reformulates personalized preference learning as a geometrically aligned optimization problem.
The paper proposes GAP-DPO (Geometry-Aligned Preference DPO), an iterative algorithm.
In each iteration, GAP-DPO selects preference pairs based on the direction of the utility gradient.
GAP-DPO mitigates distribution shift caused by off-policy sampling through an epoch-wise regeneration mechanism that regenerates samples at each epoch.
On personalized text generation benchmarks, GAP-DPO achieves consistent improvements over standard DPO variants across three dimensions: style fidelity, preference alignment, and generation quality.
The study proposes gradient alignment as a unified principle for personalized preference optimization, arguing that preference pair selection is an intrinsic part of the optimization geometry rather than a data preprocessing step.
All Facts (6)
The paper reformulates personalized preference learning as a geometrically aligned optimization problem.
50%UnverifiedIn each iteration, GAP-DPO selects preference pairs based on the direction of the utility gradient.
50%UnverifiedGAP-DPO mitigates distribution shift caused by off-policy sampling through an epoch-wise regeneration mechanism that regenerates samples at each epoch.
50%UnverifiedOn personalized text generation benchmarks, GAP-DPO achieves consistent improvements over standard DPO variants across three dimensions: style fidelity, preference alignment, and generation quality.
50%UnverifiedThe study proposes gradient alignment as a unified principle for personalized preference optimization, arguing that preference pair selection is an intrinsic part of the optimization geometry rather than a data preprocessing step.
50%UnverifiedThe paper proposes GAP-DPO (Geometry-Aligned Preference DPO), an iterative algorithm.
50%