Direct Preference Optimization
An algorithm that directly optimizes a policy using preference data, simplifying the traditional reinforcement learning training pipeline and requiring no explicit reward model.
Core Facts
Timeline (last 90 days)
Direct preference optimization (DPO) shows that under a specific reward structure, the RL objective can be equivalently transformed into likelihood maximization over a certain target distribution, thereby bypassing an explicit reward model and policy gradients.
All Facts (2)
DPO (Direct Preference Optimization) is an alignment method proposed in 2023 that can skip reward model training and directly optimize the policy model from human preference data.
90%UnverifiedDirect preference optimization (DPO) shows that under a specific reward structure, the RL objective can be equivalently transformed into likelihood maximization over a certain target distribution, thereby bypassing an explicit reward model and policy gradients.
50%