Direct preference optimization (DPO) shows that under a specific reward structure, the RL objective can be equivalently transformed into likelihood maximization over a certain target distribution, thereby bypassing an explicit reward model and policy gradients.