Preference Optimization
A class of alignment methods that dispense with an explicit reward model. Represented by DPO, these methods directly optimize a policy model using paired preference data.
Timeline (last 90 days)
Post-distillation preference optimization can improve the student's harm-avoidance ability, but cannot eliminate the differences brought by the source teacher.
Through stage-level analysis, the paper found that preference optimization is one of the primary contributors to the Narrative Captivity phenomenon.
Optimization performed to align with user preferences may cause models to lean toward agreeing with and empathizing with one party in the conversation, at the cost of neutrality and independent judgment.
All Facts (3)
Post-distillation preference optimization can improve the student's harm-avoidance ability, but cannot eliminate the differences brought by the source teacher.
50%UnverifiedThrough stage-level analysis, the paper found that preference optimization is one of the primary contributors to the Narrative Captivity phenomenon.
50%UnverifiedOptimization performed to align with user preferences may cause models to lean toward agreeing with and empathizing with one party in the conversation, at the cost of neutrality and independent judgment.
50%