Alignment Tax
A phenomenon observed in large language models after safety fine-tuning, whereby the model produces false refusals or over-refuses legitimate requests in sensitive but lawful professional contexts. This capability degradation introduced by safety alignment represents a central tension in the field of large model safety governance.
Timeline (last 90 days)
The 'Alignment Tax' refers to performance degradation in certain capability dimensions that occurs after a model undergoes alignment training such as RLHF and safety fine-tuning.
Companies such as OpenAI and Anthropic have acknowledged the existence of alignment trade-offs in their technical reports.
The "Alignment Tax" refers to the performance cost that must be paid to ensure AI systems are safely aligned; an excessively high alignment tax incentivizes developers and users to circumvent safety measures
The Alignment Tax problem refers to the possibility that self-improvement may unintentionally undermine safety and alignment while improving task performance.
All Facts (4)
The 'Alignment Tax' refers to performance degradation in certain capability dimensions that occurs after a model undergoes alignment training such as RLHF and safety fine-tuning.
50%UnverifiedCompanies such as OpenAI and Anthropic have acknowledged the existence of alignment trade-offs in their technical reports.
50%UnverifiedThe "Alignment Tax" refers to the performance cost that must be paid to ensure AI systems are safely aligned; an excessively high alignment tax incentivizes developers and users to circumvent safety measures
50%UnverifiedThe Alignment Tax problem refers to the possibility that self-improvement may unintentionally undermine safety and alignment while improving task performance.
50%