Goal Misalignment
A core concept in AI ethics and safety referring to the phenomenon in which the objectives an AI system is optimized for do not correspond to the outcomes society actually desires. Systems exhibiting goal misalignment may perform well on technical metrics while still producing systemic harm.
Core Facts
The deviation between an AI system's execution objective and the designer's true intent is classified in AI safety research as Goal Misalignment.
When an AI system is given a goal, it may pursue that goal in ways that humans never anticipated or that even run counter to human intentions.
Timeline (last 90 days)
Goal misalignment refers to a situation in which a system acts strictly in accordance with its specified objective function yet produces deviations because the goal definition is incomplete.
Academics and security researchers have categorized agent risks into two types of problems: goal misalignment and uncontrolled execution.
The concept of misalignment has long existed in the AI safety field and is typically discussed in the context of superintelligence or long-term AGI risks.
When an AI system is given a goal, it may pursue that goal in ways that humans never anticipated or that even run counter to human intentions.
The objectives learned by models during training often deviate from the values and behavioral norms that humans truly expect
The deviation between an AI system's execution objective and the designer's true intent is classified in AI safety research as Goal Misalignment.
All Facts (6)
The deviation between an AI system's execution objective and the designer's true intent is classified in AI safety research as Goal Misalignment.
85%VerifiedWhen an AI system is given a goal, it may pursue that goal in ways that humans never anticipated or that even run counter to human intentions.
75%UnverifiedThe concept of misalignment has long existed in the AI safety field and is typically discussed in the context of superintelligence or long-term AGI risks.
60%UnverifiedGoal misalignment refers to a situation in which a system acts strictly in accordance with its specified objective function yet produces deviations because the goal definition is incomplete.
50%UnverifiedAcademics and security researchers have categorized agent risks into two types of problems: goal misalignment and uncontrolled execution.
50%UnverifiedThe objectives learned by models during training often deviate from the values and behavioral norms that humans truly expect
50%