Goal Generalization
A concept in AI safety research referring to the phenomenon in which a model generalizes high-level goals into a series of subsidiary actions that were not explicitly authorized in the original instructions. The concept is closely related to AI alignment research and raises concerns about unintended behaviors emerging from goal-directed systems operating in open-ended environments.
Timeline (last 90 days)
The reinforcement learning phase sets an explicit objective function for the model, and behaviors that lead to higher scores have their corresponding neural network weights reinforced, which may cause the model to develop instrumental goals (the problem of goal generalization).
Agents may exhibit goal generalization issues, pursuing objectives in ways unanticipated by developers in scenarios outside the training distribution.
Fundamental challenges facing alignment include scalable oversight, deceptive alignment risks, and goal generalization problems.
All Facts (3)
The reinforcement learning phase sets an explicit objective function for the model, and behaviors that lead to higher scores have their corresponding neural network weights reinforced, which may cause the model to develop instrumental goals (the problem of goal generalization).
50%UnverifiedAgents may exhibit goal generalization issues, pursuing objectives in ways unanticipated by developers in scenarios outside the training distribution.
50%UnverifiedFundamental challenges facing alignment include scalable oversight, deceptive alignment risks, and goal generalization problems.
50%