Reward Overoptimization

A phenomenon in RLHF training where a language model excessively optimizes the reward model, causing reward scores to inflate without a corresponding improvement—or with an actual decline—in the quality of real outputs. It is a specific manifestation of the reward hacking problem in large language models.