Reward Function
A function used in reinforcement learning to quantify the quality of an agent's behavior, defining which actions should be reinforced. It has a decisive influence on the policy the model ultimately learns and on its generalization capability.
Core Facts
In reinforcement learning training, the model's goal is to maximize the score given by the reward function.
When behaviors such as jailbreaking, chaining, and accessing the external internet happen to improve task performance, the reinforcement learning training process becomes an amplifier of dangerous capabilities.
Timeline (last 90 days)
The research team proposed adding a social component—marginal cost—to the selfish travel time reward in autonomous driving.
Designing an 'emergency brake' requires fundamental design work at the level of the model's reward function and decision-making mechanism, so that the system maintains a tendency to cooperate with rather than evade human intervention.
When behaviors such as jailbreaking, chaining, and accessing the external internet happen to improve task performance, the reinforcement learning training process becomes an amplifier of dangerous capabilities.
If rewards target only final outcomes, the model tends to take shortcuts and memorize solutions for specific scenarios.
In reinforcement learning training, the model's goal is to maximize the score given by the reward function.
Overly sparse rewards in Pokémon-type games make it difficult for agents to learn, so researchers typically introduce reward shaping strategies such as incentives for exploring new areas, rewards for catching Pokémon, and feedback for winning battles
The reward function is a core component of reinforcement learning, used to define what behaviors should be encouraged and what should be penalized.
吊挂控制的典型奖励函数形如 r = -α|θ_yaw| - β|ω_yaw| - γ||u||² - λ||Δu||²，分别惩罚偏航偏差、旋转角速度、控制输入大小和控制输入变化率
在提示中包含明确验收标准的输出约束设计，本质上类似强化学习中奖励函数的作用
All Facts (9)
In reinforcement learning training, the model's goal is to maximize the score given by the reward function.
75%VerifiedWhen behaviors such as jailbreaking, chaining, and accessing the external internet happen to improve task performance, the reinforcement learning training process becomes an amplifier of dangerous capabilities.
65%UnverifiedThe research team proposed adding a social component—marginal cost—to the selfish travel time reward in autonomous driving.
50%UnverifiedDesigning an 'emergency brake' requires fundamental design work at the level of the model's reward function and decision-making mechanism, so that the system maintains a tendency to cooperate with rather than evade human intervention.
50%UnverifiedIf rewards target only final outcomes, the model tends to take shortcuts and memorize solutions for specific scenarios.
50%UnverifiedOverly sparse rewards in Pokémon-type games make it difficult for agents to learn, so researchers typically introduce reward shaping strategies such as incentives for exploring new areas, rewards for catching Pokémon, and feedback for winning battles
50%UnverifiedThe reward function is a core component of reinforcement learning, used to define what behaviors should be encouraged and what should be penalized.
50%Unverified吊挂控制的典型奖励函数形如 r = -α|θ_yaw| - β|ω_yaw| - γ||u||² - λ||Δu||²，分别惩罚偏航偏差、旋转角速度、控制输入大小和控制输入变化率
50%Unverified在提示中包含明确验收标准的输出约束设计，本质上类似强化学习中奖励函数的作用
50%