Reward Shaping
A method in reinforcement learning for designing and adjusting reward functions to guide an agent toward desired behaviors. It is one of the core challenges in RL debugging and training.
Core Facts
Timeline (last 90 days)
Training starts with sparse rewards, and then iterates through reward shaping via an event frequency table; in the event frequency table, movement penalties account for 48.3% of all events, exploring tiles accounts for 21.7%, and attacking monsters accounts for 14.5%.
Traditional approaches include introducing intermediate rewards, reward shaping, or hindsight relabeling (such as HER), but these methods either rely on manual design or are difficult to adapt to the highly heterogeneous scenarios in MLE where code execution feedback is complex.
Reward shaping refers to designing additional auxiliary reward terms beyond the sparse original reward to guide the policy, which can accelerate convergence, but if the auxiliary rewards are inconsistent with the final objective, the policy may game the score without truly improving the win rate; in practice a smaller weight is often multiplied in and decayed in the later stages of training.
Embedding the social marginal cost into each agent's individual reward is a reward shaping strategy that requires no modification to the underlying MARL algorithm architecture.
If the reward shaping signal is too weak, the AI will take shortcuts; if it is too strong or poorly designed, it may introduce new exploits or constrain the exploration space.
Reward shaping refers to adding extra, denser guiding reward signals on top of the original sparse reward to help agents learn faster and more correctly.
With the help of reward shaping and league play, the author ultimately trained agents to exhibit relatively interesting combat behavior.
In sparse-reward scenarios, agents often need techniques such as curriculum learning, reward shaping, or intrinsic motivation-based exploration to converge effectively.
Overly sparse rewards in Pokémon-type games make it difficult for agents to learn, so researchers typically introduce reward shaping strategies such as incentives for exploring new areas, rewards for catching Pokémon, and feedback for winning battles
The reward shaping method was first systematized by Andrew Ng and others in the 1999 paper "Policy Invariance Under Reward Transformations".
1 more timeline events
All Facts (13)
奖励函数设计不当会导致奖励欺骗（Reward Hacking）现象，即智能体找到在奖励函数上得分高但实际无意义甚至有害的行为路径
85%VerifiedIn sparse-reward scenarios, agents often need techniques such as curriculum learning, reward shaping, or intrinsic motivation-based exploration to converge effectively.
75%Unverified约束驱动法的提示词策略只设定边界条件而不预设实现路径，与强化学习中的'奖励塑形'（reward shaping）理念一致
65%UnverifiedTraining starts with sparse rewards, and then iterates through reward shaping via an event frequency table; in the event frequency table, movement penalties account for 48.3% of all events, exploring tiles accounts for 21.7%, and attacking monsters accounts for 14.5%.
50%UnverifiedTraditional approaches include introducing intermediate rewards, reward shaping, or hindsight relabeling (such as HER), but these methods either rely on manual design or are difficult to adapt to the highly heterogeneous scenarios in MLE where code execution feedback is complex.
50%UnverifiedReward shaping refers to designing additional auxiliary reward terms beyond the sparse original reward to guide the policy, which can accelerate convergence, but if the auxiliary rewards are inconsistent with the final objective, the policy may game the score without truly improving the win rate; in practice a smaller weight is often multiplied in and decayed in the later stages of training.
50%UnverifiedEmbedding the social marginal cost into each agent's individual reward is a reward shaping strategy that requires no modification to the underlying MARL algorithm architecture.
50%UnverifiedReward shaping refers to adding extra, denser guiding reward signals on top of the original sparse reward to help agents learn faster and more correctly.
50%UnverifiedIf the reward shaping signal is too weak, the AI will take shortcuts; if it is too strong or poorly designed, it may introduce new exploits or constrain the exploration space.
50%UnverifiedWith the help of reward shaping and league play, the author ultimately trained agents to exhibit relatively interesting combat behavior.
50%UnverifiedOverly sparse rewards in Pokémon-type games make it difficult for agents to learn, so researchers typically introduce reward shaping strategies such as incentives for exploring new areas, rewards for catching Pokémon, and feedback for winning battles
50%UnverifiedExperiments found that the task_score under the shaped reward configuration was only 0.125, compared to 0.417 for the naive reward configuration, with an effect size of d=4.47.
50%UnverifiedThe reward shaping method was first systematized by Andrew Ng and others in the 1999 paper "Policy Invariance Under Reward Transformations".
50%