TD Residual
A core signal in reinforcement learning that measures the deviation between the actual outcome and the value estimate, and is used for gradient updates of the value function and the policy.
Timeline (last 90 days)
The temporal difference (TD) residual is defined as the immediate reward plus the discounted value estimate of the next state, minus the value estimate of the current state.
Under trajectory-level sparse rewards, the TD residual propagates backward along the entire trajectory, causing the credit of early actions to be diluted or even distorted by later outcomes.
All Facts (2)
The temporal difference (TD) residual is defined as the immediate reward plus the discounted value estimate of the next state, minus the value estimate of the current state.
50%UnverifiedUnder trajectory-level sparse rewards, the TD residual propagates backward along the entire trajectory, causing the credit of early actions to be diluted or even distorted by later outcomes.
50%