Value of Information
A quantity measuring the contribution of a piece of information to future decisions and rewards. In reinforcement learning it can uniformly lower-bound the memory-acquisition action values and define the optimal retrieval set.
Timeline (last 90 days)
For latency- and energy-constrained robotic systems, the full-bank scoring approach of traditional attention mechanisms is hard to bear, whereas the retrieval set defined by the Value of Information allows the agent to access memory more selectively.
A recent study proposes that the two problems of memory retrieval and credit assignment are governed by the same quantity—the Value of Information.
When the Value of Information is read at an observation, it gives a provable lower bound on the optimal value of the acquisition action.
This provable lower bound provides a lower bound for the TD target and attributes this lower bound to the observation that produced the information, thereby solving the credit assignment problem.
When the Value of Information is read at a decision, it defines the optimal retrieval set.
A pilot retrieval system trained on the Value of Information signal successfully solved memory tasks with reduced context.
The Value of Information can be estimated from archived trajectories with the help of a learned world model, rather than needing to be estimated in real time during online interaction.
The benefit of estimating the Value of Information offline is high sample reuse and no interference of training with the real system's operation, but the cost is greater sensitivity to the accuracy of the world model, as model bias propagates to the value estimate.
Value of Information (VoI) equals the current optimal expected loss minus the optimal expected loss after acquiring evidence; purchasing evidence is only worthwhile when VoI exceeds the cost of acquiring it.
All Facts (9)
For latency- and energy-constrained robotic systems, the full-bank scoring approach of traditional attention mechanisms is hard to bear, whereas the retrieval set defined by the Value of Information allows the agent to access memory more selectively.
50%UnverifiedA recent study proposes that the two problems of memory retrieval and credit assignment are governed by the same quantity—the Value of Information.
50%UnverifiedWhen the Value of Information is read at an observation, it gives a provable lower bound on the optimal value of the acquisition action.
50%UnverifiedThis provable lower bound provides a lower bound for the TD target and attributes this lower bound to the observation that produced the information, thereby solving the credit assignment problem.
50%UnverifiedWhen the Value of Information is read at a decision, it defines the optimal retrieval set.
50%UnverifiedA pilot retrieval system trained on the Value of Information signal successfully solved memory tasks with reduced context.
50%UnverifiedThe Value of Information can be estimated from archived trajectories with the help of a learned world model, rather than needing to be estimated in real time during online interaction.
50%UnverifiedThe benefit of estimating the Value of Information offline is high sample reuse and no interference of training with the real system's operation, but the cost is greater sensitivity to the accuracy of the world model, as model bias propagates to the value estimate.
50%UnverifiedValue of Information (VoI) equals the current optimal expected loss minus the optimal expected loss after acquiring evidence; purchasing evidence is only worthwhile when VoI exceeds the cost of acquiring it.
50%