Occupancy Measure
A distribution that characterizes the frequency with which a policy visits state-action pairs in an MDP. The occupancy measure d^π(s,a) of a policy π denotes the expected number of visits to a state-action pair. Algorithms based on it can control error accumulation globally.
Timeline (last 90 days)
The occupancy measure d^π(s,a) characterizes the expected number of visits or the normalized distribution of a policy π visiting state-action pairs (s,a) in an MDP, and in episodic MDPs it simultaneously encodes the long-term behavior of the policy.
Occupancy-measure-based algorithms rewrite the policy optimization problem as convex optimization over the occupancy measure polytope, thereby globally controlling error accumulation across all state-action pairs, which is the fundamental reason their regret bound has one fewer H factor.
All Facts (2)
The occupancy measure d^π(s,a) characterizes the expected number of visits or the normalized distribution of a policy π visiting state-action pairs (s,a) in an MDP, and in episodic MDPs it simultaneously encodes the long-term behavior of the policy.
50%UnverifiedOccupancy-measure-based algorithms rewrite the policy optimization problem as convex optimization over the occupancy measure polytope, thereby globally controlling error accumulation across all state-action pairs, which is the fundamental reason their regret bound has one fewer H factor.
50%