Regularized Q-function
A technique that introduces convex regularization terms (such as entropy or L2 regularization) into Q-function updates, causing the Q-value updates for each state-action pair to be coupled with the global policy distribution through the regularizer. It acts as a global stabilizer and eliminates the horizon gap in policy optimization.
Timeline (last 90 days)
A recent theoretical study, by regularizing the Q-function, improves the regret bound of policy optimization under adversarial MDPs by a full factor of H, thereby eliminating the horizon gap.
A regularized Q-function refers to introducing a convex regularizer (such as entropy regularization or L2 regularization) into the Q-function update, so that the Q-value update of each state-action pair is coupled with the global policy distribution through the regularizer, acting as a global stabilizer.
The paper extends the algorithm to adversarial linear mixture MDPs and obtains the same improvement in horizon dependence, indicating that the regularized Q-function idea does not depend on the finiteness of the tabular setting.
All Facts (3)
A recent theoretical study, by regularizing the Q-function, improves the regret bound of policy optimization under adversarial MDPs by a full factor of H, thereby eliminating the horizon gap.
50%UnverifiedA regularized Q-function refers to introducing a convex regularizer (such as entropy regularization or L2 regularization) into the Q-function update, so that the Q-value update of each state-action pair is coupled with the global policy distribution through the regularizer, acting as a global stabilizer.
50%UnverifiedThe paper extends the algorithm to adversarial linear mixture MDPs and obtains the same improvement in horizon dependence, indicating that the regularized Q-function idea does not depend on the finiteness of the tabular setting.
50%