GA-GRPO

The paper "When Does External Guidance Help LLM Reasoning?" proposes a unified framework called Guidance-Augmented GRPO (GA-GRPO), providing convergence rates, bias bounds, and optimal weighting rules for external guidance methods.

When the guidance signal deviates from the model's own sampling distribution, the policy gradient estimate incurs bias.

GA-GRPO abstracts external guidance as a stochastic guidance operator G; by choosing different G, vanilla GRPO, LUFFY, ExPO, PAPO, and TAPO all become special cases of this framework.

In GA-GRPO, the magnitude of bias is controlled by the guidance divergence δ_G, i.e., the total variation (TV) distance between the guidance-augmented sampling distribution and the model's own distribution.

GA-GRPO converges at a rate of O(1/√T) to an O(δ√T) neighborhood of a GRPO stationary point.

The paper derives the closed-form solution for the optimal guidance weight that minimizes mean squared error: λ*(T, δ, σ₀²) = σ₀² / (σ₀² + R_max² · δ² · T).

The optimal guidance weight formula reveals that external guidance should be trusted more in the early stages of training when the model's own variance is large, and that the guidance weight should be gradually reduced as training progresses, providing a theoretical basis for guidance annealing strategies.

The paper proves a matching minimax lower bound, showing that the Ω(δ²T) bias term is unavoidable — a bias cost must be paid whenever the guidance distribution differs from the model distribution.

The paper validates on Qwen2.5-Math-7B-Base across nine mathematical and out-of-distribution benchmarks.

GA-GRPO with optimal weights matches or surpasses TAPO, LUFFY, ExPO, and vanilla GRPO in performance, while requiring 31% fewer GPU hours.

If the quality of the guidance source is poor (large δ), extended training will amplify the bias, in which case the guidance weight should be reduced or external guidance should be stopped altogether.