[KongchangAI]
ConceptAgentic-OPD

Agentic-OPD

An online policy distillation method for restoring the agent capabilities of large language models after reinforcement learning training. It dynamically selects historical checkpoints as teachers and restricts distillation to local regions where the student fails and the teacher performs better, thereby recovering complementary capabilities from discarded checkpoints while avoiding capability regression.

Timeline (last 90 days)

Nov 10

The Agentic-OPD method uses online policy distillation (OPD) to mine capabilities from the pool of discarded checkpoints, thereby restoring lost capability coverage.

Unverified50%
Nov 10

Distilling from a checkpoint pool presents two key challenges: which checkpoint to distill from (varies by problem, so static selection does not work) and where to trust the teacher (indiscriminate distillation leads to capability regression).

Unverified50%
Nov 10

Agentic-OPD treats the best validation checkpoint as the student and all saved checkpoints as frozen teacher candidates.

Unverified50%
Nov 10

The HCN (Highest Coverage Next) strategy dynamically selects teachers based on their weighted coverage of the student's unsolved problems.

Unverified50%
Nov 10

The LAD (Locality-Aware Distillation) module restricts knowledge transfer to the student's failure trajectories, and only to problems on which the teacher performs better.

Unverified50%
Nov 10

The paper evaluates on five agent task domains, 24 benchmarks and task subsets, and Agentic-OPD improves by up to 7.4 points over its initialization checkpoint.

Unverified50%
Nov 10

A completed RL training run can be re-viewed as a reusable policy pool, and 'post-RL capability recovery' is a stage independent of improvements to the RL algorithm itself.

Unverified50%

All Facts (7)

Source Articles