KnowGPO

KnowGPO is not suitable for scenarios where task trajectories are short and simple, where a reliable condition checker cannot be built, where knowledge requirements are too complex to decompose into atomic conditions, or where belief revision or contradictory data needs to be handled.

The KnowGPO experiments cover three benchmarks: Xbench, BrowseComp-ZH and GAIA.

The core mechanism of KnowGPO is: decompose the query into atomic knowledge conditions, treat the state at each step of the trajectory as the subset of conditions already satisfied, group by knowledge state, compare within groups to assign credit, and use the incremental knowledge gain between adjacent steps as the process reward.

The KnowGPO system architecture contains four components: a knowledge decomposer, a condition checker, a grouping function and an advantage estimator.

The paper does not clearly state whether the knowledge decomposer is rule-based, LLM-generated or learned.

The condition checker is the oracle of the entire KnowGPO framework, and noise in the checker will degrade the grouping.

The paper does not clearly address the case where mutually contradictory data is retrieved mid-trajectory, nor does it describe a rollback mechanism.

KnowGPO outperforms pure outcome-reward and process-supervision baselines by 3.8 to 4.2 points on knowledge-intensive benchmarks, and an 8B model's performance can rival a 32B open-source system.

KnowGPO introduces a new attack surface: the condition checker may be fooled by observations that falsely satisfy conditions, thereby rewarding the retrieval of junk.

The applicable scenarios for KnowGPO include: multi-turn search agents, trajectories longer than 10 steps, sparse and noisy terminal rewards, the ability to define or generate atomic knowledge conditions for the task domain, and infrastructure to run the condition checker at each step.

Complex financial analysis tasks may require more than 50 atomic conditions, leading to sparse grouping and latency overhead.