VinePPO
A reinforcement learning method for credit assignment in long-horizon agents that estimates state values by sampling multiple continuations from intermediate states along a trajectory in place of a learned critic network, at the cost of substantial sampling overhead.
Timeline (last 90 days)
Under the condition of matched per-step generation budgets, FOCUS at the Qwen3.5-9B scale improves by 7.2 percentage points over the strongest baseline (the higher of GRPO/VinePPO) on the BrowseComp-Plus benchmark, reaching 55.4 versus 48.2.
The core idea of VinePPO is to perform multiple continuation samples at intermediate states of a trajectory and use the mean empirical return of the continuations as a Monte Carlo estimate of the value of that state, replacing the learned critic.
The sampling overhead of VinePPO grows linearly with the number of branch points.
FOCUS conducts experiments on three types of benchmarks — legal reasoning, deep search, and code localization — comparing against GRPO and VinePPO, and outperforms both.
All Facts (4)
Under the condition of matched per-step generation budgets, FOCUS at the Qwen3.5-9B scale improves by 7.2 percentage points over the strongest baseline (the higher of GRPO/VinePPO) on the BrowseComp-Plus benchmark, reaching 55.4 versus 48.2.
50%UnverifiedThe core idea of VinePPO is to perform multiple continuation samples at intermediate states of a trajectory and use the mean empirical return of the continuations as a Monte Carlo estimate of the value of that state, replacing the learned critic.
50%UnverifiedThe sampling overhead of VinePPO grows linearly with the number of branch points.
50%UnverifiedFOCUS conducts experiments on three types of benchmarks — legal reasoning, deep search, and code localization — comparing against GRPO and VinePPO, and outperforms both.
50%