CoPE
A reinforcement learning method for language agents that constructs multi-step action prediction objectives on successful trajectories. The action policy and the prediction task co-evolve on the same backbone network, which improves sample efficiency without adding inference overhead. The shared representation lets prediction signals shape the policy while keeping deployment cost unchanged.
Timeline (last 90 days)
The researchers come from institutions including UIUC and proposed the CoPE (Co-Evolving Acting and Forecasting) method, which introduces a multi-step action objective supervision form to let the policy network simultaneously learn acting and forecasting during training.
The core mechanism of CoPE is that, on successful on-policy trajectories, it simultaneously supervises the policy backbone to generate subsequent multi-step action sequences and each action to be individually predicted from its preceding history.
CoPE maintains the conventional one-step-at-a-time action approach during inference, produces no additional computational overhead, and requires no additional models, separate pre-training, or ensemble inference.
CoPE was validated on five benchmarks: ALFWorld, ScienceWorld, WebShop, τ²-bench, and AppWorld.
On ScienceWorld, CoPE's sample efficiency improves by up to 1.9x compared to ECHO and up to 2.0x compared to GRPO.
CoPE exceeds GRPO and various RL baselines that use predictive auxiliary objectives in aggregate performance.
CoPE's gains all come from a single policy backbone, and temporal ensembling of the shared backbone is sufficient to absorb the multi-step supervision signal.
Sample efficiency and inference efficiency are two independent metrics; CoPE saves interactions during training and maintains single-step actions during inference, with no additional overhead in either.
CoPE partially transforms the sparse reward problem that has long plagued agent RL into a dense multi-step prediction problem that requires no additional annotation.
All Facts (9)
The researchers come from institutions including UIUC and proposed the CoPE (Co-Evolving Acting and Forecasting) method, which introduces a multi-step action objective supervision form to let the policy network simultaneously learn acting and forecasting during training.
50%UnverifiedThe core mechanism of CoPE is that, on successful on-policy trajectories, it simultaneously supervises the policy backbone to generate subsequent multi-step action sequences and each action to be individually predicted from its preceding history.
50%UnverifiedCoPE maintains the conventional one-step-at-a-time action approach during inference, produces no additional computational overhead, and requires no additional models, separate pre-training, or ensemble inference.
50%UnverifiedCoPE was validated on five benchmarks: ALFWorld, ScienceWorld, WebShop, τ²-bench, and AppWorld.
50%UnverifiedOn ScienceWorld, CoPE's sample efficiency improves by up to 1.9x compared to ECHO and up to 2.0x compared to GRPO.
50%UnverifiedCoPE exceeds GRPO and various RL baselines that use predictive auxiliary objectives in aggregate performance.
50%UnverifiedCoPE's gains all come from a single policy backbone, and temporal ensembling of the shared backbone is sufficient to absorb the multi-step supervision signal.
50%UnverifiedSample efficiency and inference efficiency are two independent metrics; CoPE saves interactions during training and maintains single-step actions during inference, with no additional overhead in either.
50%UnverifiedCoPE partially transforms the sparse reward problem that has long plagued agent RL into a dense multi-step prediction problem that requires no additional annotation.
50%