L2RR
A rollout efficiency optimization method for GRPO. In settings where the source and target data distributions do not match and the target data cannot be shared, it learns a source-prompt sampling distribution from limited aggregate feedback on the target side and aligns downstream performance through bilevel optimization.
Timeline (last 90 days)
The core idea of the L2RR method is that the target data owner can provide limited aggregate feedback (such as accuracy) from a small held-out target set without exposing target prompts or rollout trajectories.
L2RR learns a discrete sampling distribution over source prompts by optimizing a bilevel objective: maximizing accuracy on the held-out target dataset while penalizing those rollouts that provide little incremental learning information.
Bilevel optimization is a class of nested optimization problems: the outer level optimizes a set of hyperparameters or distribution parameters, while the inner level solves for the model's optimal policy given those parameters. L2RR treats the sampling distribution over source prompts as the outer variable and the GRPO policy update as the inner problem.
L2RR was empirically evaluated on six mathematical reasoning benchmarks and three model scales, and the results show that L2RR improves target-domain performance while also improving rollout efficiency.
L2RR offers a collaborative paradigm: the target data owner does not need to share raw data and only needs to provide aggregate feedback to help the trainer optimize the sampling strategy.
All Facts (5)
The core idea of the L2RR method is that the target data owner can provide limited aggregate feedback (such as accuracy) from a small held-out target set without exposing target prompts or rollout trajectories.
50%UnverifiedL2RR learns a discrete sampling distribution over source prompts by optimizing a bilevel objective: maximizing accuracy on the held-out target dataset while penalizing those rollouts that provide little incremental learning information.
50%UnverifiedBilevel optimization is a class of nested optimization problems: the outer level optimizes a set of hyperparameters or distribution parameters, while the inner level solves for the model's optimal policy given those parameters. L2RR treats the sampling distribution over source prompts as the outer variable and the GRPO policy update as the inner problem.
50%UnverifiedL2RR was empirically evaluated on six mathematical reasoning benchmarks and three model scales, and the results show that L2RR improves target-domain performance while also improving rollout efficiency.
50%UnverifiedL2RR offers a collaborative paradigm: the target data owner does not need to share raw data and only needs to provide aggregate feedback to help the trainer optimize the sampling strategy.
50%