Dataset Mismatch? How L2RR Optimizes GRPO's Rollout Efficiency

Reinforcement Learning from Verifiable Rewards (RLVR) has become a key technique for improving the reasoning capabilities of large models, and Group Relative Policy Optimization (GRPO) is one of its mainstream methods. But GRPO faces a real tension: rollout efficiency and downstream performance are hard to achieve simultaneously. Existing selective rollout methods rely solely on source-side training dynamics to optimize the prompt sampling distribution, yet they overlook a critical issue—source training datasets and target test datasets often have different distributions. When such a mismatch exists, directly sharing target prompts is not feasible due to legal or privacy restrictions. So, can we leverage limited feedback from the target side to improve GRPO's efficiency and performance without exposing target data? A new study's proposed L2RR method answers this affirmatively.

GRPO's Efficiency Dilemma and the Limitations of Selective Rollout

GRPO optimizes the policy through grouped sampling and relative rewards, performing excellently on tasks like mathematical reasoning. However, its training process requires a large number of rollouts, making computation costly. To improve efficiency, researchers have proposed selective rollout methods, which dynamically select a subset of source prompts for sampling rather than treating all prompts uniformly. These methods typically optimize a discrete distribution over source prompts based on source-side training dynamics (such as loss changes and gradient information).

The problem is that in real-world scenarios, source training sets and target test sets rarely share the same distribution. For example, source data may come from public math problem banks, while target data consists of private problems in a specific domain. When distributions mismatch, selecting prompts based solely on source-side dynamics may lead to poor performance on the target domain. More challenging still, target prompts themselves often cannot be shared with the RLVR trainer due to privacy or legal reasons. This seems to create a deadlock: we need to use target information but cannot access target data.

Rollout in the context of reinforcement learning and large model alignment refers to the process where the model generates multiple complete response trajectories for a given prompt, which are then used to compute rewards and estimate policy gradients. In GRPO, each prompt requires sampling a group of responses to compute intra-group relative advantages, so the number of rollouts directly determines the computational cost of training. Rollout efficiency refers to how, under a fixed compute budget, sampling opportunities are allocated to the most informative prompts, thereby achieving equal or better policy improvement with fewer trajectories. Understanding this is the prerequisite for grasping the motivation behind selective rollout: not all prompts contribute equally to policy updates, and redundant or overly easy prompts generate large amounts of low-gradient signal, wasting compute.

L2RR: Reweighting Rollouts with Target-Side Aggregated Feedback

The paper's proposed Learning to Reweight Rollouts (L2RR) method cleverly circumvents this limitation. Its core idea is: the target data owner can provide limited aggregated feedback (such as accuracy) from a small held-out target set without exposing target prompts or rollout trajectories. L2RR explicitly incorporates this target-side performance feedback into GRPO training.

Specifically, L2RR learns a discrete sampling distribution over source prompts by optimizing a bilevel objective: maximizing accuracy on the held-out target dataset while penalizing rollouts that provide little incremental learning information. This bilevel optimization ensures that the sampling distribution both favors source prompts useful for the target domain and avoids wasting computation on redundant prompts. Compared with methods relying solely on source-side dynamics, L2RR introduces target-side signals that directly align with downstream performance.

Bilevel optimization is a class of nested optimization problems: the outer level optimizes a set of hyperparameters or distribution parameters, while the inner level solves for the model's optimal policy given those parameters. L2RR treats the sampling distribution over source prompts as the outer variable and the GRPO policy update as the inner problem, with the outer objective being to maximize accuracy on the held-out target set and penalize low-information rollouts. The difficulty lies in that outer gradients must be backpropagated through the inner training process, typically using implicit differentiation or first-order approximations to reduce computational cost. This framework enables the sampling distribution to be directly driven by downstream target performance rather than relying solely on source-side training dynamics, thereby still aligning with the target domain under distribution mismatch.

Aggregated feedback refers to the target data owner returning only a statistic (such as accuracy or average reward on the held-out set) rather than fine-grained information about individual prompts or trajectories. This design is important for privacy protection: aggregated quantities typically satisfy differential privacy or at least do not leak the semantic content of individual samples, preventing the trainer from reverse-engineering the target prompts themselves. L2RR relies precisely on such low-bandwidth signals, which are sufficient to estimate the direction of the sampling distribution's impact on target domain performance but insufficient to reconstruct target data. The granularity and frequency of aggregated feedback thus become key trade-offs between communication cost and optimization effectiveness.

Experimental Validation: Six Benchmarks and Three Model Scales

The study conducted empirical evaluations on six mathematical reasoning benchmarks and three model scales. Results show that L2RR improves target domain performance while also increasing rollout efficiency. This means that when source and target data distributions mismatch, L2RR can more effectively utilize computational resources and achieve better downstream performance. Although the paper does not disclose specific improvement magnitudes, this direction is critical for practical deployment of RLVR—after all, data silos under privacy constraints are the norm, not the exception.

Implications for RLVR Practice

The value of L2RR lies not only in methodological innovation but also in providing a viable collaboration paradigm: target data owners need not share raw data—they only need to provide aggregated feedback to help the trainer optimize sampling strategies. This is particularly critical for applying RLVR in privacy-sensitive domains such as healthcare and finance. At the same time, this work reminds us that dataset mismatch is a non-negligible factor in GRPO practice, and ignoring it may cause efficiency optimization to deviate from its goals. In the future, designing more efficient feedback mechanisms and reducing communication costs will be worthwhile directions to explore.

One-line Summary

L2RR uses target-side aggregated feedback to reweight GRPO's rollout sampling, improving efficiency and performance under distribution mismatch with privacy constraints.

Paragraph Summary

This article addresses GRPO's efficiency dilemma in Reinforcement Learning from Verifiable Rewards (RLVR), pointing out that selective rollout methods rely solely on source-side training dynamics to optimize prompt sampling distributions while overlooking the real-world problem of distribution mismatch between source training sets and target test sets, and that target prompts often cannot be shared due to privacy or legal restrictions. To address this, the paper proposes L2RR, which learns a discrete sampling distribution over source prompts through bilevel optimization, maximizing target domain accuracy and penalizing low-information rollouts while only obtaining limited aggregated feedback from the target side (such as held-out set accuracy). Experiments on six mathematical reasoning benchmarks and three model scales show that L2RR simultaneously improves target domain performance and rollout efficiency, providing a viable paradigm for RLVR collaboration in privacy-sensitive scenarios.