RISED: Rubric-Guided Selection and Self-Distillation for Multi-Environment Agent Training

The New Challenge of Training General-Purpose Agents

Jointly training a single large language model (LLM) agent across diverse interactive environments is widely regarded as a promising path toward general-purpose AI agents. This direction has attracted growing research interest in recent years — if a model can simultaneously learn from coding, web navigation, tool use, and other varied task settings, it has the potential to develop cross-domain generalization rather than becoming a narrow specialist confined to a single task.

However, achieving this is far from straightforward. When multiple environments are fed into the same training pipeline, how to allocate training resources sensibly and how to filter for high-value training samples become critical factors in determining final performance. The work presented here, RISED — short for Rubrics for Agentic Multi-Environment Selection and Self-Distillation — is a direct response to these challenges.

Two Key Limitations of Existing Approaches

Currently dominant curriculum learning and data selection strategies share two notable shortcomings.

Coarse-Grained, Environment-Level Allocation

Existing methods tend to allocate training budget at the environment level, or rely primarily on local, reward-based signals to select data. The problem with this approach is that it doesn't explicitly account for the relationships between rollouts (trajectories generated through the model's interaction with an environment) across different environments. As a result, it cannot make finer-grained selections at the prompt-group level. In other words, each environment is treated as an isolated training unit, and potentially complementary or redundant relationships between samples across environments are ignored.

Broken Group-Relative Reward Signals

The second issue is more subtle. Because different environments are learned at different rates, a single training batch may simultaneously contain rollout groups where every sample fails and groups where every sample succeeds. For both of these extreme cases, the group-relative reward signal breaks down — when all samples in a group are either all correct or all incorrect, there is no relative quality gradient among group members to learn from. This data is effectively wasted. This is a well-known training efficiency loss that arises when using group-relative optimization methods such as GRPO-style algorithms.

GRPO (Group Relative Policy Optimization) is one of the leading algorithms for reinforcement learning fine-tuning of LLMs in recent years, introduced by the DeepSeek^Ad team and widely used in training models like DeepSeek-R1. Its core idea is to sample multiple outputs for the same prompt (forming a "group") and estimate the advantage function using the relative reward rankings within the group, thereby eliminating the need for a separately trained critic network as in traditional PPO. This design significantly reduces training overhead, but it introduces a structural weakness: when all outputs in a group share the same reward (all correct or all incorrect), no meaningful relative advantage can be computed, the gradient is zero, and these samples contribute nothing to parameter updates. In multi-environment joint training, because task difficulty varies enormously across environments, the model's capabilities in some environments may already be "saturated" (all correct) or not yet bootstrapped (all incorrect) at a given training stage. This raises the proportion of such "silent" samples in each batch, further amplifying training inefficiency.

RISED's Core Ideas

The name RISED encodes two technical threads.

Rubric-Guided Sample Selection

"Rubrics" (scoring criteria) signals that RISED moves beyond relying purely on raw reward values. Instead, it introduces more structured evaluation criteria to guide sample selection across multiple environments. By applying these rubrics, the model can establish a more principled basis for comparing prompt-groups across environments, enabling data selection decisions that account for the relationships between rollouts from different environments rather than treating each environment in isolation.

Curriculum Learning refers to presenting training data to a model in a deliberate order or along a difficulty gradient, rather than sampling randomly — an approach inspired by the human learning progression from easy to hard. In LLM reinforcement learning training, curriculum strategies often manifest as dynamically adjusting the sampling weights of different tasks or environments — for example, prioritizing medium-difficulty tasks the model has not yet mastered, while skipping samples that are too easy or too hard. Existing multi-environment curriculum methods typically schedule at the "environment" granularity, dynamically raising or lowering each environment's sampling proportion based on average reward or success rate. RISED's Rubrics mechanism refines this granularity to the prompt-group level, meaning that even within a single environment, sample groups corresponding to different prompts can be selectively filtered — making data selection much more responsive to the model's actual learning state on specific sub-tasks.

Self-Distillation to Fill Signal Gaps

To address the lack of group-relative reward signals in all-fail and all-succeed groups, RISED introduces a Self-Distillation mechanism. Self-distillation allows the model to extract supervisory signals from high-quality trajectories it has generated itself, providing an additional learning source for samples that would otherwise be unable to learn through group-relative rewards. This way, data that would have been "silenced" due to uneven learning speeds across environments can be recycled, improving overall data utilization during training.

The classic form of knowledge distillation uses the output distribution of a "teacher model" as soft labels to supervise a "student model," transferring knowledge from a large model to a smaller one. Self-Distillation dispenses with a separate teacher model, instead having the model learn from high-quality outputs generated by its own past checkpoints or current inference. In a reinforcement learning context, self-distillation typically means converting the model's own high-reward, complete trajectories into supervised learning signals (e.g., behavioral cloning objectives), and directly training on these trajectories via maximum likelihood — thereby bypassing the absence of group-relative reward signals. This mechanism shares its spirit with iterative self-improvement methods like STaR and ReST: the core idea is "use the model's own successes to guide its further improvement." The distinction is that RISED deploys this specifically to repair signal failures at the edge cases of multi-environment training.

Why This Matters

Building general-purpose agents is one of the most ambitious goals in AI today. Compared to maintaining a collection of siloed specialist models, a single agent capable of generalizing across environments offers clear advantages in deployment cost, maintenance complexity, and capability transfer. But the engineering and algorithmic challenges of multi-environment joint training are precisely the core obstacle standing between this vision and reality.

RISED's contribution lies in shifting attention away from coarse-grained scheduling questions like "how much training budget to give each environment," and toward the more fundamental efficiency questions of "how to perform fine-grained selection across samples from different environments" and "how to recover wasted training signals." Improvements of this kind may not be as flashy as new model architectures, but for reinforcement learning-driven agent training, data selection and signal utilization efficiency often directly determine whether training can converge to a stronger policy.

Summary

RISED addresses two practical pain points in multi-environment agent training: the absence of fine-grained, cross-environment selection at the prompt-group level, and the breakdown of reward signals caused by extreme rollout groups within a batch. Through two complementary mechanisms — rubric-guided sample selection and self-distillation — it aims to improve data utilization and generalization in multi-environment joint training.

It is worth noting that the publicly available materials at this time only disclose the method's motivation and overall framework. Specific algorithmic implementation details, experimental environment configurations, and quantitative performance improvements are still pending the release of the full paper. For researchers following reinforcement learning and general-purpose agent work, this is a development worth tracking closely.