RDS-MOPD: Using Robust Optimization to Solve the Domain Allocation Challenge in Multi-Teacher Distillation

RDS-MOPD makes DRO reliably allocate multi-teacher distillation budgets under noise through three modifications.
The article discusses allocating cross-domain training budgets in multi-teacher online policy distillation (MOPD). Existing progress-based methods only reward fast-improving domains and lack a safety net for lagging ones, letting weak spots worsen. DRO can tilt resources toward the worst domains, but noisy signals from few rollouts cause three problems: mistaking noise for the worst case, spikes hijacking the budget, and indiscriminate subsidies for inherent weaknesses. The authors propose RDS-MOPD, redesigning three DRO components—noise-threshold allocation switching, outlier bounds, and redefining the worst case as regression relative to a domain's own best level—with theoretical guarantees. Experiments rank it first on most benchmarks.
Multi-teacher online policy distillation (MOPD) has a straightforward goal: compress the capabilities of multiple domain-expert teachers—mathematical reasoning, code generation, instruction following, and others—into a single compact student model. But all domains share one limited training budget, and how that budget is split among domains directly determines the success or failure of distillation.
Existing approaches allocate budget based on each domain's learning progress. That sounds reasonable, but the problem is that it offers no guarantee for lagging domains—a domain may go long without sufficient training, ultimately dragging down overall performance. To address this gap, researchers introduced distributionally robust optimization (DRO) into MOPD, tilting training resources toward the worst-performing domains. The idea itself holds up, but putting it into practice ran into new troubles.

Why the Old Budget Allocation Approach Fails
The core tension in MOPD is the pull between a "shared budget" and "domain differences." Mathematical reasoning and code generation require different forms of training signal and converge at different rhythms. If budget is allocated by average progress, strong domains keep absorbing budget while weak domains fall into a vicious cycle: the less training they get, the worse they perform, and the harder it is to secure resources in the next round of allocation.
Methods that allocate by learning progress try to mitigate this, but they only reward domains that "improve quickly" and lack any safety net for domains that "regress" or "stagnate." In other words, they optimize overall efficiency, not the worst case. When a domain happens to be the student model's weak spot, this allocation strategy keeps that weak spot weak indefinitely.
The introduction of DRO is meant to fill this safety-net gap. It shifts the optimization objective from "average performance" to "worst-domain performance," which in theory guarantees that no domain is systematically neglected.
Distributionally Robust Optimization (DRO) is a class of optimization frameworks for decision-making under uncertainty. Classical empirical risk minimization focuses only on average loss, whereas DRO assumes the true data distribution lies within an "ambiguity set" centered on the empirical distribution, and minimizes the expected loss under the worst distribution within that set. In the context of budget allocation, it is equivalent to assigning each domain a dynamically adjustable weight that tilts toward the currently worst-performing domain, thereby theoretically guaranteeing bounded worst-case performance. The radius of the DRO ambiguity set is a key hyperparameter: a larger radius is more conservative and favors even allocation, while a smaller radius is more aggressive and stays closer to the empirical optimum. This property also explains why noise significantly affects its performance, as discussed later—the radius estimate itself is sensitive to signal quality.
Three Pitfalls of Applying DRO Directly
The paper's authors point out an easily overlooked reality: the training signal for each domain is estimated from a small number of rollouts and is inherently noisy. Running DRO directly on this noisy signal triggers three types of problems.
Mistaking noise for the worst case. A sampling fluctuation in one round may make a domain's estimated performance suddenly appear worse, and DRO mistakenly treats this as a genuine performance decline, tilting budget in the wrong direction.
Occasional spikes hijacking the budget. The influence of a single anomalous sample gets amplified—one偶然 bad result is enough to change the entire allocation landscape, and stability goes out the window.
Protecting domains that are "inherently difficult" rather than those that are "genuinely regressing." Some domains are simply hard to learn, and their performance is naturally lower than others. If DRO fails to distinguish among them, it will keep pouring resources into these "hard nuts" rather than into domains that previously performed well but have recently regressed.
The common root of these three pitfalls is that the original form of DRO assumes the signal is reliable, whereas the signal in the MOPD setting is not.
Rollout refers to the several complete output trajectories that a model samples for the same input during training or inference, and it is a common means of estimating policy performance in reinforcement learning and online distillation. Because sampling is stochastic, a domain's performance estimated from a small number of rollouts is essentially a finite-sample estimate, and its variance grows as the sample size shrinks. In MOPD, each domain's training signal is aggregated from exactly this kind of small-sample rollout, so the estimate fluctuates around the true level. The smaller the sample size and the greater the within-domain output variance, the more pronounced the fluctuation. This is precisely the source of the necessity for noise-threshold design and outlier-bound constraints: without explicitly modeling this estimation error, an allocation decision based on a single round of rollouts amounts to optimizing on noise.
Three Redesigns in RDS-MOPD
To address these problems, the authors propose RDS-MOPD, which retains DRO's objective framework but redesigns three of its components and provides theoretical guarantees.
The first change is allocation switching with a noise threshold. Budget reallocation is triggered only when the difference between domains clearly exceeds the noise level. When the difference falls within the noise range, the allocation stays unchanged. This directly plugs the loophole of "noise being misjudged as the worst case."
The second change is bounding constraints on single outliers. The influence of any single anomalous signal is capped within a controllable range, preventing one偶然 spike from dominating the entire budget trajectory. This solves the problem of "occasional spikes hijacking the budget."
The third change is redefining the "worst case." The worst domain is no longer defined as the one with the absolute worst performance, but as the one that has regressed the most relative to its own best level. This shift in perspective is crucial: it separates "inherently difficult" from "genuinely regressing," directing budget toward domains that once performed well but have recently slipped, rather than indiscriminately subsidizing inherent weaknesses.
All three changes come with theoretical guarantees, showing that they are not engineering expedients but well-founded adjustments within the optimization framework.
What the Experimental Results Show
The paper reports that RDS-MOPD ranks first on most metrics across three types of benchmarks—mathematical reasoning, code generation, and instruction following—outperforming recent comparable methods. This result spans three domains with markedly different characteristics, indicating that the method does not work on only a single task.
Reading backward from the results, the value of RDS-MOPD lies in turning "robustness" from an abstract goal into an operational allocation rule. DRO provides the direction, the three redesigns address the noise and stability issues in practice, and only their combination makes worst-case optimization truly usable in the MOPD setting.
For teams actually doing multi-teacher distillation, the takeaway from this work is not just a specific algorithm but a more general principle: any dynamic scheduling mechanism that relies on online-estimated signals must confront signal noise head-on; otherwise, the more aggressive the optimization objective, the greater the risk of being led astray by noise.
One-line Summary
RDS-MOPD makes DRO reliably allocate multi-teacher distillation budgets under noise through three modifications.
Paragraph Summary
The article discusses the problem of allocating cross-domain training budgets in multi-teacher online policy distillation (MOPD). Existing methods that allocate by learning progress only reward fast-improving domains and lack a safety net for lagging domains, easily allowing weak spots to worsen. Introducing distributionally robust optimization (DRO) can tilt resources toward the worst domains, but in MOPD each domain's training signal comes from a small number of rollouts and is noisy; applying DRO directly leads to three problems: mistaking noise for the worst case, occasional spikes hijacking the budget, and indiscriminately subsidizing inherent weaknesses. The authors propose RDS-MOPD, redesigning three components of DRO: allocation switching with a noise threshold, bounding constraints on single outliers, and redefining the worst case in terms of regression relative to a domain's own best level, with theoretical guarantees. Experiments show it ranks first on most benchmarks for mathematical reasoning, code generation, and instruction following.
Related articles

FBI Warns About AI Hacking Tool Artex: Systems Breached in Three Hours
FBI warns about AI hacking tool Artex: describe targets in natural language for automated scanning and intrusion, with some attacks achieving full privileged access within three hours. It affected over 2,000 Microsoft 365 organizations and South Korean banks. Copies still circulate after the developer took it down in October; security fundamentals remain key.

FBI-Warned AI Hacking Tool ARTEX: Cost Disruption and Defensive Restructuring
The FBI-warned AI hacking tool ARTEX reduces penetration testing costs to $150-$1,500, causing financial data breaches and supply chain attacks. Learn how it works, real cases, and a five-step defense strategy.

FBI Warns: ARTEX AI Hacking Tool Breaches Systems in Three Hours
FBI warns the open-source AI pentesting framework ARTEX is being abused, with some intrusions gaining privileged access in under three hours, over 1,600 consoles exposed, and links to South Korean financial data breaches and the ConnectWise supply chain attack.