RDS-MOPD
A robust budget allocation method for multi-teacher on-policy distillation. Within a distributionally robust optimization framework, it introduces noise-threshold switching, outlier-bound constraints and a worst-domain definition based on relative regression, addressing unstable resource allocation across domains under a limited training budget.
Timeline (last 90 days)
RDS-MOPD retains the objective framework of DRO but redesigns three components: allocation switching with a noise threshold, bounding constraints on single anomalies, and a redefinition of the 'worst case'.
The allocation switching mechanism with a noise threshold in RDS-MOPD specifies that budget reallocation is triggered only when the differences among domains clearly exceed the noise level.
RDS-MOPD redefines the 'worst case': the worst domain is no longer defined as the domain with the worst absolute performance, but as the domain that has 'regressed the most relative to its own best level'.
All three changes in RDS-MOPD come with theoretical guarantees.
The paper reports that on three types of benchmarks—mathematical reasoning, code generation, and instruction following—RDS-MOPD ranks first on most metrics, outperforming recent comparable methods.
All Facts (5)
RDS-MOPD retains the objective framework of DRO but redesigns three components: allocation switching with a noise threshold, bounding constraints on single anomalies, and a redefinition of the 'worst case'.
50%UnverifiedThe allocation switching mechanism with a noise threshold in RDS-MOPD specifies that budget reallocation is triggered only when the differences among domains clearly exceed the noise level.
50%UnverifiedRDS-MOPD redefines the 'worst case': the worst domain is no longer defined as the domain with the worst absolute performance, but as the domain that has 'regressed the most relative to its own best level'.
50%UnverifiedAll three changes in RDS-MOPD come with theoretical guarantees.
50%UnverifiedThe paper reports that on three types of benchmarks—mathematical reasoning, code generation, and instruction following—RDS-MOPD ranks first on most metrics, outperforming recent comparable methods.
50%