[KongchangAI]
ConceptRobust Distributionally-Shifted Multi-teacher On-Policy Distillation

RDS-MOPD

A robust budget allocation method for multi-teacher on-policy distillation. Within a distributionally robust optimization framework, it introduces noise-threshold switching, outlier-bound constraints and a worst-domain definition based on relative regression, addressing unstable resource allocation across domains under a limited training budget.

Timeline (last 90 days)

Nov 10

RDS-MOPD retains the objective framework of DRO but redesigns three components: allocation switching with a noise threshold, bounding constraints on single anomalies, and a redefinition of the 'worst case'.

Unverified50%
Nov 10

The allocation switching mechanism with a noise threshold in RDS-MOPD specifies that budget reallocation is triggered only when the differences among domains clearly exceed the noise level.

Unverified50%
Nov 10

RDS-MOPD redefines the 'worst case': the worst domain is no longer defined as the domain with the worst absolute performance, but as the domain that has 'regressed the most relative to its own best level'.

Unverified50%
Nov 10

All three changes in RDS-MOPD come with theoretical guarantees.

Unverified50%
Nov 10

The paper reports that on three types of benchmarks—mathematical reasoning, code generation, and instruction following—RDS-MOPD ranks first on most metrics, outperforming recent comparable methods.

Unverified50%

All Facts (5)

Source Articles