Janus-MOPD
A framework for dynamic analysis of geometric representations in multi-teacher on-policy distillation for large reasoning models, revealing domain heterogeneity and subspace locking by tracking the evolution of the student model's internal representation space.
Timeline (last 90 days)
The paper proposes the Janus-MOPD framework, which tracks student learning dynamics from the perspective of internal geometric representations to reveal the deeper causes of domain heterogeneity.
Janus-MOPD finds that the student exhibits a three-stage learning dynamic of expansion-consolidation-optimization (ECO).
In the expansion stage, the student's representation space shows increased representation variance and rising effective dimensionality; in the consolidation stage, it shows a stabilizing representation distribution and increased intra-class compactness; in the optimization stage, it shows subtle directional adjustments and subspace alignment.
The learning process does not proceed linearly but goes through three stages of capability expansion, stable consolidation, and fine-grained optimization, each corresponding to a different pattern of representation space change.
In terms of spatial dimensions, the researchers summarized two types of subspace locking phenomena: intra-domain and cross-domain.
Janus-MOPD's geometric tracking method provides a diagnostic tool that can be used to analyze bottlenecks in multi-domain training and to guide data allocation and loss function design.
All Facts (6)
The paper proposes the Janus-MOPD framework, which tracks student learning dynamics from the perspective of internal geometric representations to reveal the deeper causes of domain heterogeneity.
50%UnverifiedJanus-MOPD finds that the student exhibits a three-stage learning dynamic of expansion-consolidation-optimization (ECO).
50%UnverifiedIn the expansion stage, the student's representation space shows increased representation variance and rising effective dimensionality; in the consolidation stage, it shows a stabilizing representation distribution and increased intra-class compactness; in the optimization stage, it shows subtle directional adjustments and subspace alignment.
50%UnverifiedThe learning process does not proceed linearly but goes through three stages of capability expansion, stable consolidation, and fine-grained optimization, each corresponding to a different pattern of representation space change.
50%UnverifiedIn terms of spatial dimensions, the researchers summarized two types of subspace locking phenomena: intra-domain and cross-domain.
50%UnverifiedJanus-MOPD's geometric tracking method provides a diagnostic tool that can be used to analyze bottlenecks in multi-domain training and to guide data allocation and loss function design.
50%