margin-dispersion ratio
A metric for measuring the geometric properties of model outputs, defined as the ratio of the lead of the correct logit over incorrect logits to the dispersion of incorrect logits. It is used to explain and predict differences in fine-tuning performance across optimizers.
Timeline (last 90 days)
The researchers introduce the margin-spread ratio to explain the reversal phenomenon in fine-tuning performance.
In language models, Muon has a larger margin-spread ratio under small data budgets, while AdamW has a larger one under large data budgets.
The margin-spread ratio can serve as a diagnostic tool for predicting fine-tuning performance.
All Facts (3)
The researchers introduce the margin-spread ratio to explain the reversal phenomenon in fine-tuning performance.
50%UnverifiedIn language models, Muon has a larger margin-spread ratio under small data budgets, while AdamW has a larger one under large data budgets.
50%UnverifiedThe margin-spread ratio can serve as a diagnostic tool for predicting fine-tuning performance.
50%