NatScale
A contrastive calibration allocation framework for mixed-precision quantization of MoE models. It estimates expert sensitivity weights via gradients and calibrates them through a power transform toward an information-theoretic contrastive objective, thereby converting ranking information into quantifiable absolute error weights.
Timeline (last 90 days)
NatScale is a contrastive-calibrated allocation framework whose workflow comprises three stages: gradient-based expert weight estimation, power-transform calibration to an information-theoretic contrastive objective, and joint evaluation of gate/up-projection/down-projection.
NatScale uses gradient information to estimate a sensitivity weight for each expert, and the gradient signal is closer to the task objective than the mere weight norm.
NatScale introduces a model-specific power transform that maps the raw gradient weights onto an information-theoretic contrastive objective, preserving the expert ordering during calibration and only adjusting the relative gaps between weights.
In NatScale, when the power-transform parameter α > 1, larger weights are further amplified and smaller weights are compressed, making the distribution steeper; when α < 1, the distribution is flatter; this process does not change the ordering among experts.
An information-theoretic contrastive objective usually refers to a loss function that measures distributional differences such as KL divergence or cross-entropy, and NatScale uses the calibrated weights as the error penalty coefficients in these objectives.
Each expert in a MoE layer contains three key configurations: gate, up, and down; NatScale jointly evaluates the impact of the combination of the three on the overall error of the expert output, rather than evaluating a single component separately.
NatScale combines the calibrated weights and error estimates with an existing prior that covers three dimensions: weight spectrum, variance, and precision decay.
The paper validates on three models, comparing five allocation baseline methods, for a total of nine settings.
Across all nine settings, NatScale achieves the lowest native-to-quantized KL divergence.
In the nine settings, NatScale achieves the highest value on the Top-token agreement rate metric in all cases.
2 more timeline events
All Facts (12)
NatScale is a contrastive-calibrated allocation framework whose workflow comprises three stages: gradient-based expert weight estimation, power-transform calibration to an information-theoretic contrastive objective, and joint evaluation of gate/up-projection/down-projection.
50%UnverifiedNatScale uses gradient information to estimate a sensitivity weight for each expert, and the gradient signal is closer to the task objective than the mere weight norm.
50%UnverifiedNatScale introduces a model-specific power transform that maps the raw gradient weights onto an information-theoretic contrastive objective, preserving the expert ordering during calibration and only adjusting the relative gaps between weights.
50%UnverifiedIn NatScale, when the power-transform parameter α > 1, larger weights are further amplified and smaller weights are compressed, making the distribution steeper; when α < 1, the distribution is flatter; this process does not change the ordering among experts.
50%UnverifiedAn information-theoretic contrastive objective usually refers to a loss function that measures distributional differences such as KL divergence or cross-entropy, and NatScale uses the calibrated weights as the error penalty coefficients in these objectives.
50%UnverifiedEach expert in a MoE layer contains three key configurations: gate, up, and down; NatScale jointly evaluates the impact of the combination of the three on the overall error of the expert output, rather than evaluating a single component separately.
50%UnverifiedNatScale combines the calibrated weights and error estimates with an existing prior that covers three dimensions: weight spectrum, variance, and precision decay.
50%UnverifiedThe paper validates on three models, comparing five allocation baseline methods, for a total of nine settings.
50%UnverifiedAcross all nine settings, NatScale achieves the lowest native-to-quantized KL divergence.
50%UnverifiedIn the nine settings, NatScale achieves the highest value on the Top-token agreement rate metric in all cases.
50%UnverifiedIn seven settings, NatScale achieves the lowest perplexity.
50%UnverifiedThe core insight of the paper is that quantization allocation should not stop at sensitivity ranking, but should shift toward objective-aware allocation, that is, reverse-calibrating each expert's error weight according to the final optimization objective.
50%