Power Transform
A mathematical operation that maps a raw value x to x^α, where α is an adjustable parameter. By tuning the exponent α it changes the steepness of a weight distribution, altering relative gaps without changing the ordering.
Timeline (last 90 days)
NatScale introduces a model-specific power transform that maps the raw gradient weights onto an information-theoretic contrastive objective, preserving the expert ordering during calibration and only adjusting the relative gaps between weights.
In NatScale, when the power-transform parameter α > 1, larger weights are further amplified and smaller weights are compressed, making the distribution steeper; when α < 1, the distribution is flatter; this process does not change the ordering among experts.
All Facts (2)
NatScale introduces a model-specific power transform that maps the raw gradient weights onto an information-theoretic contrastive objective, preserving the expert ordering during calibration and only adjusting the relative gaps between weights.
50%UnverifiedIn NatScale, when the power-transform parameter α > 1, larger weights are further amplified and smaller weights are compressed, making the distribution steeper; when α < 1, the distribution is flatter; this process does not change the ordering among experts.
50%