μP
Timeline (last 90 days)
μP (Maximal Update Parametrization) is a scaling theory about how neural networks are parameterized, whose core goal is to keep optimal hyperparameters (especially the learning rate) stable across different model widths, thereby enabling zero-shot transfer.
The derivation premises of μP rely on the low-rank nature and anisotropic distribution of the gradient signal.
Differential privacy requires per-sample clipping of gradients and injection of high-dimensional noise, and the spectral characteristics of the noise are entirely different from those of low-rank gradients, causing the learning-rate scaling rules of μP to no longer apply.
In non-private training, μP has been shown to enable zero-shot learning-rate transfer from narrow to wide on Transformers.
训练阶段化方法包括GPT-4等模型采用的渐进式sequence length增长、Chinchilla论文启发的动态batch size调度、μP参数化的分层学习率缩放
All Facts (5)
μP (Maximal Update Parametrization) is a scaling theory about how neural networks are parameterized, whose core goal is to keep optimal hyperparameters (especially the learning rate) stable across different model widths, thereby enabling zero-shot transfer.
50%UnverifiedThe derivation premises of μP rely on the low-rank nature and anisotropic distribution of the gradient signal.
50%UnverifiedDifferential privacy requires per-sample clipping of gradients and injection of high-dimensional noise, and the spectral characteristics of the noise are entirely different from those of low-rank gradients, causing the learning-rate scaling rules of μP to no longer apply.
50%UnverifiedIn non-private training, μP has been shown to enable zero-shot learning-rate transfer from narrow to wide on Transformers.
50%Unverified训练阶段化方法包括GPT-4等模型采用的渐进式sequence length增长、Chinchilla论文启发的动态batch size调度、μP参数化的分层学习率缩放
50%