DP-μP
A theoretical framework that extends Maximal Update Parametrization (μP) to differentially private training. Through signal-to-noise ratio conditions and spectral scaling formulas, it enables zero-shot hyperparameter transfer under privacy-preserving optimizers.
Timeline (last 90 days)
The DP-μP framework derives a signal-to-noise ratio (SNR) condition that characterizes the relative strength of signal and noise at each layer under the DP constraint, and based on this determines how learning rates should scale across layers.
DP-μP proposes a spectral scaling formula that combines the SNR condition with matrix spectral properties to give concrete scaling rules for different optimizers.
The optimizers covered by DP-μP include DP-SGD, DP-Adam and DP-Muon.
The paper validates the effectiveness of DP-μP on three architectures—MLP, ViT and GPT-2—with an experimental design that searches learning rates on narrower models and then applies them directly to wider models without additional tuning.
Experimental results show that the transferred performance of DP-μP matches or even exceeds per-model individually tuned baselines while significantly reducing computational cost.
The DP-μP framework relies on the SNR condition and the spectral scaling assumptions, and in practical deployment it is necessary to verify whether these assumptions hold.
The paper's empirical evidence covers image and language models, but performance under more modalities and training paradigms remains to be tested.
All Facts (7)
The paper's empirical evidence covers image and language models, but performance under more modalities and training paradigms remains to be tested.
50%UnverifiedDP-μP proposes a spectral scaling formula that combines the SNR condition with matrix spectral properties to give concrete scaling rules for different optimizers.
50%UnverifiedThe optimizers covered by DP-μP include DP-SGD, DP-Adam and DP-Muon.
50%UnverifiedThe DP-μP framework derives a signal-to-noise ratio (SNR) condition that characterizes the relative strength of signal and noise at each layer under the DP constraint, and based on this determines how learning rates should scale across layers.
50%UnverifiedExperimental results show that the transferred performance of DP-μP matches or even exceeds per-model individually tuned baselines while significantly reducing computational cost.
50%UnverifiedThe DP-μP framework relies on the SNR condition and the spectral scaling assumptions, and in practical deployment it is necessary to verify whether these assumptions hold.
50%UnverifiedThe paper validates the effectiveness of DP-μP on three architectures—MLP, ViT and GPT-2—with an experimental design that searches learning rates on narrower models and then applies them directly to wider models without additional tuning.
50%