DP-μP: A New Framework for Zero-Shot Hyperparameter Transfer in Differentially Private Training

In large-model training, hyperparameter tuning has always been a critical yet expensive step. The emergence of Maximal Update Parametrization (μP) made zero-shot hyperparameter transfer possible—find the optimal learning rate on a small model and apply it directly to a wider model. However, when training is subject to differential privacy (DP) constraints, this theory seems to break down. A recent paper proposes DP-μP to address this problem.

Why μP Fails Under Differential Privacy

The core assumption of μP is that gradients have a low-rank structure and that parameter updates are dominated by feature learning. In standard training, this assumption roughly holds. But differential privacy requires per-sample gradient clipping and injection of high-dimensional noise, and the spectral properties of the noise are fundamentally different from those of low-rank gradients.

The paper points out that high-dimensional noise is distributed more uniformly across all directions, whereas the gradient signal is concentrated in a few principal directions. This mismatch in spectral properties means the learning-rate scaling rules derived from μP no longer apply. Specifically, injecting noise into every dimension disrupts the relative ratios of learning rates across layers, causing hyperparameters tuned on a small model to degrade noticeably when transferred to a large model.

This is not something that fine-tuning can fix—it is a mismatch at the theoretical level.

μP (Maximal Update Parametrization) is a scaling theory about how neural networks are parameterized. Its core goal is to keep optimal hyperparameters (especially the learning rate) stable across different model widths, thereby enabling zero-shot transfer. The key lies in applying width-dependent adjustments to the initialization variance, learning rate, and output-layer scaling of each layer's parameters, so that network behavior in the infinite-width limit (i.e., the dynamics of "feature learning") remains consistent. What distinguishes μP from standard parametrization (SP) is that it ensures, as width tends to infinity, the update magnitude of each layer's weights stays on the same order as the network output, thereby sparing large models from retuning. The theory has been partially validated on architectures such as Transformers, but its derivation relies on the low-rank and anisotropic distribution of the gradient signal.

The Core of DP-μP: SNR Conditions and Spectral Scaling

The starting point of DP-μP is to clarify the interaction among gradient clipping, noise injection, and feature learning. The authors derive a signal-to-noise ratio (SNR) condition that characterizes the relative strength of signal versus noise at each layer under DP constraints. This condition determines how learning rates should be scaled across layers, rather than following the fixed rules of standard μP.

On this basis, the paper proposes a spectral scaling formula that combines the SNR condition with matrix spectral properties to give concrete scaling rules for different optimizers. The optimizers covered include DP-SGD, DP-Adam, and DP-Muon. This means the framework is not tailored to a single optimizer but instead aims to provide a general solution.

In terms of theoretical contribution, this amounts to extending the scope of μP from non-private training to private training, filling a gap that had previously been overlooked.

Differentially Private Stochastic Gradient Descent (DP-SGD) is the foundational optimizer for privacy-preserving machine learning. In each mini-batch, it computes gradients per sample, clips each sample's gradient by its L2 norm (limiting the norm to a threshold C), then sums the clipped gradients and injects Gaussian noise (the noise scale is determined by the privacy budget ε, δ and the clipping threshold). This mechanism ensures that the influence of any single sample on the final model is strictly bounded, thereby providing provable privacy guarantees. DP-Adam embeds the above clipping-and-noising steps into Adam's momentum estimates; DP-Muon is the private version that uses the Muon optimizer (based on matrix orthogonalization updates) for matrix parameters. What these optimizers share is that noise injection significantly alters the spectral structure of gradients, which in turn affects the relative scaling of learning rates across layers.

Experimental Validation: Zero-Shot Transfer Across Architectures

The paper validates DP-μP on three architectures: MLP, ViT, and GPT-2. The key to the experimental design is zero-shot transfer: search for the learning rate on a narrower model, then apply it directly to a wider model without additional tuning.

The results show that after transfer, DP-μP matches or even exceeds per-model individually tuned baselines, while significantly reducing computational cost. This is especially meaningful in the DP setting, because privacy training itself incurs additional computational overhead (noise generation, clipping, privacy accounting), making hyperparameter search even more costly. If zero-shot transfer is reliable, a great deal of repeated tuning can be eliminated.

You may not have noticed that the experiments cover multiple architectures from MLP to Transformer, indicating that the scaling rules are not effective only for a single class of models.

Zero-shot hyperparameter transfer refers to searching for optimal hyperparameters (such as learning rate and batch size) on a smaller or narrower proxy model, then applying these hyperparameters directly to a larger or wider model according to some scaling rule, without any additional hyperparameter tuning on the target model. This concept relies on the scaling rules provided by parametrization theories (such as μP), which make the optimal hyperparameters vary predictably with model size. In non-private training, μP has been shown to enable zero-shot learning-rate transfer from narrow to wide models on Transformers; however, in the differentially private setting, because noise injection alters the spectral properties of gradients, standard μP scaling rules fail, necessitating DP-μP to re-derive scaling formulas for privacy optimizers.

Implications for Privacy-Preserving Machine Learning Practice

Differentially private training has long faced a practical dilemma: the privacy budget is limited, and every round of tuning may consume additional budget or require complex privacy accounting. The hyperparameter transfer capability offered by DP-μP allows practitioners to complete tuning on a small-scale proxy model and then transfer the results to the production model, reducing waste of the privacy budget.

Another angle is model-scale expansion. When DP models of different widths need to be trained, the cost of per-size tuning grows linearly with the number of models. DP-μP's scaling rules allow one set of hyperparameters to cover multiple sizes, which is of direct value to teams that frequently adjust model scale.

Of course, the framework relies on SNR conditions and spectral scaling assumptions, and in real deployments one needs to verify whether these assumptions hold. The paper's empirical coverage includes image and language models, but performance under more modalities and training paradigms remains to be tested.

Conclusion

The significance of DP-μP lies in identifying the root cause of μP's failure under differential privacy—the mismatch between the spectral properties of noise and gradients—and providing a theory-driven correction. It makes zero-shot hyperparameter transfer feasible again under privacy constraints, covering mainstream optimizers such as DP-SGD, DP-Adam, and DP-Muon. For teams that need to efficiently train large models under a privacy budget, this is a direction worth watching.