Overparameterization
A regime in which the number of model parameters exceeds the number of training samples. In this regime non-convex loss functions exhibit benign landscape properties, such as the absence of spurious local minima and the property that all saddle points are strict.
Timeline (last 90 days)
Classical PAC-Bayes generalization theory fails in overparameterized settings (D >> n), because the prior is defined on the Euclidean parameter space, and the relative entropy grows linearly with the network dimension D, causing the generalization bound to rapidly inflate beyond 1000%.
For any compactly supported target distribution, as long as the number of particles is sufficiently large, the MMD at every local minimum in the weighted MMD optimization landscape can be arbitrarily small.
In the field of weighted MMD optimization, previous results on overparameterized benign landscapes were mostly concentrated in neural networks or specific non-convex problems, and the corresponding theory for MMD particle optimization has remained a blank.
Zhang et al. proved in 2017 that standard deep networks can perfectly fit training data with random labels.
All Facts (4)
Classical PAC-Bayes generalization theory fails in overparameterized settings (D >> n), because the prior is defined on the Euclidean parameter space, and the relative entropy grows linearly with the network dimension D, causing the generalization bound to rapidly inflate beyond 1000%.
50%UnverifiedFor any compactly supported target distribution, as long as the number of particles is sufficiently large, the MMD at every local minimum in the weighted MMD optimization landscape can be arbitrarily small.
50%UnverifiedIn the field of weighted MMD optimization, previous results on overparameterized benign landscapes were mostly concentrated in neural networks or specific non-convex problems, and the corresponding theory for MMD particle optimization has remained a blank.
50%UnverifiedZhang et al. proved in 2017 that standard deep networks can perfectly fit training data with random labels.
50%