Dropout

Methods to mitigate overfitting and improve generalization include data augmentation (rotation, cropping, brightness adjustment), regularization, Dropout, and introducing more real-world samples.

Dropout randomly 'turns off' a portion of neurons during training, forcing the network not to rely on specific nodes, equivalent to a cheap form of model ensembling.

The paper proves that there exist networks of constant depth and size Õ_{n,d}(p^{-9}ε^{-max{d/n,2}}log(1/δ)) that satisfy the probabilistic uniform approximation requirement with probability at least 1-δ.

The upper-bound network size has a ninth-power dependence of p^{-9} on the Dropout retention probability p, meaning that the lower the retention probability, the more sharply the required network size grows.

An excessively high Dropout rate (i.e., p being too small) incurs a heavy penalty on approximation capability, which is worth considering when designing regularization strategies.

This research establishes a non-asymptotic, probabilistically guaranteed approximation theory for Dropout, refining qualitative approximation results into quantitative characterizations of network size and success probability.

Even when approximating only a single fixed affine function, at a sufficiently high confidence level, the cost of the output layer must reach ((1-p)/p)ε^{-2}log(1/δ).

The arXiv preprint arXiv:2610.02253v1 studies uniform error approximation of target functions in the unit ball of Sobolev space W^{n,∞}([0,1]^d) by ReLU networks with Dropout.

In 2016, Yarin Gal proved that Dropout training is equivalent to a form of approximate Bayesian inference.

Common strategies for addressing overfitting on small datasets include data augmentation, regularization techniques (such as Dropout and weight decay), early stopping, and controlling model capacity.