Universal Approximation Theorem
A fundamental theorem in neural network theory stating that a single-hidden-layer feedforward network with a nonlinear activation function can approximate any continuous function to arbitrary precision when the number of neurons is sufficiently large. The theorem guarantees only existence, not learnability.
Timeline (last 90 days)
The universal approximation theorem only guarantees existence, not learnability; the network may need exponentially wide hidden layers, while deep structures can achieve the same effect with far fewer parameters.
The standard form of the universal approximation theorem states: for any continuous function f defined on a compact set and any ε>0, there exists a single-hidden-layer network such that for all inputs x, |f(x) - network output| < ε.
In constructive proofs of the universal approximation theorem, the number of neurons typically grows with 1/ε, which is the source of the "pessimistic bound" on the required network size.
The universal approximation theorem targets continuous functions; for discontinuous functions and cases where the domain is unbounded, the standard statement requires additional conditions.
Approximation is not the same as generalization: the existence of a wide network that can fit the training data and whether that network can perform well on new data are two different things.
All Facts (5)
The universal approximation theorem only guarantees existence, not learnability; the network may need exponentially wide hidden layers, while deep structures can achieve the same effect with far fewer parameters.
50%UnverifiedThe standard form of the universal approximation theorem states: for any continuous function f defined on a compact set and any ε>0, there exists a single-hidden-layer network such that for all inputs x, |f(x) - network output| < ε.
50%UnverifiedIn constructive proofs of the universal approximation theorem, the number of neurons typically grows with 1/ε, which is the source of the "pessimistic bound" on the required network size.
50%UnverifiedThe universal approximation theorem targets continuous functions; for discontinuous functions and cases where the domain is unbounded, the standard statement requires additional conditions.
50%UnverifiedApproximation is not the same as generalization: the existence of a wide network that can fit the training data and whether that network can perform well on new data are two different things.
50%