Understanding the Universal Approximation Theorem with Building Blocks: Why Neural Networks Can Approximate Any Function

Can a feedforward neural network with just a single hidden layer really approximate any continuous function? This is one of the most counterintuitive and frequently asked propositions in deep learning theory. In Chapter 4 of Neural Networks and Deep Learning, Michael Nielsen offers a visual argument that relies on almost no formulas: a neural network is essentially a pile of tiny "bricks," and by stacking bricks you can piece together a function curve of any shape.

A creator on Reddit turned this argument into a step-by-step animated video, accompanied by a Python code repository, turning an abstract proof into a process you can actually "see." This article follows that visual line of reasoning to unpack the constructive logic behind the universal approximation theorem.

What the Universal Approximation Theorem Actually Says

The statement of the theorem is not complicated: a single-hidden-layer feedforward network with a nonlinear activation function can approximate any continuous function to arbitrary precision, provided the number of hidden-layer neurons is sufficiently large.

This sentence is often misread as "one hidden layer is enough, so deep networks are unnecessary." In reality, the theorem only guarantees existence, not learnability—the network may need an exponentially wide hidden layer, whereas deep architectures can achieve the same effect with far fewer parameters. This is precisely what Nielsen emphasizes repeatedly in Chapter 4: the theorem answers the question of the upper bound of expressive capacity, not training feasibility.

The original Reddit post summarized the core intuition in one sentence: "a neural network really is a pile of tiny bricks"—the network is not fitting a complex curve directly, but piecing that curve together from many simple little blocks.

How the Bricks Are Made: From Step Functions to Sigmoid

To understand stacking, you first need to see clearly what a single neuron can do.

Weighted Sum Plus Bias: Creating a "Switch Point"

A single neuron first computes a weighted sum of its inputs plus a bias, then feeds the result into an activation function. In the visual argument, the key is to tune the bias so that the weighted sum jumps from a large negative value to a large positive value near some point. At this point the Sigmoid output exhibits a steep S-shaped transition—essentially a "softened" step function.

Increase the weight, and the transition becomes steeper, with the S-shape approaching an ideal step ever more closely. Thus a single neuron becomes a "brick with a slope."

The Sigmoid function is defined as σ(z) = 1/(1+e^{-z}), with derivative σ'(z) = σ(z)(1-σ(z)), which attains its maximum value of 0.25 at z=0. This property explains why simply increasing the weight makes the transition steeper: as the weight w increases, the rate of change of the weighted sum z = wx+b near x grows, and after passing through the Sigmoid the width of the output transition interval is roughly proportional to 1/|w|. Therefore, adjusting the weight controls the steepness of the "brick's" slope, while adjusting the bias controls where the slope occurs. This is exactly the mathematical origin of the two adjustable knobs used to construct a step approximation.

Subtracting Two Steps: Getting a Bump-Shaped Brick

A single step can only raise the function as a whole. To create a local bump, you need to subtract two steps: use one neuron to lift at the left endpoint and another neuron to lower at the right endpoint, and subtract the two outputs to get a rectangular pulse.

This is the most critical "building block" unit in the visual argument: a pair of hidden-layer neurons can construct a bump of any position and any width. Stacking these bumps together yields a piecewise approximation of the target function.

Constructing a bump by subtracting two Sigmoids essentially exploits the difference approximation of Sigmoids: when the transition points of two Sigmoids are located at the two ends of an interval [a, b], their difference is approximately constant inside the interval and approaches zero outside it, thus forming a rectangular pulse. More formally, if σ(w(x-a)) - σ(w(x-b)) with w large, its value is approximately 1 between a and b and approximately 0 outside the interval, with width b-a. This is completely consistent with the classical signal processing idea of obtaining a rectangular window by subtracting two Heaviside step functions, except that the Sigmoid provides a smooth, differentiable version, making subsequent gradient-based training possible.

Stacking Bricks: The Constructive Logic of Approximating Any Curve

With bump-shaped bricks in hand, all that remains is stacking.

The Reddit video follows Nielsen's path step by step: the more curved the target function, the more bricks are needed; the narrower the bricks, the smaller the approximation error. By increasing the number of hidden-layer neurons sufficiently, a finite number of bumps can enclose the target curve to arbitrary precision.

This construction provides the skeleton of the theorem's proof, but it also exposes its cost. To approximate a rapidly oscillating function, the number of required bricks may grow exponentially with the precision requirement. This is precisely the core tension Nielsen points out: the universal approximation theorem guarantees an upper bound on expressive capacity, yet gives a very pessimistic bound on the required network size.

The video's author mentions in the post that the GitHub repository includes corresponding Python code, allowing you to reproduce the stacking process of these bricks by hand—for understanding the theorem, building it once yourself is more effective than reading the proof ten times.

The "Can" and "Cannot" of a Single Hidden Layer

The visual argument is easy to over-interpret, and several boundaries deserve to be clarified separately.

First, the theorem is about continuous functions. For discontinuous functions or cases with an unbounded domain, the standard statement requires additional conditions.

Second, approximation is not generalization. That a wide network exists that can fit the training data, and that this network can perform well on new data, are two different things. The visual argument only addresses the former.

Third, depth brings efficiency, not expressive power. A single hidden layer is not inferior to deep networks in expressive capacity, but deep architectures can express the same family of functions with fewer parameters, which in practice determines trainability. Nielsen's book discusses this relationship specifically, while the Reddit video focuses on the construction itself.

The standard form of the theorem is usually stated as: for any continuous function f defined on a compact set and any ε>0, there exists a single-hidden-layer network such that |f(x) - network output| < ε for all inputs x. Here "compact set" means a bounded closed set, excluding the case of an unbounded domain; and "continuous" excludes jump points. Moreover, the classical proofs of the theorem (such as Cybenko 1989, Hornik 1991) rely on the activation function satisfying conditions such as being non-polynomial and bounded, which both Sigmoid and ReLU satisfy. It is worth noting that the theorem does not require the number of hidden-layer neurons to grow polynomially as precision increases—in constructive proofs the number of neurons typically grows with 1/ε, which is precisely the source of the "pessimistic bound."

Why This Visualization Is Worth Watching

Regarding proofs of the universal approximation theorem, most textbooks choose to complete them in the language of measure theory or functional analysis; readers can verify each step but find it hard to form intuition. The value of Nielsen's Chapter 4 lies in replacing the proof with a "building blocks" narrative: first make bricks, then stack bricks, and finally piece together the curve with bricks.

What the Reddit creator did was to further animate this narrative chain and pair it with runnable code. For those reading the book or stuck on the abstract statement of the theorem, this presentation lowers the barrier to understanding and turns "why neural networks can fit everything" from a slogan into a set of operations you can reproduce yourself.

If you plan to try it yourself, the recommended order is: first reproduce a single Sigmoid step in code, then reproduce the bump from subtracting two steps, and finally stack multiple bumps to approximate a nontrivial function. After these three steps, the theorem no longer needs to be memorized.

One-line Summary

A visual "building blocks" breakdown of the universal approximation theorem: how a single-hidden-layer network approximates any continuous function.

Paragraph Summary

This article centers on the universal approximation theorem, using Michael Nielsen's visual argument from Chapter 4 along with a Reddit creator's animation and code to unpack the constructive logic by which a single-hidden-layer network approximates any continuous function. The core idea is to tune a single Sigmoid neuron's weights so it approximates a step function, subtract two steps to obtain a local bump "brick," and then stack multiple bumps to approximate the target curve. The article also clarifies the theorem's boundaries: it guarantees only existence, not trainability or generalization; it gives a pessimistic bound on network size; and the advantage of deep architectures lies in parameter efficiency rather than expressive power. Finally, it recommends a three-step code reproduction to truly understand the theorem.