Deep Learning's Three Architectures and the Universal Approximation Theorem: A Complete Guide to MLPs, CNNs, and Transformers

Deep learning is often wrapped in layer upon layer of terminology: MLP, CNN, Transformer, the Universal Approximation Theorem (UAT), Taylor expansion… Each term on its own isn't hard, but together they can leave you scratching your head. A Reddit post titled Demystifying Deep Learning: MLPs, CNNs, Transformers, UAT and Taylor Explained Simply attempts to connect them all in the most straightforward way possible. This article follows its line of thinking and re-examines these concepts within a single framework.

Three Network Architectures, Three Inductive Biases

MLP (Multilayer Perceptron), CNN (Convolutional Neural Network), and Transformer are often discussed side by side, but they don't address the same level of problem.

MLP is the most basic form: a linear transformation plus a nonlinear activation function, stacked over several layers. It makes the fewest assumptions and is therefore the most general, but it makes almost no use of input structure.

CNN bakes "locality" and "translation equivariance" into its structure—convolution kernels slide spatially with shared weights, meaning the model inherently assumes that the relationship between adjacent pixels matters more than that between distant ones. This inductive bias allows CNNs to achieve better results on image tasks with fewer parameters.

Transformer goes further, using self-attention to let any two positions in a sequence interact directly, completely abandoning the sequential dependency imposed by recurrent structures. The cost is that parameter count and computation grow quadratically with sequence length; the benefit is the ability to model long-range dependencies and better parallelism.

The three are not substitutes for one another but rather a spectrum of inductive bias strength: MLP makes the fewest assumptions, CNN makes moderate assumptions, and Transformer's assumptions lean toward "global correlation first." Choosing an architecture is essentially asking, "What structural priors exist in my data?"

Inductive bias refers to the preferences or constraints a model imposes on its hypothesis space during learning—that is, what kind of solutions the model "tends" to consider more reasonable. Every machine learning model necessarily carries inductive bias; otherwise it could not generalize from finite samples to unseen data. CNN, through local connectivity and weight sharing, implicitly assumes that "adjacent inputs are more relevant" and that "features are translation-invariant in space"; Transformer's self-attention weakens the locality assumption, allowing any position to interact directly. The stronger the inductive bias, the higher the sample efficiency on tasks that match that assumption—but once the assumption diverges from the true structure of the data, performance suffers. Architecture selection is therefore fundamentally a matter of matching prior assumptions to data structure.

The Universal Approximation Theorem: Neural Networks' "Existence Guarantee"

The Universal Approximation Theorem (UAT) is one of the most frequently cited results in deep learning theory: a single-hidden-layer feedforward network with enough hidden units can approximate any continuous function on a compact set to arbitrary precision.

The significance of this theorem lies in "existence"—it tells us that the expressive capacity ceiling of neural networks as function approximators is theoretically not a problem. But both the original post and the theoretical literature emphasize that UAT does not guarantee:

• that the network can actually be trained to that optimal solution via gradient descent;
• whether the required number of hidden units is feasible in practice;
• whether the amount of data needed for training is available.

In other words, UAT answers "can it be represented," not "can it be learned." The reason deep networks outperform shallow wide networks in practice comes more from the sample efficiency and optimization advantages of hierarchical feature reuse than from raw expressive power.

Compact set is a fundamental concept in topology, intuitively understood in Euclidean space as a set that is "bounded and closed," such as the closed interval [0,1] or the unit sphere. UAT requires approximation to occur on a compact set because continuous functions have good properties on compact sets (such as being bounded and uniformly continuous), which ensures the mathematical rigor of the approximation. If the domain is unbounded, a continuous function may be unbounded or behave pathologically, and the approximation result no longer holds. In practice, input data is usually normalized to fall within a bounded range, so the compact set assumption is generally satisfiable. Understanding this helps clarify the boundaries of UAT's applicability: it does not hold unconditionally for arbitrary functions on arbitrary domains.

Taylor Expansion: A Mathematical Lens for Understanding Neural Networks

Taylor expansion appears in this discussion because it offers a perspective on how a network progressively approximates a target function.

First-order Taylor expansion approximates a function with a tangent line, corresponding to the linear approximation in gradient descent; second-order expansion introduces curvature information, corresponding to second-order optimization methods like Newton's method. Viewing neural network training as a series of local approximations and corrections to the loss function, Taylor expansion is the mathematical language that describes these corrections.

From this perspective, UAT gives a global existence result, while Taylor expansion describes local approximation behavior—one addresses "whether it exists," the other addresses "how it approaches," forming two complementary dimensions for understanding a network's approximation capability.

The difference between first-order and second-order optimization methods can be clearly understood through Taylor expansion. Gradient descent uses only first-order derivatives (gradients), equivalent to locally approximating the loss surface with a linear function, with step size controlled by the learning rate and convergence typically linear. Newton's method uses second-order derivatives (the Hessian matrix) to introduce curvature information, equivalent to approximating with a quadratic function, achieving quadratic convergence near the minimum—but the cost of computing and storing the Hessian is O(n²) or even O(n³). With the extremely high parameter dimensionality of deep learning, exact second-order methods are infeasible, so in practice quasi-Newton methods (such as L-BFGS) or adaptive methods using only diagonal/low-rank approximations (such as Adam's estimation of the second moment of gradients) are commonly used. Taylor expansion is precisely the unifying analytical language behind the design of these methods.

Connecting the Concepts into a Single Thread

Putting the above together yields a relatively clear narrative:

1. UAT tells us that neural networks can theoretically approximate any continuous function—this is the confidence underlying the entire field;
2. MLP is the most direct embodiment of this existence result, with the simplest structure and fewest assumptions;
3. CNN and Transformer introduce inductive biases tailored to specific data structures on top of MLP, trading structural priors for sample efficiency and generalization ability;
4. Taylor expansion provides the local tool for analyzing "how the approximation process unfolds."

The value of the original Reddit post lies not in proposing new theory but in placing these often separately taught concepts back into a single narrative framework. For readers new to deep learning, this kind of connection is more helpful for building intuition than memorizing each term in isolation. It should be noted that the post is popular-science content of a conceptual organizing nature; its statements of the theorems are somewhat simplified, and rigorous mathematical details still require reference to original papers and textbooks.

Common Misconceptions

Several recurring misunderstandings surrounding these concepts are worth clarifying.

The first is treating UAT as proof that "deep networks are superior to shallow networks." The theorem itself concerns only the expressive power of single-hidden-layer networks and does not distinguish the optimization advantages brought by depth.

The second is believing that Transformer can completely replace CNN. On tasks with strong local correlation, such as images, convolution's inductive bias still offers parameter efficiency advantages, and subsequent hybrid architectures show that the two are more complementary in engineering practice.

The third is understanding Taylor expansion as some kind of training algorithm. It is an analytical tool, not an optimization method itself; saying "train the network with Taylor expansion" is a common conflation of terms.

Drawing these boundaries clearly is more valuable than memorizing the conclusions themselves.

One-line Summary

A conceptual overview connecting MLP, CNN, and Transformer through three threads: inductive bias, UAT, and Taylor expansion.

Paragraph Summary

Centered on a Reddit popular-science post, this article places often separately taught deep learning concepts back into a single framework. It first compares MLP, CNN, and Transformer, pointing out that their core difference lies in the strength of inductive bias: MLP makes the fewest assumptions, CNN builds in locality and translation equivariance, and Transformer favors global correlation—architecture selection is essentially a matter of matching data priors. It then clarifies that the Universal Approximation Theorem (UAT) only guarantees the existence of expressive power for single-hidden-layer networks on compact sets, not trainability, sample efficiency, or parameter feasibility. Finally, using Taylor expansion, it explains the complementary relationship between local approximation and global existence results, traces a conceptual thread from theoretical confidence to architecture selection to training analysis, and clarifies several common misuses.