Constructive Bounds for the Topological Expressivity of Neural Networks: From Decision Boundaries to Output Spaces

Why Topological Expressivity Deserves a Second Look

The expressivity of a neural network — the class of functions it can represent — is one of the core concepts for understanding the upper limits of deep learning models. Among the many ways to measure expressivity, topological expressivity is particularly well suited for characterizing binary classification problems. It is typically measured by the Betti numbers of the decision boundary, $\beta(\Phi^{-1}(0))$. Betti numbers are topological invariants that describe the number of "holes" in a space: $\beta_0$ counts connected components, $\beta_1$ counts one-dimensional loops, and higher-order $\beta_k$ correspond to higher-dimensional voids.

This paper identifies a long-overlooked limitation: while decision-boundary-based metrics are popular, they simply do not apply to regression problems or other tasks that lack a decision boundary. Without one, the entire analytical framework centered on $\Phi^{-1}(0)$ loses its foothold.

Betti numbers come from homology theory in algebraic topology — they are a sequence of integers that describe the "hole" structure of a topological space. Intuitively, $\beta_0$ counts the number of disconnected "pieces," $\beta_1$ counts one-dimensional loops (like the hole through a torus), $\beta_2$ counts cavities enclosed by closed surfaces (like the interior of a hollow sphere), and so on. When two topological spaces are homeomorphic (i.e., have the same shape), all their Betti numbers must be equal, making Betti numbers powerful topological invariants. In the machine learning context, using Betti numbers to measure decision boundary complexity precisely captures how many "blobs" a network can separate or how many "hole-like" regions it can enclose — geometric information that simply counting linear regions cannot capture.

A New Perspective: Studying the Topology of the Output Space

The core shift proposed by the authors is to stop analyzing the topology of the decision boundary and instead directly study the topological structure of the network's output, $\beta(\Phi(X))$. The advantage of this perspective is its generality — whether for classification or regression, a network always has an output space, making the topological complexity of the output space a more universally applicable measure of expressivity.

This shift is not merely a formal substitution. It reframes the question from "how complex a boundary can the network carve out" to "how complex a geometric object can the network map its input space to." For tasks that require preserving or generating complex geometric structures — such as manifold learning or generative modeling — the latter framing is clearly more appropriate.

Constructive Upper Bounds: Expressing Wedge Sums of Spheres with ReLU Networks

The paper's main theorem delivers a constructive result — meaning it not only proves existence but also provides a concrete architecture. The authors show that a ReLU network with the following architecture:

$$\left(d,\ \sum_{k=1}^{d} \beta_k(3k+2),\ \sum_{k=1}^{d}\beta_k(k+1)\right)$$

can express spaces homotopy equivalent to wedge sums of spheres, and these spaces can have arbitrary Betti numbers. The three components of the tuple correspond to the input dimension, the width of the first hidden layer, and the width of the second hidden layer.

This result provides an explicit architectural upper bound for any connected ($\beta_0 = 1$) sequence of Betti numbers. In other words, given the topological complexity you want the network's output to achieve, this formula directly tells you how wide a network is sufficient — which is precisely the practical value of a constructive bound.

Wedge sum is a fundamental operation in topology for combining two spaces: take a basepoint from each space and glue those two basepoints together into a single point. For example, the wedge sum of two circles looks like the letter "∞," and the wedge sum of two spheres is two spheres sharing a single contact point. A key property of wedge sums is that their homology groups (and hence Betti numbers) equal the direct sum of those of each factor. This means that by taking wedge sums of spheres of various dimensions, one can precisely "assemble" any desired sequence of Betti numbers. This is the core logic behind the authors' use of wedge sum constructions to cover arbitrary topological complexity — it provides a modular topological "building block" system that makes constructive proofs feasible and the width cost computable.

Tighter Bounds via Product Structures

While the wedge sum construction is general, its width scales linearly with the sum of Betti numbers, which can be costly. The authors go further by constructing spaces homotopy equivalent to products of $N$ spheres (with dimensions $d_1, \dots, d_N$), obtaining significantly better bounds:

$$\left(\sum_{n=1}^N d_n,\ 3\sum_{n=1}^N d_n + 2N,\ \sum_{n=1}^N d_n + N\right)$$

The corresponding Betti numbers are given by the polynomial $\prod_n (1 + x^{d_n})$. The elegance of product spaces is that they can produce an exponentially rich combination of Betti numbers with a relatively small architecture — because the Betti numbers of a product are the tensor product expansion of those of its factors. A product of $N$ spheres naturally generates on the order of $2^N$ topological features, while the network width grows only linearly with $\sum d_n$ and $N$.

This reveals a deep intuition: structured topological complexity is far more efficiently expressible by neural networks than "unstructured" complexity. Organizing the target space via products rather than direct stacking can dramatically compress the required network capacity.

The product of topological spaces behaves very differently from the wedge sum in terms of Betti numbers. By the Künneth formula, the Betti numbers of a product space are determined by the convolution (i.e., polynomial multiplication) of those of its factors: if space $A$ has Betti number generating polynomial $P_A(x)$ and space $B$ has $P_B(x)$, then $A \times B$ corresponds to $P_A(x) \cdot P_B(x)$. This is exactly the origin of $\prod_n(1 + x^{d_n})$ in the paper — the generating polynomial of a $d_n$-dimensional sphere is $1 + x^{d_n}$ (since $\beta_0 = \beta_{d_n} = 1$ and all others are zero), and the product of $N$ spheres multiplies these polynomials together, naturally yielding $2^N$ nonzero Betti numbers. This exponential topological richness at a modest architectural cost is the fundamental reason why the product construction is far more efficient than the wedge sum construction in terms of expressivity.

A Conjecture for Tighter Bounds

The paper concludes with an ambitious conjecture: for spaces homeomorphic to a sphere, there exist much tighter bounds corresponding to the architecture:

$$(d,\ d+1,\ d+1)$$

If this conjecture holds, it would mean that expressing a $d$-dimensional sphere requires only two hidden layers each of width $d+1$ — an extraordinarily parsimonious architecture. It suggests that ReLU networks may be far more efficient at expressing basic topological objects than current proven upper bounds indicate.

From a theoretical standpoint, this conjecture pushes the problem toward "optimal expressivity": existing constructive results establish sufficient conditions, while tight bounds aim to approach necessary conditions. The gap between the two is precisely the fertile ground for future research.

Implications for Theory and Practice

This work offers several layers of value to the deep learning theory community. First, it broadens the applicability of topological expressivity beyond classification tasks. Second, the constructive proofs directly yield actionable architectural recipes rather than abstract existence arguments. Third, the product construction reveals a non-trivial relationship between topological complexity and network capacity, providing new insight into why deep networks can efficiently represent complex geometry.

Of course, as a primarily theoretical work, there remains a gap between its conclusions and the behavior of networks trained in practice — expressivity describes what can be represented, not what gradient descent can learn. But the clear architectural upper bounds and the tantalizing tight-bound conjecture undoubtedly set well-defined targets for future research.