ReLU Network

The paper keeps the parameter magnitude at a constant level, meaning the weights do not need to grow exponentially, which is friendlier for fixed-point or low-precision deployment.

For any smoothness s>0 and dimension D≥1, a ReLU network with width O(N), depth O(L), and parameter magnitude O(1) can achieve uniform approximation error (N^2L^2 log N)^{-s/D}, and no fixed relationship between width and depth is required.

Previous ERM analyses based on neural networks typically had an extra log n factor, which could only be removed with more refined model selection or early stopping strategies.

On the corresponding truncated network class, empirical risk minimization (ERM) achieves the minimax regression rate n^{-2s/(2s+D)} with no additional logarithmic factor.

The paper proves that, under appropriate network depth and width configurations, the convergence rate of dense ReLU networks can overcome the curse of dimensionality, achieving polynomial-level sample complexity.

The paper studies two classes of estimators: the preliminary estimator and the constrained estimator; the preliminary estimator trains a separate binary regression model for each measurable set, which is simple to implement and highly parallelizable, but predictions across different sets may violate probabilistic consistency.

The paper conjectures that a stronger tight bound exists for spaces homeomorphic to spheres, with the corresponding architecture being (d, d+1, d+1).

The paper proves that a ReLU network with input dimension d, first hidden layer width Σβk(3k+2), and second hidden layer width Σβk(k+1) can express spaces homotopy equivalent to wedge sums of spheres and can possess arbitrary Betti numbers.

The arXiv preprint arXiv:2610.02253v1 studies uniform error approximation of target functions in the unit ball of Sobolev space W^{n,∞}([0,1]^d) by ReLU networks with Dropout.

Lu等人于2017年证明了宽度有限但深度无限的ReLU网络同样是普适逼近器