Distributional Regression for Arbitrary Response Variables Using ReLU Neural Networks

Distributional regression is a core problem in statistics and machine learning: given a set of covariates, we want to predict not only the mean of the response variable but also characterize its full conditional distribution. Traditional methods typically assume the response variable lies in Euclidean space or rely on scalar thresholds to discretize the response space. However, real-world data are often more complex—the response variable may be categorical, functional, or even an object in a non-Euclidean space. How to build a unified, theoretically guaranteed distributional regression framework for arbitrary response spaces is a long-standing challenge.

A recent paper proposes a modeling approach based on probabilities of measurable-set events, attempting to fundamentally bypass the constraints of Euclidean structure and scalar thresholds. The method uses ReLU neural networks for estimation and theoretically proves that the convergence rate can overcome the curse of dimensionality.

From Scalar Thresholds to Measurable-Set Event Probabilities

Classical distributional regression methods typically discretize the response space, for example by dividing a continuous response into intervals using a set of scalar thresholds and then modeling the conditional probability for each interval. This strategy works well for real-valued responses, but once the response variable becomes complex—such as graph structures, tree structures, or infinite-dimensional function spaces—thresholds and Euclidean distances lose their meaning.

The core innovation of the paper is to shift the modeling target from "the response value itself" to "the probability that the response falls into a certain measurable set." Specifically, assuming the response space is some general set, the researchers choose a finite family of measurable sets and redefine distributional regression as: for each measurable set, estimate the conditional probability that the response falls into that set. This formulation preserves a direct connection to the classical event-probability framework—a conditional distribution is essentially the collection of conditional probabilities over all measurable sets.

The advantage of this perspective is that it does not require the response space to have a vector-space structure or a metric structure, and thus it can naturally handle categorical responses, ordinal responses, functional responses, and more general abstract responses.

Measurable Sets and Families of Measurable Sets: In measure theory, given a measurable space $(\Omega, \mathcal{F})$, a measurable set is a set belonging to the $\sigma$-algebra $\mathcal{F}$. The conditional distribution $P(Y \in A \mid X)$ is defined for every measurable set $A$ and satisfies probability axioms such as additivity. The core idea of this paper is to select a finite family of measurable sets ${A_1, \dots, A_m}$ and transform the distributional regression problem into estimating conditional probabilities for these $m$ sets separately. The key to this approach is that as long as the chosen family of sets is sufficiently rich, it can approximate the true conditional distribution to arbitrary accuracy. For example, for real-valued responses, a family of intervals can replace traditional scalar thresholds; for categorical responses, a family of singleton sets fully determines the distribution. Therefore, the method is naturally applicable to various abstract response spaces, but the choice of the set family affects estimation accuracy and computational complexity.

Two Complementary Estimators: Preliminary Estimation and Constrained Estimation

Within this framework, the paper studies two types of estimators.

Preliminary Estimator

The first type is the "preliminary estimator," whose approach is the most direct: for each measurable set, train a separate regression model to predict the event indicator function for that set. Each subproblem is a standard binary regression task and can be fitted with a ReLU network. The advantage of this method is its simplicity of implementation and good parallelism, but its drawbacks are also obvious—predictions across different measurable sets may contradict one another, for example, the predicted probability for a set may be greater than that for its subset, violating the basic consistency requirements of probability theory.

Constrained Estimator

The second type is the "constrained estimator," which enforces inclusion consistency by constructing a directed acyclic graph (DAG) over a finite family of sets. Specifically, if set A contains set B, then the event "the response is in B" necessarily implies "the response is in A," so the conditional probabilities should satisfy monotonicity. The DAG structure encodes these inclusion relations, and the constrained estimator explicitly incorporates these monotonicity constraints into the optimization process, thereby ensuring that the output predicted probabilities are logically self-consistent.

The two types of estimators each have trade-offs: the preliminary estimator is computationally easier to handle, while the constrained estimator is stronger in statistical consistency. The paper establishes general error bounds for both, allowing researchers to choose the appropriate method according to the specific scenario.

Directed Acyclic Graph (DAG) and Inclusion Consistency: A DAG is a directed graph with no directed cycles, often used to represent conditional dependencies or hierarchical relationships among variables. In this paper, the nodes of the DAG correspond to measurable sets, and directed edges represent inclusion relations between sets: if set $A \subseteq B$, then an edge is drawn from $A$ to $B$, meaning that the event "the response is in $A$" implies "the response is in $B$," so the conditional probabilities must satisfy $P(Y \in A \mid X) \le P(Y \in B \mid X)$. Since inclusion relations are transitive and antisymmetric, no cycles can form, so they can be encoded by a DAG. When training the ReLU network, the constrained estimator adds this monotonicity as a hard constraint or regularization term to the optimization objective, thereby ensuring that the output probability estimates are logically self-consistent across the entire family of sets. The size and structure of the DAG directly affect the difficulty of optimization, so in practice a trade-off must be made between expressive power and computational cost.

Theoretical Results: Explicit Convergence Rates and Overcoming the Curse of Dimensionality

The theoretical contributions of the paper are reflected at two levels.

The first level is a general error bound. Under relatively mild assumptions—for example, that the complexity of the family of measurable sets is controllable and that the approximation capability of the ReLU network is sufficient—the paper derives generalization error upper bounds for the preliminary estimator and the constrained estimator separately. These bounds are expressed in terms of complexity measures such as the covering number or VC dimension of the family of measurable sets, and their form is fairly general, not depending on the specific structure of the response space.

The second level is specialization to dense ReLU networks. The paper applies the general error bounds to dense ReLU networks and obtains explicit convergence rates. The key conclusion is that under appropriate network depth and width configurations, the convergence rate can overcome the curse of dimensionality. That is, even if the dimension of the response space or covariate space is high, as long as the conditional distribution has a certain smoothness structure, the ReLU network can achieve the desired estimation accuracy with polynomial-level sample complexity.

This result is theoretically crucial. There has already been extensive research on the approximation capability of deep ReLU networks, but work that combines it with the statistical convergence of distributional regression and provides explicit rates is relatively rare. The contribution of the paper lies in establishing a complete theoretical chain from network structure to statistical accuracy.

Curse of Dimensionality and Convergence Rates: The curse of dimensionality refers to the fact that as the dimension of the input space increases, the sample size required by traditional nonparametric estimation methods grows exponentially, causing estimation accuracy to deteriorate sharply. In distributional regression, if no smoothness assumptions are made about the conditional distribution, the convergence rate in high dimensions will be very slow. This paper proves that by leveraging the approximation capability of ReLU networks for conditional probability functions and assuming that the conditional distribution has a certain smoothness (such as Hölder smoothness), a polynomial-level convergence rate can be achieved, that is, the sample complexity grows only polynomially with the dimension rather than exponentially. This conclusion relies on a refined characterization of the relationship between network depth, width, and the smoothness of the target function, providing theoretical guarantees for high-dimensional distributional regression.

Experimental Validation: Simulations and Real Data

In addition to theoretical analysis, the paper also validates the practicality of the framework through simulation studies and real-data experiments. The simulation studies examine different types of response spaces and configurations of set families, comparing the performance differences between the preliminary estimator and the constrained estimator. The real-data experiments demonstrate the applicability of the method on actual datasets.

Experimental results show that the framework has practical advantages when dealing with nonstandard response variables, especially when the response space lacks a natural Euclidean structure, where event-probability-based modeling is more flexible and accurate than traditional threshold methods.

Significance of the Method and Boundaries of Applicability

The value of this work can be understood from two angles. At the methodological level, it provides a general strategy for extending distributional regression from Euclidean space to arbitrary response spaces, and this extension is not purely abstract—it preserves the connection to the classical event-probability framework, allowing theoretical tools to be reused. At the practical level, the implementation path based on ReLU networks means that the method can be executed efficiently with modern deep learning infrastructure.

Of course, the applicability of the framework depends on the choice of the family of measurable sets. If the family of sets is too simple, it may not sufficiently characterize the details of the conditional distribution; if the family of sets is too complex, the construction and optimization of the DAG in the constrained estimator may become computationally expensive. How to strike a balance between expressive power and computational feasibility is an issue that needs to be weighed in practical applications.

Overall, this paper provides a new perspective on distributional regression that has both theoretical depth and practical potential, and it is especially worth reading for researchers and practitioners concerned with modeling nonstandard response variables.

One-line Summary

The paper proposes a distributional regression framework based on measurable-set event probabilities, implemented with ReLU networks, and proves that it can overcome the curse of dimensionality.

Paragraph Summary

Aiming at the challenge of distributional regression for arbitrary response spaces, this paper proposes a unified modeling paradigm centered on measurable-set event probabilities, breaking free from the constraints of Euclidean structure and scalar thresholds. Methodologically, it studies a preliminary estimator (independent regression of event indicators) and a constrained estimator (enforcing inclusion consistency through a DAG), and establishes general error bounds for both. The theoretical analysis specializes the error bounds to dense ReLU networks, yielding explicit convergence rates and proving that under appropriate smoothness the curse of dimensionality can be overcome. Simulations and real-data experiments validate the effectiveness of the framework for nonstandard response variables. The method connects to the classical event-probability framework methodologically and relies on deep learning infrastructure in practice, but its applicability depends on the choice of the family of measurable sets, requiring a trade-off between expressiveness and computational cost.