Gradient Descent on Two ReLU Neurons: Global Landscape and Bifurcation Dynamics

Title

Gradient Descent on Two ReLU Neurons: Global Landscape and Bifurcation Dynamics

Meta Description

This article interprets the global loss landscape and bifurcation dynamics of gradient descent in a two-neuron ReLU teacher-student model. The study shows that the dynamics are organized by the bisector direction and the splitting direction; all nonzero critical points are either global minima or saddle points; and gradient flow undergoes three stages—collapse, escape, and specialization—revealing a multi-stage symmetry-breaking mechanism.

Article Content

Understanding how gradient descent learns features remains a core challenge in neural network theory. A recent paper studies this problem in a minimal multi-index setting: a well-specified two-neuron ReLU teacher-student model with isotropic Gaussian inputs, trained from small initialization using population gradient flow. The authors reveal that the dynamics are organized by two directions: the "easy" bisector direction that carries the main signal, and the "hard" splitting direction that governs specialization. Analysis of the population squared loss landscape shows that all critical points with nonzero student weights are either global minima or saddle points. Gradient flow trajectories show that the student neurons first collapse toward the bisector saddle point, then escape and specialize to the teacher neurons. These results provide a landscape and dynamical explanation for the multi-stage symmetry-breaking behavior that emerges in simple multi-index models.

Rich Dynamics in a Minimal Model

The paper adopts a seemingly simple setup: a teacher network with two ReLU neurons, a student network also with two ReLU neurons, and inputs sampled from an isotropic Gaussian distribution. Training uses population gradient flow (i.e., gradient descent in the infinite-sample limit), starting from small initialization. Despite its small size, the model already exhibits multi-stage gradient dynamics, including symmetry breaking and specialization. This provides an analytically tractable window for understanding the nature of feature learning in real neural networks.

The authors point out that the organization of the dynamics depends on two key directions. The bisector direction carries the main signal and is relatively easy to learn; the splitting direction controls specialization among neurons and is the hard part of learning. This distinction helps explain why gradient descent first performs "coarse tuning" in the early stage and then enters "fine tuning."

Population gradient flow is an idealized training dynamics model commonly used in theoretical analysis. It assumes an infinite sample size, so the loss function is the expected loss over the true data distribution (population loss) rather than the empirical loss on a finite training set. In this setting, the discrete updates of gradient descent are replaced by continuous-time gradient flow (an ordinary differential equation), allowing convergence and trajectories to be analyzed with tools from dynamical systems theory. Population gradient flow ignores mini-batch noise and the randomness of finite samples, but it can clearly reveal how the geometry of the loss landscape affects the optimization path, making it a classic starting point for understanding feature learning dynamics.

The teacher-student model is a standard framework for analyzing feature learning in neural networks. The teacher network is a target network with known weights that generates labels or pseudo-labels; the student network starts from random initialization and is trained to imitate the teacher's output. By comparing the evolution of student and teacher weights, one can study whether the network learns the true feature structure represented by the teacher. The advantage of this framework is that it allows precise control over the structure of the objective function, enabling analysis of gradient descent behavior under analytically tractable conditions, such as symmetry breaking and neuron specialization.

Loss Landscape: Classification of Critical Points

To characterize the loss structure, the authors analyze the population squared loss landscape. A key conclusion is that any critical point with all student weights nonzero is either a global minimum or a saddle point. This means there are no suboptimal local minima in the landscape—all non-global critical points are saddle points, which gradient flow can escape.

This result echoes previous research on matrix factorization and overparameterized models, but here it is given in the specific setting of a teacher-student ReLU network. It shows that even in the presence of nonconvexity, the geometry of the landscape still favors optimization: saddle points, rather than bad local minima, are the main obstacle.

Critical points are points where the gradient of the loss function is zero, including local minima, global minima, saddle points, and local maxima. In nonconvex optimization, the type of critical point determines whether gradient flow can escape and where it ultimately converges. A saddle point is a critical point where the loss increases in some directions and decreases in others; gradient flow can escape along directions of negative curvature. The core conclusion of this paper—that all critical points with nonzero student weights are either global minima or saddle points—implies that there are no bad local minima in the loss landscape, so gradient flow will not get trapped in suboptimal solutions. This property is often called a "benign landscape" and is the geometric basis for why overparameterized models are easy to optimize.

Gradient Flow Trajectories: Collapse, Escape, and Specialization

The paper further traces the trajectories of gradient flow. The student neurons first collapse toward a saddle point along the bisector direction, at which point the two neurons tend to become identical and the network has not yet specialized. Subsequently, the dynamics escape from that saddle point and differentiate along the splitting direction, ultimately with each student neuron specializing to the corresponding teacher neuron.

This "collapse first, specialize later" pattern explains multi-stage symmetry breaking: the process by which the initial symmetry is broken does not happen all at once, but rather through an intermediate saddle-point stage. The authors attribute this dynamics to the combined effect of standard algorithmic and architectural choices (such as small initialization, ReLU activation, and population gradient flow).

Symmetry breaking refers to a situation in which the system has some symmetry in its initial state (for example, two student neurons have identical or equivalent weights), but this symmetry is spontaneously broken during training, causing the neurons to differentiate and take on different functions. In the two-neuron ReLU teacher-student model, small initialization makes the two student neurons approximately symmetric; gradient flow first pulls them toward the bisector direction (while still maintaining symmetry), and then the instability of the saddle point leads to symmetry breaking, with the neurons differentiating along the splitting direction and each specializing to a different teacher neuron. This process is multi-stage: first coarse tuning to the symmetric saddle point, then fine tuning to the specialized solution. Symmetry breaking is a ubiquitous phenomenon in feature learning in neural networks, and understanding its dynamics helps explain the mechanisms by which neurons differentiate in function across layers in deep networks.

Implications for Neural Network Theory

This work belongs to frontier research in neural network theory on feature learning and optimization dynamics. Through an analytically tractable minimal model, it demonstrates how gradient descent gradually learns structured representations from a featureless state. Although the model is simple, the bisector-splitting direction organization, the classification of critical points, and multi-stage symmetry breaking that it reveals may provide insights for understanding similar phenomena in deeper networks.

You may not have noticed that the analysis relies on idealized assumptions such as population gradient flow and isotropic Gaussian inputs, and generalization to finite samples and more complex data distributions still requires further study. But as a theoretical foundation, it clearly characterizes the global landscape and bifurcation behavior of gradient descent in a two-neuron ReLU network.

The isotropic Gaussian input assumption means that input vectors are sampled from a Gaussian distribution with zero mean and identity covariance matrix, i.e., each dimension is independent and identically distributed with equal variance. This assumption gives the input distribution rotational symmetry, thereby simplifying the computation of the expected loss and making the direction of the weights (rather than only their norm) the focus of analysis. In real data, inputs often have correlations and non-Gaussian structure, so the conclusions of this paper cannot be directly generalized. However, the isotropic Gaussian assumption is a common starting point in theoretical analysis because it allows analytical computation of activation patterns in ReLU networks in high-dimensional space and clearly isolates the role of directional dynamics (such as the bisector and splitting directions).

One-line Summary

In a two-neuron ReLU teacher-student model, gradient descent first collapses to a bisector saddle point and then specializes, and the landscape has no suboptimal local minima.

Paragraph Summary

This paper studies the feature learning dynamics of a minimal two-neuron ReLU teacher-student model under population gradient flow. The authors find that the dynamics are organized by the bisector direction (easy, carrying the main signal) and the splitting direction (hard, controlling specialization). Analysis of the population squared loss landscape shows that all critical points with nonzero student weights are either global minima or saddle points, with no suboptimal local minima. Gradient flow trajectories show that student neurons first collapse to the bisector saddle point, then escape and specialize to the teacher neurons along the splitting direction. This provides a landscape and dynamical explanation for multi-stage symmetry breaking, and the authors note that the analysis is based on idealized assumptions such as population gradient flow and isotropic Gaussian inputs.