Symmetry Breaking
A dynamical phenomenon in which a system's initial state possesses a certain symmetry that is spontaneously broken during training, causing neurons to differentiate and take on distinct functions.
Timeline (last 90 days)
Symmetry breaking refers to a system initially having some symmetry, but during training the symmetry is spontaneously broken, causing neurons to differentiate and take on different functions.
The multi-stage symmetry breaking in this model is attributed to the combined effect of standard algorithmic and architectural choices (such as small initialization, ReLU activation, and population gradient flow).
In the two-neuron ReLU teacher-student model, small initialization initially makes the two student neurons approximately symmetric; gradient flow first pulls them toward the bisector direction while maintaining symmetry, and the instability of the saddle point subsequently leads to symmetry breaking.
All Facts (3)
Symmetry breaking refers to a system initially having some symmetry, but during training the symmetry is spontaneously broken, causing neurons to differentiate and take on different functions.
50%UnverifiedThe multi-stage symmetry breaking in this model is attributed to the combined effect of standard algorithmic and architectural choices (such as small initialization, ReLU activation, and population gradient flow).
50%UnverifiedIn the two-neuron ReLU teacher-student model, small initialization initially makes the two student neurons approximately symmetric; gradient flow first pulls them toward the bisector direction while maintaining symmetry, and the instability of the saddle point subsequently leads to symmetry breaking.
50%