Bisector Direction
In a two-neuron model, the direction that carries the main signal. Learning along this direction is relatively easy, and gradient flow in its early phase performs coarse adjustment primarily along it.
Timeline (last 90 days)
Gradient flow trajectories show that the student neurons first collapse to near the saddle point along the bisector direction, then escape and specialize along the splitting direction to the corresponding teacher neurons.
In the two-neuron ReLU teacher-student model, the organization of gradient dynamics is dominated by two directions: the bisector direction carries the main signal and is easy to learn, while the splitting direction controls specialization and is hard to learn.
In the two-neuron ReLU teacher-student model, small initialization initially makes the two student neurons approximately symmetric; gradient flow first pulls them toward the bisector direction while maintaining symmetry, and the instability of the saddle point subsequently leads to symmetry breaking.
All Facts (3)
Gradient flow trajectories show that the student neurons first collapse to near the saddle point along the bisector direction, then escape and specialize along the splitting direction to the corresponding teacher neurons.
50%UnverifiedIn the two-neuron ReLU teacher-student model, the organization of gradient dynamics is dominated by two directions: the bisector direction carries the main signal and is easy to learn, while the splitting direction controls specialization and is hard to learn.
50%UnverifiedIn the two-neuron ReLU teacher-student model, small initialization initially makes the two student neurons approximately symmetric; gradient flow first pulls them toward the bisector direction while maintaining symmetry, and the instability of the saddle point subsequently leads to symmetry breaking.
50%