Splitting Direction
In a two-neuron model, the direction that controls neuron specialization. It is the difficult part of learning; in the later phase gradient flow differentiates along this direction to complete fine-tuning.
Timeline (last 90 days)
Gradient flow trajectories show that the student neurons first collapse to near the saddle point along the bisector direction, then escape and specialize along the splitting direction to the corresponding teacher neurons.
In the two-neuron ReLU teacher-student model, the organization of gradient dynamics is dominated by two directions: the bisector direction carries the main signal and is easy to learn, while the splitting direction controls specialization and is hard to learn.
All Facts (2)
Gradient flow trajectories show that the student neurons first collapse to near the saddle point along the bisector direction, then escape and specialize along the splitting direction to the corresponding teacher neurons.
50%UnverifiedIn the two-neuron ReLU teacher-student model, the organization of gradient dynamics is dominated by two directions: the bisector direction carries the main signal and is easy to learn, while the splitting direction controls specialization and is hard to learn.
50%