Edge of Stability
A phenomenon in gradient descent training with large step sizes: when the step size exceeds the classical stability threshold, the loss does not diverge, and sharpness fluctuates near the critical value of 2 divided by the step size while convergence still occurs.
Timeline (last 90 days)
Edge stability is a phenomenon of large-step-size gradient descent training discovered in recent years: when the step size exceeds the classical stability threshold, the loss does not diverge; instead, sharpness fluctuates around a critical value and the loss still converges.
The edge stability phenomenon has been observed in both fully connected networks and Transformers under standard training.
In adversarial training, loss sharpness can exhibit two-stage dynamics, differing from the edge stability behavior observed in standard training.
All Facts (3)
Edge stability is a phenomenon of large-step-size gradient descent training discovered in recent years: when the step size exceeds the classical stability threshold, the loss does not diverge; instead, sharpness fluctuates around a critical value and the loss still converges.
50%UnverifiedThe edge stability phenomenon has been observed in both fully connected networks and Transformers under standard training.
50%UnverifiedIn adversarial training, loss sharpness can exhibit two-stage dynamics, differing from the edge stability behavior observed in standard training.
50%