Population Gradient Flow
An idealized training dynamics model commonly used in theoretical analysis. It assumes an infinite sample size and replaces discrete gradient descent with continuous-time gradient flow, which facilitates analysis of convergence and trajectories using dynamical systems theory.
Timeline (last 90 days)
The paper's analysis is based on idealized assumptions such as population gradient flow and isotropic Gaussian inputs.
Population gradient flow assumes an infinitely large sample size, with the loss function being the expected loss over the true data distribution, thereby replacing discrete gradient descent with continuous-time gradient flow.
The multi-stage symmetry breaking in this model is attributed to the combined effect of standard algorithmic and architectural choices (such as small initialization, ReLU activation, and population gradient flow).
All Facts (3)
The paper's analysis is based on idealized assumptions such as population gradient flow and isotropic Gaussian inputs.
50%UnverifiedPopulation gradient flow assumes an infinitely large sample size, with the loss function being the expected loss over the true data distribution, thereby replacing discrete gradient descent with continuous-time gradient flow.
50%UnverifiedThe multi-stage symmetry breaking in this model is attributed to the combined effect of standard algorithmic and architectural choices (such as small initialization, ReLU activation, and population gradient flow).
50%