Gradient Flow
A mathematical framework that interprets neural network training as a continuous dynamical system. It represents the limit of discrete gradient descent as the learning rate approaches zero in the form of the ordinary differential equation dθ/dt = −∇L(θ), providing a theoretical tool for analyzing training dynamics.
Timeline (last 90 days)
The paper uses random matrix theory to derive deterministic equivalents of the population error for an optimally trained teacher model and a student model trained via gradient flow
The paper analyzes two extended settings: more general stopping times and target functions supported on multiple spherical harmonic orders
Standard backpropagation is essentially a discrete iteration of gradient descent, and when the learning rate is regarded as an infinitesimally small time step, this discrete process converges to the gradient flow ordinary differential equation dθ/dt = −∇L(θ).
All Facts (3)
The paper uses random matrix theory to derive deterministic equivalents of the population error for an optimally trained teacher model and a student model trained via gradient flow
50%UnverifiedThe paper analyzes two extended settings: more general stopping times and target functions supported on multiple spherical harmonic orders
50%UnverifiedStandard backpropagation is essentially a discrete iteration of gradient descent, and when the learning rate is regarded as an infinitesimally small time step, this discrete process converges to the gradient flow ordinary differential equation dθ/dt = −∇L(θ).
50%