Online Stochastic Gradient Descent
An optimization algorithm that computes gradients and updates parameters using one sample or a small batch at a time, continuously receiving new samples during training; its continuous-time limit can be described by an ordinary differential equation.
Timeline (last 90 days)
This theoretical study uses the teacher-student model to derive, in the limit as the feature dimension D tends to infinity, ordinary differential equations that precisely describe the learning dynamics of the student network.
In the teacher-student model, the continuous-time limit of online stochastic gradient descent (online SGD) as the learning rate tends to zero is described by a set of ordinary differential equations.
All Facts (2)
This theoretical study uses the teacher-student model to derive, in the limit as the feature dimension D tends to infinity, ordinary differential equations that precisely describe the learning dynamics of the student network.
50%UnverifiedIn the teacher-student model, the continuous-time limit of online stochastic gradient descent (online SGD) as the learning rate tends to zero is described by a set of ordinary differential equations.
50%