Teacher-Student Model
A framework commonly used in theoretical analysis in which a teacher network provides the true rules for data generation and a student network, trained from random initialization, approximates the teacher's outputs, enabling precise study of learning dynamics.
Timeline (last 90 days)
In the teacher-student model, when there is no correlation between input positions (c=0), the convolutional student and the locally connected student follow the same learning trajectory, but the convolutional student learns exactly P times as fast as the locally connected student (P being the number of input positions).
This theoretical study uses the teacher-student model to derive, in the limit as the feature dimension D tends to infinity, ordinary differential equations that precisely describe the learning dynamics of the student network.
In the teacher-student model, the continuous-time limit of online stochastic gradient descent (online SGD) as the learning rate tends to zero is described by a set of ordinary differential equations.
The teacher-student model is a framework commonly used in theoretical analysis: the teacher network is a fixed reference network, and the student network starts from random initialization and tries through training to approximate the teacher's outputs or parameters.
All Facts (4)
In the teacher-student model, when there is no correlation between input positions (c=0), the convolutional student and the locally connected student follow the same learning trajectory, but the convolutional student learns exactly P times as fast as the locally connected student (P being the number of input positions).
50%UnverifiedThis theoretical study uses the teacher-student model to derive, in the limit as the feature dimension D tends to infinity, ordinary differential equations that precisely describe the learning dynamics of the student network.
50%UnverifiedIn the teacher-student model, the continuous-time limit of online stochastic gradient descent (online SGD) as the learning rate tends to zero is described by a set of ordinary differential equations.
50%UnverifiedThe teacher-student model is a framework commonly used in theoretical analysis: the teacher network is a fixed reference network, and the student network starts from random initialization and tries through training to approximate the teacher's outputs or parameters.
50%