Weight Sharing
A mechanism in neural networks in which the same set of parameters is reused at multiple positions; it is a key design of convolutional networks for reducing parameter counts and improving statistical efficiency.
Timeline (last 90 days)
Weight sharing in LoopRec keeps the total number of parameters from growing with maximum depth.
Spatial correlation reduces the advantage of weight sharing: the asymptotic error of the convolutional student rises as the input spatial correlation c increases, whereas the asymptotic error of the locally connected student is independent of c and is overall lower at the same learning rate.
When the learning rate is adjusted according to training duration, the locally connected student performs better in the early stage while the convolutional student wins in the later stage, with an analytically predictable crossover point between the two.
In the teacher-student model, when there is no correlation between input positions (c=0), the convolutional student and the locally connected student follow the same learning trajectory, but the convolutional student learns exactly P times as fast as the locally connected student (P being the number of input positions).
Locality and weight sharing do not always synergize: when inputs are highly correlated, weight sharing may harm performance, and the benefit of locality depends on how well the kernel size matches the task structure.
When designing network architectures, one should adjust the weight sharing strategy according to data correlation and choose the convolution kernel size carefully.
Under correlated inputs, forcing weight sharing may introduce bias, causing the convolutional student to fail to fully recover the teacher parameters even with infinite data, whereas the locally connected network, by not sharing weights, can adapt more flexibly to the data.
All Facts (7)
Weight sharing in LoopRec keeps the total number of parameters from growing with maximum depth.
50%UnverifiedSpatial correlation reduces the advantage of weight sharing: the asymptotic error of the convolutional student rises as the input spatial correlation c increases, whereas the asymptotic error of the locally connected student is independent of c and is overall lower at the same learning rate.
50%UnverifiedWhen the learning rate is adjusted according to training duration, the locally connected student performs better in the early stage while the convolutional student wins in the later stage, with an analytically predictable crossover point between the two.
50%UnverifiedIn the teacher-student model, when there is no correlation between input positions (c=0), the convolutional student and the locally connected student follow the same learning trajectory, but the convolutional student learns exactly P times as fast as the locally connected student (P being the number of input positions).
50%UnverifiedLocality and weight sharing do not always synergize: when inputs are highly correlated, weight sharing may harm performance, and the benefit of locality depends on how well the kernel size matches the task structure.
50%UnverifiedWhen designing network architectures, one should adjust the weight sharing strategy according to data correlation and choose the convolution kernel size carefully.
50%UnverifiedUnder correlated inputs, forcing weight sharing may introduce bias, causing the convolutional student to fail to fully recover the teacher parameters even with infinite data, whereas the locally connected network, by not sharing weights, can adapt more flexibly to the data.
50%