The Independent Roles of Locality and Weight Sharing in Convolutional Neural Networks

The success of Convolutional Neural Networks (CNNs) is often attributed to locality and weight sharing, yet how these two properties individually shape the learning process and how their effects depend on the spatial correlation of natural images remain poorly understood. A recent theoretical study uses a teacher-student model to precisely analyze the roles of these two properties and reveals how spatial correlation affects the advantage of weight sharing.

Research Background and Model Setup

Past explanations have broadly attributed CNN's superior performance to locality and weight sharing without disentangling their individual contributions. This study constructs a teacher-student framework: the teacher network is a convolutional network operating over P input positions with correlation c between positions; the student network adopts either a convolutional or a locally connected architecture and is trained via online stochastic gradient descent. In the limit where the feature dimension D tends to infinity, the authors derive ordinary differential equations that precisely describe the student network's learning dynamics.

This theoretical setup allows researchers to independently control locality (via convolutional kernel size) and weight sharing (via whether parameters are shared across positions), thereby observing their effects on the learning trajectory, sample complexity, and generalization error.

The teacher-student model is a framework commonly used in theoretical analysis: the teacher network is a fixed (or pre-trained) reference network, and the student network starts from random initialization and attempts through training to approximate the teacher's outputs or parameters. The teacher network provides the "true" rule for data generation, and the student network's learning process can be tracked precisely. The advantage of this framework lies in its ability to precisely control the data distribution and network structure, thereby enabling analytical study of learning dynamics. In this paper, the teacher network is a convolutional network with fixed parameters used to generate labeled data; the student network adopts different structures (convolutional or locally connected) and is trained via online stochastic gradient descent. In the limit where the feature dimension D tends to infinity, the learning process can be precisely described by a set of ordinary differential equations, providing a mathematical tool for analyzing the roles of locality and weight sharing.

Online stochastic gradient descent (online SGD) is an optimization algorithm that computes gradients and updates parameters using only one (or a small batch of) sample at a time. Unlike batch gradient descent, online SGD continuously receives new samples during training, and its learning dynamics can be approximated by stochastic differential equations. In the teacher-student model, the student network learns sample by sample from a data stream, and the continuous-time limit of online SGD (as the learning rate tends to zero) is described by a set of ordinary differential equations, allowing researchers to analytically analyze the learning trajectory, convergence speed, and generalization error. This paper leverages precisely this limit to derive exact equations for the evolution of the student network's parameters over training time, and thereby compares the learning behavior of different structures.

Equivalence and Acceleration Under Uncorrelated Inputs

When there is no correlation between input positions (c=0), the convolutional student and the locally connected student follow the same learning trajectory, but the convolutional student learns exactly P times faster than the locally connected student. This result intuitively reflects the statistical efficiency gain brought by weight sharing: due to parameter sharing, the convolutional network effectively learns at P positions simultaneously, thereby accelerating the gradient descent process.

The study also finds that locality affects sample complexity. When the student network's kernel size matches the teacher's, only O_D(D) samples are needed to recover the teacher's parameters; if the kernel size is larger, O_D(D log D) samples are required. This means that an excessively large kernel size increases sample requirements, and the locality design must match the task.

Sample complexity refers to the number of training samples required for a learning algorithm to achieve a given generalization error. In the teacher-student model, sample complexity is typically expressed as a function of the feature dimension D. This paper finds that when the student network's kernel size matches the teacher's, only O(D) samples are needed to recover the teacher's parameters; if the kernel size is larger, O(D log D) samples are required. The difference between O(D) and O(D log D) stems from the statistical efficiency of parameter estimation: an excessively large kernel size requires estimating more parameters, thereby increasing sample requirements. This result indicates that the locality design (i.e., kernel size) should match the intrinsic structure of the task; otherwise, it incurs additional sample overhead.

Spatial Correlation Weakens the Advantage of Weight Sharing

In natural images, adjacent pixels are often highly correlated (c>0), and this correlation alters the above conclusions. The study shows that spatial correlation reduces the advantage of weight sharing. Specifically, the asymptotic error of the convolutional student increases as c grows, while the asymptotic error of the locally connected student is independent of c and overall lower at the same learning rate.

This finding challenges the intuition that "weight sharing is always beneficial." Under correlated inputs, forcibly sharing weights may introduce bias, whereas the locally connected network, by not sharing weights, can adapt more flexibly to the data.

Asymptotic error refers to the generalization error to which the student network converges as the number of training samples tends to infinity. In the teacher-student model, asymptotic error reflects the student network's optimal performance under infinite data and is typically determined by model misspecification and optimization bias. In this paper, the asymptotic error of the convolutional student increases as the input spatial correlation c grows, while the asymptotic error of the locally connected student is independent of c. This is because weight sharing forces different positions to use the same parameters; when input positions are highly correlated, this constraint introduces bias, making it impossible to fully recover the teacher's parameters even with infinite data. The locally connected network, by contrast, does not share weights and can adapt more flexibly to the correlation structure, resulting in lower asymptotic error. This concept reveals the potential cost of weight sharing under correlated data.

Dynamic Transition Under Learning Rate Scheduling

The authors further analyze the relationship between learning rate and training time. When the learning rate is adjusted according to training duration, the locally connected student performs better in the early stage, while the convolutional student wins in the later stage, with an analytically predictable transition point between the two. This indicates that the advantage of weight sharing is not static but depends on the training progress: in the early stage of training, the locally connected network can capture local patterns more quickly; as training proceeds, the statistical efficiency advantage brought by weight sharing gradually emerges.

Implications for CNN Design and Understanding

This study provides a fine-grained decomposition of the factors behind CNN's success. Locality and weight sharing do not always synergize: when inputs are highly correlated, weight sharing may harm performance; and the benefit of locality depends on how well the kernel size matches the task structure. For practitioners, this means that when designing network architectures, one should adjust the weight sharing strategy according to data correlation and carefully choose the convolutional kernel size.

Although the study is based on a simplified teacher-student model, its analytical results provide testable hypotheses for understanding CNN's inductive biases and lay the groundwork for future research on these properties in more complex architectures.

One-line Summary

Theoretical analysis reveals the independent roles of locality and weight sharing in CNNs and the impact of spatial correlation.

Paragraph Summary

Using a teacher-student model, this study precisely analyzes how locality and weight sharing in convolutional neural networks affect the learning process. When input positions are uncorrelated, the convolutional student and the locally connected student follow the same learning trajectory, but the convolutional student is P times faster; locality affects sample complexity, with O(D) samples needed when the kernel size matches the teacher and O(D log D) for larger kernels. However, spatial correlation c>0 in natural images weakens the advantage of weight sharing: the convolutional student's asymptotic error increases with c, while the locally connected student's error is independent of c and overall lower. Under learning rate adjustment, the locally connected student dominates early and the convolutional student wins later, with a predictable transition point. The implication: weight sharing is not always beneficial, and when designing CNNs one should adjust the weight sharing strategy according to data correlation and carefully choose the convolutional kernel size.