Random Matrix Theory
A theoretical framework in mathematical physics and probability theory that studies the spectral properties of large random matrices. In machine learning theory, it is applied to derive precise asymptotic analyses of the generalization error of neural networks.
Timeline (last 90 days)
The research introduces the high-temperature Marchenko-Pastur (HTMP) distribution assumption to characterize heavy-tailed properties.
This study is one of the first attempts to systematically apply random matrix theory to provide a rigorous statistical foundation for empirical metrics of DNN weight spectra.
The Marchenko-Pastur law describes that when the elements of an m×n matrix are independently and identically distributed, the eigenvalue density of its sample covariance matrix converges to a deterministic distribution in the high-dimensional limit.
The paper uses random matrix theory to derive deterministic equivalents of the population error for an optimally trained teacher model and a student model trained via gradient flow
Deterministic equivalents are a core tool of random matrix theory: when the dimension p and sample size n grow to infinity proportionally (p/n → γ), random quantities converge in probability to deterministic expressions.
Marčenko and Pastur established the asymptotic theory for the eigenvalue distribution of sample covariance matrices in 1967, while random matrix theory was originally developed by Eugene Wigner in the 1950s.
The Marčenko-Pastur law describes the distribution of noise eigenvalues from the perspective of random matrix theory
All Facts (7)
The research introduces the high-temperature Marchenko-Pastur (HTMP) distribution assumption to characterize heavy-tailed properties.
50%UnverifiedThis study is one of the first attempts to systematically apply random matrix theory to provide a rigorous statistical foundation for empirical metrics of DNN weight spectra.
50%UnverifiedThe Marchenko-Pastur law describes that when the elements of an m×n matrix are independently and identically distributed, the eigenvalue density of its sample covariance matrix converges to a deterministic distribution in the high-dimensional limit.
50%UnverifiedThe paper uses random matrix theory to derive deterministic equivalents of the population error for an optimally trained teacher model and a student model trained via gradient flow
50%UnverifiedDeterministic equivalents are a core tool of random matrix theory: when the dimension p and sample size n grow to infinity proportionally (p/n → γ), random quantities converge in probability to deterministic expressions.
50%UnverifiedMarčenko and Pastur established the asymptotic theory for the eigenvalue distribution of sample covariance matrices in 1967, while random matrix theory was originally developed by Eugene Wigner in the 1950s.
50%UnverifiedThe Marčenko-Pastur law describes the distribution of noise eigenvalues from the perspective of random matrix theory
50%