[KongchangAI]
ConceptSpectral Normalization

Spectral Normalization

A technique that scales a matrix by its largest singular value. It is used for numerical stability and regularization, and is commonly applied in optimizers and generative models. By constraining the spectral norm, the technique limits the magnitude of the matrix's effect and thereby controls the Lipschitz constant of the associated transformation.

Timeline (last 90 days)

Nov 2

Muon-RT's processing pipeline is: remove the radial component from the teacher-student displacement, perform spectral normalization, and then mix it with the current gradient after Newton-Schulz orthogonalization.

Unverified50%
Aug 24

Miyato等人提出的谱归一化通过将网络每层权重矩阵除以其最大奇异值,将每层Lipschitz常数控制在1

Unverified50%

All Facts (2)

Source Articles