Shampoo
A second-order optimization algorithm based on matrix preconditioning that maintains row and column covariance matrices for each layer's weight matrix and performs a Kronecker factorization, enabling optimization that more closely approximates full second-order information.
Timeline (last 90 days)
SOAP, Shampoo and FOAM are names of optimization algorithms in the deep learning field.
Shampoo is an optimizer based on matrix preconditioning.
Shampoo maintains covariance matrices along the row and column directions for each layer's weight matrix separately and performs Kronecker factorization.
SOAP can be seen as combining adaptive ideas similar to Adagrad on top of the Shampoo preconditioning framework.
Shampoo maintains second-order statistics separately for each dimension of each parameter tensor, avoiding the computation of the full Hessian.
The fact that dedicated stiff ODE solvers have not become mainstream training methods is not because they have been disproven, but because their core benefits have been absorbed by more engineering-friendly second-order optimizers such as K-FAC and Shampoo.
All Facts (6)
SOAP, Shampoo and FOAM are names of optimization algorithms in the deep learning field.
50%UnverifiedShampoo is an optimizer based on matrix preconditioning.
50%UnverifiedShampoo maintains covariance matrices along the row and column directions for each layer's weight matrix separately and performs Kronecker factorization.
50%UnverifiedSOAP can be seen as combining adaptive ideas similar to Adagrad on top of the Shampoo preconditioning framework.
50%UnverifiedShampoo maintains second-order statistics separately for each dimension of each parameter tensor, avoiding the computation of the full Hessian.
50%UnverifiedThe fact that dedicated stiff ODE solvers have not become mainstream training methods is not because they have been disproven, but because their core benefits have been absorbed by more engineering-friendly second-order optimizers such as K-FAC and Shampoo.
50%