Shampoo maintains covariance matrices along the row and column directions for each layer's weight matrix separately and performs Kronecker factorization.