K-FAC
An approximate second-order optimization method based on a Kronecker-factored approximation of the Fisher information matrix. It compresses the otherwise O(n²) curvature matrix into a product of two smaller matrices, enabling convergence in fewer iterations than first-order methods under large-batch training conditions.
Timeline (last 90 days)
K-FAC utilizes the Kronecker decomposition structure of the Fisher information matrix to compress the originally O(n²) curvature matrix into the product of two smaller matrices.
The fact that dedicated stiff ODE solvers have not become mainstream training methods is not because they have been disproven, but because their core benefits have been absorbed by more engineering-friendly second-order optimizers such as K-FAC and Shampoo.
All Facts (2)
K-FAC utilizes the Kronecker decomposition structure of the Fisher information matrix to compress the originally O(n²) curvature matrix into the product of two smaller matrices.
50%UnverifiedThe fact that dedicated stiff ODE solvers have not become mainstream training methods is not because they have been disproven, but because their core benefits have been absorbed by more engineering-friendly second-order optimizers such as K-FAC and Shampoo.
50%