Fisher Information Matrix
A central concept in statistics and information geometry that measures the sensitivity of a probability distribution to changes in its parameters. In natural gradient descent it serves as a preconditioning matrix. Computing the full matrix has complexity O(p²).
Timeline (last 90 days)
The Fisher-Rao metric is a Riemannian metric on the space of probability distributions that measures the statistical distance between two probability distributions via the Fisher information matrix.
K-FAC utilizes the Kronecker decomposition structure of the Fisher information matrix to compress the originally O(n²) curvature matrix into the product of two smaller matrices.
The Fisher information matrix measures the sensitivity of model parameters to the output distribution; the higher a parameter's Fisher value, the greater the impact of changing it on the output.
The computational complexity of the full Fisher information matrix is O(p^2), where p is the number of parameters, which is infeasible for modern deep networks.
All Facts (4)
The Fisher-Rao metric is a Riemannian metric on the space of probability distributions that measures the statistical distance between two probability distributions via the Fisher information matrix.
50%UnverifiedK-FAC utilizes the Kronecker decomposition structure of the Fisher information matrix to compress the originally O(n²) curvature matrix into the product of two smaller matrices.
50%UnverifiedThe Fisher information matrix measures the sensitivity of model parameters to the output distribution; the higher a parameter's Fisher value, the greater the impact of changing it on the output.
50%UnverifiedThe computational complexity of the full Fisher information matrix is O(p^2), where p is the number of parameters, which is infeasible for modern deep networks.
50%