Generalized Gauss-Newton Approximation
A method for approximating the curvature (Hessian matrix) of a neural network loss function by using the outer product of the Jacobian matrix to approximate the Hessian, substantially reducing computational complexity.
Timeline (last 90 days)
To verify the transferability of the theory to neural network models, the researchers use a counterfactual proxy to estimate the influence of spurious-correlation dependence and use the generalized Gauss-Newton approximation to estimate the curvature of the language model.
The generalized Gauss-Newton approximation is a method for approximating the curvature (Hessian matrix) of a neural network loss function, using the outer product of Jacobian matrices to approximate the Hessian, thereby estimating curvature at acceptable computational cost.
All Facts (2)
To verify the transferability of the theory to neural network models, the researchers use a counterfactual proxy to estimate the influence of spurious-correlation dependence and use the generalized Gauss-Newton approximation to estimate the curvature of the language model.
50%UnverifiedThe generalized Gauss-Newton approximation is a method for approximating the curvature (Hessian matrix) of a neural network loss function, using the outer product of Jacobian matrices to approximate the Hessian, thereby estimating curvature at acceptable computational cost.
50%