Second-Order Approximation
A method that uses the Hessian or Fisher information matrix of the loss function to locally characterize the change in loss caused by parameter perturbations; compared with first-order gradients, it reflects the local curvature of the loss surface.
Timeline (last 90 days)
Under a second-order approximation, LIRA estimates the magnitude of local loss perturbation caused by recent LoRA factor movements, using this as the score for each layer.
Whether the estimation quality of the second-order approximation remains stable during phases of drastic training change is a direction that future work can explore in depth.
All Facts (2)
Under a second-order approximation, LIRA estimates the magnitude of local loss perturbation caused by recent LoRA factor movements, using this as the score for each layer.
50%UnverifiedWhether the estimation quality of the second-order approximation remains stable during phases of drastic training change is a direction that future work can explore in depth.
50%