First-Order Analysis
A mathematical analysis tool that applies a Taylor expansion to the effect of weight perturbations on model outputs and retains only the first-order linear term. It is used to explain that small weight perturbations are equivalent to logit shifts.
Timeline (last 90 days)
A first-order analysis shows that small weight perturbations act mainly through the logit shifts they induce.
The basic idea of the first-order analysis is to Taylor-expand the effect of weight perturbation on model output and keep only the first-order linear term: the output change is approximately the inner product of the perturbation vector and the gradient of the output with respect to the weights.
All Facts (2)
A first-order analysis shows that small weight perturbations act mainly through the logit shifts they induce.
50%UnverifiedThe basic idea of the first-order analysis is to Taylor-expand the effect of weight perturbation on model output and keep only the first-order linear term: the output change is approximately the inner product of the perturbation vector and the gradient of the output with respect to the weights.
50%