Reverse-Mode Automatic Differentiation
An implementation mode of automatic differentiation that first builds a computation graph during the forward pass and then traverses it in reverse, applying the chain rule to compute gradients; it is the gradient computation method adopted by default in mainstream deep learning frameworks.
Timeline (last 90 days)
Early work on reverse-mode automatic differentiation can be traced back to Linnainmaa (1970), and Paul Werbos applied it to multilayer networks in 1982.
Reverse-mode automatic differentiation can compute the gradients of a scalar loss with respect to all parameters in O(n) time, whereas forward mode requires O(n) forward passes
All Facts (2)
Early work on reverse-mode automatic differentiation can be traced back to Linnainmaa (1970), and Paul Werbos applied it to multilayer networks in 1982.
50%UnverifiedReverse-mode automatic differentiation can compute the gradients of a scalar loss with respect to all parameters in O(n) time, whereas forward mode requires O(n) forward passes
50%