State Transition Jacobian Matrix
In an RNN, the matrix of partial derivatives describing the sensitivity of the hidden state to the hidden state at the previous time step. The exponential decay behavior of its spectral norm or singular values affects gradient propagation.
Timeline (last 90 days)
Under the assumption that the state transition Jacobian decays exponentially, the error between the truncated gradient and the full-context gradient decays exponentially as the distance from the context boundary increases.
The state-transition Jacobian matrix characterizes the sensitivity of the RNN hidden state to the hidden state at the previous time step, and the norm behavior of its product determines whether gradients can be effectively propagated to distant points.
The choice of truncation window should be designed in conjunction with the decay rate of the state-transition Jacobian matrix: when the Jacobian decays quickly, a shorter window suffices to approach the full-context gradient, whereas otherwise a longer window is needed.
All Facts (3)
Under the assumption that the state transition Jacobian decays exponentially, the error between the truncated gradient and the full-context gradient decays exponentially as the distance from the context boundary increases.
50%UnverifiedThe state-transition Jacobian matrix characterizes the sensitivity of the RNN hidden state to the hidden state at the previous time step, and the norm behavior of its product determines whether gradients can be effectively propagated to distant points.
50%UnverifiedThe choice of truncation window should be designed in conjunction with the decay rate of the state-transition Jacobian matrix: when the Jacobian decays quickly, a shorter window suffices to approach the full-context gradient, whereas otherwise a longer window is needed.
50%