BPTT
The standard algorithm for training recurrent neural networks, which computes gradients by unrolling the computational graph along the time axis. Longer sequences increase the risk of vanishing or exploding gradients.
Timeline (last 90 days)
The computational graph and memory overhead of standard BPTT grow linearly with sequence length.
In backpropagation through time (BPTT), the longer the sequence, the more easily the repeated multiplication of gradients along the time dimension leads to explosion or vanishing, whereas independent windows cut off such long-range dependencies and instead improve robustness in cross-user scenarios.
Recurrent-based models are prone to hidden state collapse and gradient instability when trained via backpropagation through time (BPTT).
All Facts (3)
The computational graph and memory overhead of standard BPTT grow linearly with sequence length.
50%UnverifiedIn backpropagation through time (BPTT), the longer the sequence, the more easily the repeated multiplication of gradients along the time dimension leads to explosion or vanishing, whereas independent windows cut off such long-range dependencies and instead improve robustness in cross-user scenarios.
50%UnverifiedRecurrent-based models are prone to hidden state collapse and gradient instability when trained via backpropagation through time (BPTT).
50%