Recurrent Neural Network
A class of neural network architectures that process sequential data through recursive updates of a hidden state. They are adept at capturing contextual dependencies along the temporal dimension and are widely used in sequence modeling tasks.
Timeline (last 90 days)
In recurrent network auditing, if H_t is the hidden state at time t and X_t is the current input, then I(H_t; X_t) characterizes the transient component, while I(H_t; θ) or I(H_t; training distribution) characterizes the durable component carried by the weights.
In recurrent networks, the catastrophic forgetting problem is amplified: because hidden states are recursively coupled over time, a small change in the weights may be amplified exponentially through multi-step propagation.
The study finds that the output geometries of Transformers, state space models, and recurrent neural networks are aligned to a far greater degree than their activation geometries.
In recurrent neural networks, the critical state corresponds to the maximum singular value (spectral radius) of the Jacobian matrix approaching 1.
A recurrent neural network containing 417,000 parameters is capable of autonomously generating the complete 6,500-frame Bad Apple video from a single initial state.
All Facts (5)
In recurrent network auditing, if H_t is the hidden state at time t and X_t is the current input, then I(H_t; X_t) characterizes the transient component, while I(H_t; θ) or I(H_t; training distribution) characterizes the durable component carried by the weights.
50%UnverifiedIn recurrent networks, the catastrophic forgetting problem is amplified: because hidden states are recursively coupled over time, a small change in the weights may be amplified exponentially through multi-step propagation.
50%UnverifiedThe study finds that the output geometries of Transformers, state space models, and recurrent neural networks are aligned to a far greater degree than their activation geometries.
50%UnverifiedIn recurrent neural networks, the critical state corresponds to the maximum singular value (spectral radius) of the Jacobian matrix approaching 1.
50%UnverifiedA recurrent neural network containing 417,000 parameters is capable of autonomously generating the complete 6,500-frame Bad Apple video from a single initial state.
50%