100x Faster RNN Training: Parallel-in-Time Methods Tackle Chaotic System Reconstruction

A Long-Standing Bottleneck in RNN Training

Recurrent neural networks (RNNs) have a natural advantage when working with dynamical systems — they can capture nonlinear dependencies in time series data. But a persistent problem remains: RNN forward passes are inherently sequential, with each step depending on the output of the previous one. When faced with ultra-long time series from chaotic systems (T > 10^6 steps), this serial nature makes training extremely slow, with computational complexity scaling linearly with sequence length T, i.e., O[T].

This is precisely the core problem addressed in a paper selected as a NeurIPS 2026 Spotlight. The paper, Parallel-in-Time Training of Recurrent Neural Networks for Dynamical Systems Reconstruction, proposes a combined approach that claims to speed up nonlinear RNN training on chaotic dynamical system time series by more than two orders of magnitude (>100x).

The Core Idea: Combining DEER and GTF

The key insight of this work is combining two existing techniques to overcome limitations that neither can solve alone.

DEER: Turning Sequential Forward Passes into Parallel Solving

Rather than executing the RNN forward pass step by step, DEER reformulates the entire sequence computation (of length T) as a large system of equations, solved all at once via Newton-type fixed point iterations. This fully leverages the GPU's parallel computing capabilities, theoretically reducing complexity from O[T] to O[(log T)²].

The longer the sequence, the more pronounced the relative gains — for dynamical system data spanning millions of steps, this improvement is nearly transformative.

DEER (Deep Equilibrium Recurrent) borrows from the ideas of implicit deep learning: treating the RNN's temporal unrolling as an implicit equation h = F(h, x), where h is the hidden state vector across the entire sequence and x is the input. Newton-type iterations approximate the inverse of the Jacobian matrix to progressively refine the solution, rather than recursing through time steps one by one. This allows all hidden states across the sequence to be updated simultaneously in a single iteration — perfectly suited for large-scale parallel computation on GPUs. The theoretical O[(log T)²] complexity arises because Newton iteration's quadratic convergence requires roughly O(log T) iterations, and each iteration involves a nested layer of parallel operations, giving rise to the squared logarithm term. This complexity advantage grows dramatically as sequence length scales from thousands to millions of steps.

Chaotic Dynamics Break DEER

The problem is that DEER breaks down in the presence of chaotic systems. The very nature of chaotic dynamics — extreme sensitivity to initial conditions — means that small errors get amplified exponentially, causing DEER's fixed-point iterations to fail to converge. Runtime degrades to O[T log T], and the parallel advantage largely disappears. This is why naive parallel methods cannot be directly applied to dynamical systems reconstruction (DSR) tasks.

GTF: The Key to Stabilizing Training

The research team introduced Generalized Teacher Forcing (GTF) to stabilize DEER. GTF itself is a training technique previously developed for dynamical systems reconstruction, and it serves two purposes:

First, it prevents the divergence caused by chaotic dynamics, making DEER's iterative process controllable and convergent again. Second, compared to traditional teacher forcing, GTF reduces exposure bias — the distribution mismatch that arises when a model relies on ground-truth data during training but must rely on its own predictions at inference time. This issue is especially pronounced in the training of state space models (SSMs).

Combining DEER's parallelism with GTF's stabilization is what truly enables "parallel-in-time" training that is both fast and stable, allowing the model to efficiently converge on extremely long chaotic time series.

Traditional teacher forcing directly feeds ground-truth observations as the input at each training step. While this accelerates convergence, it introduces exposure bias: the model never learns to recover from its own prediction errors during training, causing error accumulation to spiral during inference. GTF uses a tunable mixing coefficient to blend ground-truth observations and the model's own predictions as the input to each subsequent step — leaning toward real data early in training to stabilize gradients, then gradually increasing the weight of model-generated predictions as training progresses. In the context of chaotic systems, GTF additionally acts as an attractor regularizer: by periodically "pulling" the hidden state back toward the true trajectory, it suppresses the exponential error amplification inherent to chaotic sensitivity. This is the fundamental mechanism that restores convergence to DEER's fixed-point iterations.

Comparison Against Mamba and Other State Space Models

In recent years, state space models (SSMs) — with Mamba as the leading example — have attracted significant research attention for their efficient handling of long sequences. However, in the specific setting of dynamical systems reconstruction, the paper claims its method substantially outperforms Mamba and other SSMs.

This finding is worth noting. It suggests that when pursuing long-sequence modeling capability, not all architectures handle highly nonlinear, sensitivity-prone processes like chaotic systems equally well. Training strategies specifically designed for dynamical reconstruction may prove more valuable in domains such as physics modeling, climate simulation, and neuroscience — wherever faithfully reproducing real dynamical systems is the goal.

Mamba's core innovation lies in its Selective State Space mechanism (Selective SSM): by making state transition matrices and selection gates dynamically dependent on input content, it escapes the limitations of traditional linear time-invariant SSMs, while using a hardware-aware parallel scan algorithm to achieve efficient computation along the sequence dimension. This makes Mamba excel at long-sequence tasks in language and audio. However, SSM state transitions are fundamentally linear recurrent structures, creating an inherent expressiveness bottleneck for reconstructing highly nonlinear chaotic attractors — the trajectory bifurcations and strange attractors characteristic of chaotic systems require strongly nonlinear dynamics to be accurately captured. This explains why a parallel training scheme designed specifically for nonlinear RNNs can outperform general-purpose linear SSM architectures in the vertical domain of dynamical reconstruction.

Why This Research Matters

The goal of dynamical systems reconstruction is to recover the underlying dynamical laws from observed time series. This class of tasks has practical relevance in weather forecasting, fluid mechanics, biological neural activity analysis, and more — and the systems involved are often both chaotic and data-intensive, requiring processing of massive long sequences.

In the past, training efficiency has been the primary barrier to applying RNNs to these problems. A speedup of more than 100x means that training that once took days could be completed in hours, enabling researchers to iterate experiments on longer, more realistic datasets. Whether the data comes from simulated or real-world systems, this efficiency gain opens the door to dynamical modeling at much larger scales.

Of course, as a preprint result (arXiv:2605.12683), the reported performance still awaits broader reproduction and validation. But from a methodological standpoint, it offers a clear lesson: when tackling specific problems, combining targeted techniques is often more effective than simply stacking general-purpose architectures.