MESSENGER: A New Memory-Enhanced Framework for Sequential Scene Flow Estimation

The Bottleneck in Scene Flow Estimation: From Two Frames to Long Sequences

Scene flow describes low-level 3D motion displacement in dynamic scenes and serves as a critical foundation for tasks like autonomous driving and robotic perception. However, traditional methods in this area have long been constrained by two core challenges.

Early pairwise estimators rely solely on instantaneous two-frame motion information, lacking long-term temporal context. This severely limits their ability to extrapolate future motion — in other words, these models can see the present but struggle to reason about what comes next.

Recent studies have explored multi-frame scene flow estimation using a sequence-to-sequence approach, but this introduces new problems: computational costs grow sharply as the number of input frames increases, and performance degrades noticeably on long-horizon predictions due to ineffective motion propagation mechanisms. Extracting long-term temporal dependencies without sacrificing efficiency has become the key challenge this field needs to overcome.

Scene flow can be understood as the 3D extension of optical flow: while optical flow describes 2D motion vectors of pixels on an image plane, scene flow assigns a 3D motion vector (dx, dy, dz) to each point in a point cloud or 3D scene. In autonomous driving, LiDAR-captured point clouds are the primary input for scene flow estimation. By estimating the 3D displacement of each point between consecutive frames, models can distinguish static backgrounds from dynamic objects (pedestrians, vehicles, etc.) and provide motion priors for downstream tasks like object tracking and trajectory prediction. The 3D End-Point Error (EPE3D) is the core metric for evaluating scene flow quality, defined as the mean Euclidean distance between predicted and ground-truth displacement vectors (typically in meters) — lower values indicate better accuracy.

MESSENGER's Core Design: Memory Buffer + Autoregressive Prediction

A recent arXiv paper proposes MESSENGER (Memory-Enhanced Sequential Scene Flow Estimation) as a new solution — a pipeline that combines memory augmentation with autoregressive next-frame prediction to break through existing limitations.

Explicit Memory Buffer Mechanism

The first key component is an explicit memory buffer that stores multiple historical flow estimates and latent states. For each input frame, the system retrieves and associates temporally stored flows and states to predict an initialization flow for the current frame via next-frame forecasting.

This design exploits the naturally occurring long-term temporal dependencies in continuous sequences, rather than naively stacking input frames. It enhances the model's understanding of motion evolution while maintaining computational efficiency.

Uncertainty-Aware Reweighting

A well-known risk of long-horizon autoregressive prediction is error accumulation — small deviations at each step compound over time. To address this, MESSENGER introduces an uncertainty-aware reweighting module that filters out unreliable retrieval results and suppresses the propagation of accumulated errors.

In essence, this module assigns a "confidence score" to each memory retrieval result, steering the network to rely more on high-confidence historical information and improving robustness in long-sequence prediction.

Error accumulation is a classic challenge in autoregressive models with a long history in sequential prediction. Early RNNs and later Transformers face similar issues in long-sequence generation: models are trained with ground-truth history as input (teacher forcing), but at inference time must rely on their own previous outputs. Once a prediction goes slightly off, subsequent steps build on that error, causing deviations to grow exponentially over time. In tasks like scene flow estimation, where physical quantities are being measured, error accumulation is particularly dangerous — small offsets in motion vectors can snowball into completely wrong trajectory predictions after just a few frames. MESSENGER's uncertainty-aware reweighting module addresses this by estimating a reliability score for each historical memory entry and down-weighting low-confidence entries during aggregation, effectively cutting off the error propagation chain.

Performance Gains from the Autoregressive Paradigm

The core philosophy of MESSENGER lies in its autoregressive forecasting paradigm. By predicting frame by frame, the network is naturally guided to progressively learn the distribution of the next frame based on historical observations, rather than outputting an entire sequence in one shot. This incremental learning approach aligns well with the physical continuity of motion.

Experiments on two mainstream autonomous driving datasets — nuScenes and Argoverse 2 — validate the effectiveness of this approach. On long-horizon future extrapolation tasks, MESSENGER reduces the 3D End-Point Error (EPE3D) by 71.6% on nuScenes and 67.7% on Argoverse 2, achieving state-of-the-art performance.

These margins are substantial by scene flow estimation standards, particularly in the long-horizon prediction setting that most rigorously tests a model's extrapolation ability. The results suggest that the combined design of memory retrieval and autoregressive prediction genuinely addresses the root of the problem.

nuScenes and Argoverse 2 are two of the most representative public benchmark datasets in autonomous driving perception. nuScenes, released by Motional (formerly nuTonomy), includes LiDAR and multi-camera data from Boston and Singapore, and is known for its complex urban scenarios and diverse weather conditions. Argoverse 2, released by Argo AI, provides high-definition map-fused LiDAR sequences; its scene flow subtask requires models to handle fine-grained motion estimation on dense point clouds. The two datasets differ in point cloud density, scene complexity, and annotation granularity. Achieving large gains on both simultaneously demonstrates that MESSENGER's design generalizes across dataset distributions rather than overfitting to any single one.

Significance and Outlook

MESSENGER's value lies not only in its benchmark improvements, but also in providing a general framework for long-sequence motion prediction: replace heavy multi-frame stacking with explicit memory, replace one-shot sequence output with autoregressive generation, and use uncertainty mechanisms to combat error accumulation. This three-part combination offers meaningful insights for dynamic scene perception tasks more broadly.

The authors have indicated that code will be open-sourced on GitHub at github.com/liujiuming123/Messenger, making it convenient for future researchers to reproduce and extend the method. For researchers and engineers focused on autonomous driving perception and 3D motion understanding, MESSENGER is well worth following.

It should be noted that this article is based on the paper's abstract. For more details on network architecture, training strategies, and comparisons with specific baselines, please refer to the full paper and open-source code.