S2Tok: Streaming 3D Gaussian Reconstruction with Persistent Spatial Tokens

The Core Challenge of Streaming 3D Reconstruction

Traditional 3D reconstruction typically assumes a complete set of input images, allowing the model to see all observations at once before performing geometric inference. But in real-world online scenarios — such as robot navigation or real-time AR mapping — images arrive as a continuous stream. This creates a fundamental challenge: the system must not only produce a sequence of geometric predictions, but also maintain a persistent scene state that can be continuously updated with new evidence while remaining renderable at any moment.

The recent arXiv paper S2Tok: Streaming 3D Gaussian Reconstruction with Persistent Spatial Tokens directly addresses this pain point. The researchers note that each new observation may simultaneously do two things: revisit regions already seen (requiring updates to existing representations) and reveal entirely new content (requiring the representation to expand). Distinguishing and handling these two cases online is what fundamentally differentiates streaming reconstruction from offline reconstruction.

3D Gaussian Splatting (3DGS) is a real-time scene representation and rendering method that has gained significant traction in recent years. Scenes are modeled as tens of thousands of 3D Gaussian ellipsoids with associated attributes (position, color, opacity, covariance, etc.), which are projected onto the image plane via differentiable rasterization to render novel views. Compared to NeRF's volumetric rendering, 3DGS is one to two orders of magnitude faster with comparable quality, making it a popular foundation for real-time 3D perception. However, traditional 3DGS relies on a complete SfM point cloud for initialization, making it unsuitable for incremental online scenarios. It is in this context that S2Tok explores how to combine Gaussian representations with streaming feed-forward inference.

S2Tok's Technical Framework

S2Tok is a feed-forward framework whose core goal is to maintain an adaptively-sized, persistent scene state from a stream of uncalibrated images. Its central idea can be summarized in one sentence: decouple updates to existing representations from selective expansion.

Specifically, the framework consists of several key modules:

Spatially-Aware Transformer

Whenever a new observation arrives, a spatially-aware Transformer integrates it with the existing persistent scene tokens. Rather than simply stacking new frames onto the history, the model lets new evidence interact and fuse with the existing state under spatial structural constraints.

Learned Admission Module

This is one of S2Tok's more elegant design choices. To avoid redundant storage — for instance, the representation shouldn't grow unboundedly when the same scene region is repeatedly observed — the framework introduces a learnable admission module that selectively decides when to expand the representation. The representation only grows when new observations genuinely cover previously unseen content. This allows the scale of the scene state to adapt to scene complexity, rather than growing linearly with the frame count.

The concept of admission control originates from networking and operating systems, referring to the mechanism for deciding whether to accept new requests under limited resources. Similar ideas appear in NeRF's sparse voxel pruning and dynamic point cloud insertion/deletion, but these typically rely on hand-designed threshold rules. S2Tok makes this decision an end-to-end learnable module, meaning the judgment of "when to expand the representation" is driven by gradient signals and can be automatically calibrated according to the task objective (the trade-off between rendering quality and storage efficiency), without manual tuning. This aligns conceptually with research on write gating in Memory-Augmented Networks.

Hierarchical Decoder and Gaussian Head

The evolving scene state ultimately needs to become renderable geometry. S2Tok uses a hierarchical decoder and Gaussian head to convert the latent state into non-pixel-aligned 3D Gaussians. This is significant: because the Gaussians are not tied to specific pixels, the system can render novel views without caching historical frames, substantially reducing the memory footprint during online operation.

Latent Spatial Tokens as Computational State

The theoretical contribution of this work lies in finding a new role for latent spatial tokens — as persistent computational state for online 3D reconstruction.

Previously, latent spatial tokens were mostly treated as static representations constructed from a collection of images, and how to maintain them online remained an open question. S2Tok's answer is to combine learned scene updates with explicit Gaussian rendering. The former is responsible for accumulating and fusing information at an abstract level; the latter translates the abstract state into concrete, visualizable geometry. This two-tier structure of "implicit state + explicit rendering" balances representational flexibility with rendering controllability.

Pioneer work in multi-view reconstruction using latent spatial tokens includes pixelNeRF, IBRNet, and more recent feed-forward reconstruction methods like MASt3R / DUSt3R. These methods typically encode images as feature tokens and aggregate scene information via cross-view attention, but operate offline in batch mode: all frames are input simultaneously and the token count is fixed. S2Tok's innovation lies in introducing the temporal dimension — the token set grows over time, and existing tokens can be modified by new observations — making them truly function as hidden states in a recurrent system. This conceptually echoes persistent memory research in video understanding and Transformer applications in sequential decision-making, but is specifically adapted for the spatial consistency of 3D geometry.

Experimental Results and Significance

According to the paper, the researchers evaluated S2Tok on four benchmark datasets. Results show that S2Tok achieves competitive streaming rendering quality while maintaining a compact Gaussian representation. In other words, it does not trade storage explosion for quality — it strikes a reasonable balance between the two.

These experimental results support the paper's central claim: latent spatial tokens can serve as persistent computational state for online 3D reconstruction. For applications requiring real-time mapping — robotics, autonomous driving, AR/VR devices — methods that continuously update with an observation stream without caching all historical frames offer clear engineering value.

Of course, as a recently released preprint, S2Tok's generalization capability, performance at extreme scales, and practical integration with existing SLAM systems remain to be validated by follow-up work. But its core insight of "distinguishing updates from expansion," along with the admission module design, offers a noteworthy direction for streaming neural reconstruction.

Summary

S2Tok's contribution lies in abstracting streaming 3D reconstruction as a problem of "maintaining a persistent scene state," and providing a feed-forward solution through the synergy of a spatially-aware Transformer, a learned admission module, and a hierarchical Gaussian decoder. It addresses the challenge of online maintenance of latent spatial tokens while finding a pragmatic balance between rendering quality and storage compactness. For researchers and engineers focused on real-time 3D perception, this is a paper worth tracking.