CoTL: Compact Token Learning Reshapes 3D Scene Representation

From Dense Pixels to Compact Tokens: A New Approach to 3D Scene Representation

3D scene reconstruction has made significant strides in neural rendering in recent years, but how to represent scenes more efficiently remains a core challenge. Most existing methods rely on dense implicit representations or large-scale Gaussian point clouds, with parameter counts growing rapidly with scene complexity. CoTL (Compact Token Learning) proposes a different path: using a small set of compact latent tokens to preserve a scene's geometry and appearance information, allowing representation capacity to scale naturally with the number of observations.

The motivation behind this work is clear—if the scene representation is compact enough, it can not only support novel view synthesis but also serve as a latent space for generative models, providing a foundation for a broader range of downstream 3D vision tasks.

3D Gaussian Splatting is important background for understanding this paper. Traditional Neural Radiance Fields (NeRF) rely on multi-layer perceptrons to implicitly query the color and density of each spatial point, requiring dense sampling along rays during rendering—computationally expensive with a non-explicit representation. 3D Gaussian Splatting instead uses a set of explicit 3D Gaussian primitives to represent the scene: each Gaussian has a position, covariance (determining shape and orientation), opacity, and spherical harmonics coefficients (determining view-dependent color). During rendering, these Gaussians are depth-sorted and projected onto the image plane for rasterization and blending, achieving real-time rendering while maintaining high quality. The cost is that the number of primitives often reaches the millions, with parameters rapidly expanding with scene scale, creating significant storage and transmission burdens. CoTL targets exactly this pain point, attempting to replace or compress massive Gaussian primitives with a small number of latent tokens. Understanding the explicit, renderable nature of Gaussians is therefore key to grasping the design motivation behind this paper's "query-based Gaussian decoder."

Encoder Design: Each View Contributes Only a Few Tokens

The core mechanism of CoTL lies in its encoder. Given a set of pose-free multi-view images, the encoder compresses each image into a small set of latent tokens. These tokens do not describe a single image in isolation; rather, they aggregate information across views to form a view-consistent scene representation.

The key design is that each additional view only extends the representation by a small number of resolution-independent tokens. This means the representation's capacity can grow flexibly with the number of available observations, rather than being constrained by fixed-size network parameters. This incremental representation approach gives CoTL natural scalability in multi-view input scenarios.

Pose-free input is an implicit premise of the encoder in this paper that deserves explanation. Most novel view synthesis methods assume camera intrinsics and extrinsics are known, with poses pre-computed by SfM or SLAM. CoTL, however, takes a set of pose-free multi-view images directly as input, meaning the encoder must implicitly infer or absorb geometric relationships between views while aggregating tokens in order to form a view-consistent representation. This increases task difficulty but also brings the method closer to real-world scenarios—such as casually taken photo collections or internet images that typically come without calibrated camera parameters. Cross-view information aggregation therefore serves not just as feature fusion but also takes on the role of implicit geometric alignment. This is one reason the design of "each view contributing only a few resolution-independent tokens" can work: the tokens carry scene-level shared structure rather than independent descriptions of individual images.

Query-Based Gaussian Decoder: Recovering Renderable Scenes from Tokens

A compact bottleneck representation faces an obvious risk: excessive information compression leading to loss of scene detail. CoTL addresses this with a query-based Gaussian decoder, which in turn constrains the encoder to retain sufficient scene information.

The decoder uses scene-independent queries to access the learned tokens through attention mechanisms, recovering a set of explicit, renderable 3D Gaussian primitives. Supervision from novel view rendering passes through the reconstructed Gaussian scene to train the entire compact representation end-to-end. This closed loop of "encoding compression—decoding reconstruction—rendering supervision" ensures that the token space does not degenerate into meaningless low-dimensional embeddings but genuinely carries renderable 3D structure.

Query-based decoding and Perceiver-style architecture are the core technical paradigms here. So-called scene-independent queries refer to a set of learnable vectors that do not depend on any specific scene, serving as the query side of the attention mechanism to "read" the compact tokens (keys and values) produced by the encoder. Since the number of queries is fixed and decoupled from input resolution, the number of decoded Gaussian primitives is also relatively controllable. This design draws on the idea of latent bottleneck architectures like Perceiver: using a small number of latent units to compress inputs of arbitrary scale, then restoring them to the target modality through cross-attention. Its advantage is that computational complexity no longer grows linearly with input pixel count, while forcing information through a bottleneck to learn more abstract, structured representations. During backpropagation of rendering supervision, gradients must pass through the decoded Gaussian scene, constituting a hard constraint on the encoder to "retain renderable information" and preventing tokens from degenerating into meaningless low-dimensional embeddings.

Experimental Validation: Dual Capabilities in Reconstruction and Generation

The paper conducted experiments on datasets including ACID. Results show that this compact, unstructured token set not only supports competitive novel view synthesis but can also serve as a latent space for generative models, enabling 3D Gaussian scene generation.

This is noteworthy: the same token space can perform both reconstruction and generation, indicating that it captures structural regularities of scenes rather than merely fitting training views. 3D Gaussian reconstruction is demonstrated here to be an effective objective for compact scene representation learning, and the learned token space provides a foundation for broader downstream tasks in 3D vision.

Latent-space generation is a key differentiator of this paper compared to pure reconstruction methods, and its significance merits elaboration. Traditionally, 3D generative models are mostly trained directly on point clouds, voxels, or Gaussian primitive spaces using diffusion or autoregressive models, but these representations are high-dimensional and structurally disorganized, making generative modeling difficult. If an encoder can compress a scene into a compact, well-structured, and semantically consistent token sequence, then that token space can directly leverage mature sequence generation models (such as diffusion models or Transformer autoregression) to learn its distribution, thereby "generating new 3D Gaussian scenes." The ACID dataset is commonly used for such scene-level generation and editing research. CoTL supporting both reconstruction and generation indicates that its token space not only preserves renderability but also possesses good statistical structure—this is precisely the practical value of the "reconstruction as a representation learning objective" approach.

Significance and Limitations of Compact Representation

The value of CoTL lies in addressing a practical issue: 3D scene representations don't always have to keep getting bigger. By compressing the representation into a small number of tokens that scale linearly with the number of views, it creates room for storage, transmission, and generation tasks.

However, based on the abstract information, some questions remain for further investigation: how compact tokens perform under extremely sparse views or large-scale scenes, whether the trade-off curve between token count and reconstruction quality is smooth, and specific metric comparisons on generation tasks. These details require consulting the full paper to assess. Overall, CoTL offers a noteworthy direction for compact 3D representation learning, and its approach of treating reconstruction as a representation learning objective is particularly inspiring.

One-line Summary

CoTL represents 3D scenes with a few compact tokens per view, supporting both novel view synthesis and generation.

Paragraph Summary

CoTL proposes a compact 3D scene representation approach: addressing the problem of existing neural rendering methods where parameter counts balloon with scene complexity, it encodes multi-view images into a small set of resolution-independent latent tokens, with representation capacity growing incrementally with the number of observations. The encoder aggregates information across views to form a view-consistent representation; a query-based Gaussian decoder accesses tokens through scene-independent queries to recover renderable 3D Gaussian primitives, trained end-to-end with rendering supervision to prevent token degeneration. Experiments show that this compact token set supports both competitive novel view synthesis and serves as a latent space for generative models for 3D Gaussian scene generation, validating the effectiveness of 3D Gaussian reconstruction as a compact representation learning objective.