SwitchSplat: Learning Discrete Object States from Casual Videos — 3DGS Finally Gets a "Switch"

When 3DGS Meets the "Movable Switch": The End of the Static Assumption

3D Gaussian Splatting (3DGS) has made high-fidelity 3D reconstruction fast and easy to use, but existing representations all assume the world is static. This assumption completely breaks down in casually captured scenes — users naturally interact with their environment: opening doors, flipping switches, moving objects. These operations introduce discrete state changes, and no current framework can model them.

Time-conditioned dynamic methods (such as 4DGS) conflate "the moment a state was observed" with "what state the scene is in," which means they can only replay previously seen sequences and cannot freely combine arbitrary object configurations. SwitchSplat was proposed precisely to fill this gap.

3D Gaussian Splatting (3DGS) is an explicit 3D scene representation method proposed in 2023: it characterizes a scene using hundreds of thousands to millions of anisotropic 3D Gaussian ellipsoids, each carrying position, covariance (determining scale and orientation), opacity, and spherical harmonic coefficients (expressing view-dependent color). During rendering, these Gaussians are projected onto the image plane and composited via depth-ordered differentiable alpha blending, achieving real-time frame rates while maintaining photorealistic quality. Unlike NeRF, which relies on implicit MLP queries, 3DGS's explicit structure and rasterization-based rendering pipeline make both training and rendering much faster, and easier to combine with downstream modules such as differentiable optimization and segmentation. For this reason, 3DGS has become one of the mainstream approaches for static scene reconstruction — but its optimization objective by default assumes that scene geometry and appearance remain fixed during training. Once an object's state in the scene changes, a single static set of Gaussians cannot simultaneously accommodate two mutually exclusive geometric configurations. This is precisely the origin of the "static assumption" discussed in this article.

Base + Slot: Decoupling Discrete States

The core of SwitchSplat is a base+slot decomposition architecture. A shared static base captures the invariant parts of the scene, while each slot independently holds two binary states (on/off) of an interactable object. Crucially, the process of assigning Gaussian points to slots is fully differentiable, meaning the model can directly discover the geometry of each state from video without any manual annotation.

The elegance of this decomposition lies in how it matches the nature of the problem: most of the scene is static, and only a few objects have discrete states. By separating static from dynamic and continuous from discrete, the model can focus on learning the parts that actually change.

A Fully Differentiable Assignment Mechanism

How does a Gaussian point know whether it belongs to the base or to a slot? SwitchSplat lets the model figure it out through differentiable optimization. During training, the model observes different states of objects from video and, through gradient signals, pushes the corresponding Gaussian points toward the correct slots. This avoids the tedious work of manually annotating state masks and makes the method easier to extend to new scenes.

4DGS (time-conditioned dynamic Gaussian) methods typically take time as an additional input: either introducing time-varying position, rotation, and opacity parameters for each Gaussian and predicting these changes via deformation fields or multi-resolution temporal features; or bucketing time and learning independent Gaussian sets for different moments followed by interpolation. The common premise of these approaches is that scene evolution is a continuous function of time, so supervision is applied only at observed time points during training. The problem is that changes in real interactive scenes are discrete jumps — a switch has only on and off configurations, with no physical "half-open" state in between. Forcing such changes onto a continuous time axis means the model can only memorize the trajectory of the observed sequence; once it is asked to combine different objects' states at a time point outside the training distribution, it produces implausible intermediate geometry or fails outright. SwitchSplat introduces an independent "state" dimension rather than simply extending the time axis, precisely to bypass this continuous-deformation assumption.

Differentiable assignment refers to relaxing the discrete decision of "which component a point belongs to" into a continuous form that can be optimized end-to-end. A common approach is to have each Gaussian output a set of weights for the base and each slot (e.g., softmax-normalized membership probabilities), and during rendering composite the radiance contributions of each component according to these weights, so that gradients from the loss function can be backpropagated to the membership parameters themselves. This differs from methods that require sampling discrete labels, rely on REINFORCE, or use non-differentiable clustering: membership probabilities can be directly optimized and gradually sharpened during training toward near-0/1 hard assignments. For SwitchSplat, this means that without pre-providing masks of "which pixels belong to the door and which to the switch," the model can, purely from the reconstruction loss on multi-state video, push the two geometrically mutually exclusive sets of Gaussians toward their corresponding slots — significantly reducing annotation cost and making the method easier to transfer to new scenes.

Free Composition at Inference: Beyond "Replay"

At inference time, users can freely combine slot states and render the resulting scene from any viewpoint. This means that even if a particular state combination never appeared during capture, the model can synthesize a plausible rendering. For example, if only "door open + light off" and "door closed + light on" were seen during capture, the model can still render the combination "door open + light on."

This is exactly what traditional dynamic methods cannot do. 4DGS essentially interpolates along the time axis or memorizes observed frames, whereas SwitchSplat models states as discrete variables, enabling compositional generalization.

Empirical Validation: Performance on a New Dataset

The research team validated SwitchSplat on a new casually captured dataset. Results show that it outperforms static and dynamic baselines in novel view synthesis quality and uniquely supports rendering state combinations never seen during capture. This capability is important for applications such as robotic manipulation, scene editing, and interactive content creation.

Key Differences from Dynamic Baselines

Compared with time-conditioned methods like 4DGS, SwitchSplat does not simply add a time dimension but introduces a state dimension. Time is merely an index of observation; state is the essential variable of the scene. This fundamental difference in modeling yields compositional generalization capability.

Limitations and Outlook

The current method focuses on binary states (on/off); objects with multiple or continuous state changes may require an extended slot design. In addition, occlusion and motion blur in casually captured videos may still affect the quality of geometry learning. Nevertheless, as the first work to extend 3DGS to discrete state composition, SwitchSplat opens a new direction for interactive 3D scene reconstruction.

Project page: https://switchsplat.github.io

One-line Summary

SwitchSplat uses base+slot decomposition to let 3DGS model discrete object states and supports rendering of unseen combinations.

Paragraph Summary

This article introduces SwitchSplat, a 3D Gaussian Splatting extension for interactive casually captured scenes. Existing methods such as 3DGS and 4DGS assume scenes are static or evolve continuously over time, and cannot handle discrete state changes like opening doors or flipping switches. SwitchSplat adopts a base+slot decomposition: a shared static base captures the invariant scene, while each slot independently holds the on/off binary states of an interactable object. Gaussian point membership is automatically learned from video via fully differentiable optimization, without manual state mask annotation. At inference, users can freely combine slot states and render from any viewpoint, synthesizing plausible results even for combinations never captured. On a new casually captured dataset, its novel view synthesis quality outperforms static and dynamic baselines and uniquely supports compositional generalization. Limitations include support for only binary states and the potential impact of occlusion and motion blur on geometry learning.