Splat2Act: Scan Your Home with a Phone to Teach Robots How to Grab Your Stuff

A Overlooked Deployment Challenge

When you bring a general-purpose robot manipulation policy home or to the office, the most natural expectation is simple: it should be able to pick up your things from your shelves, countertops, and tables. But reality falls far short of that vision — researchers found that even a powerful general-purpose policy like $π_{0.5}$ achieves a 0% grasping success rate in real deployment environments without adaptation.

The root cause is the gap between generality and personalization. General-purpose policies are trained on massive datasets and possess broad generalization capabilities, but every user's home environment, object arrangement, and lighting conditions are unique. The traditional approach to adapting a policy to a specific environment requires collecting real demonstration data — a process so costly that repeating it across every user's home is simply impractical.

Splat2Act: From Captured Scenes to Personalized VLA Manipulation

Splat2Act was built to solve exactly this problem. Its core idea is a real-to-sim-to-real pipeline that lets users simply scan a scene with their phone and photograph their objects, then automatically personalizes a pretrained vision-language-action (VLA) policy to that specific environment and object set.

From Phone Scan to Building-Scale Digital Twin

Splat2Act has one key distinction from prior real-to-sim work: previous methods typically only reconstructed small-scale scenes like a single tabletop, whereas Splat2Act directly handles building-scale indoor environments.

The pipeline works ambitiously. It first uses 3D Gaussian Splatting (3DGS) to construct a photorealistic digital twin, while simultaneously generating collision geometry for physical interactions in simulation. The system then automatically discovers tabletop surfaces and non-tabletop receptacles in the scene — things like cabinets and countertops where objects can be placed.

From there, the pipeline automatically constructs feasible grasping tasks and generates successful demonstration data using a "privileged simulation oracle." Throughout the entire process, the only thing a human needs to do is scan the scene and photograph the objects — everything else is fully automated.

The significance of this design is that it reduces the most expensive and labor-intensive part of data collection — repeatedly operating a robot in each real environment to gather demonstrations — to an absolute minimum.

3D Gaussian Splatting (3DGS) is a real-time 3D scene reconstruction technique that emerged in 2023. Its core idea is to represent a scene using millions of 3D Gaussian ellipsoids, each with position, color, opacity, and covariance attributes, rather than traditional meshes or voxels. Compared to NeRF (Neural Radiance Fields), 3DGS renders tens of times faster and can directly output geometric information usable by physics engines. In robotics, both properties are critical: high-speed rendering enables large-scale parallel simulation, while accurate geometry output is a prerequisite for collision detection and grasp planning. Splat2Act leverages 3DGS to simultaneously achieve photorealistic visual fidelity and simulatable physical geometry, ensuring that demonstration data generated in virtual environments closely matches the visual distribution of the real world — a foundational technical requirement for successful sim-to-real transfer.

Data Generation at Scale

The research team applied this pipeline to 101 scanned campus environments, with a median navigable area of 104.87 square meters and an average of 7 receptacles per environment.

The pipeline ultimately produced 852 receptacles and over 68,000 grasping demonstrations. This figure vividly demonstrates the scalability advantage of an automated pipeline over manual collection — collecting tens of thousands of demonstrations across hundreds of real environments by hand would be virtually impossible.

This large-scale simulation data generation capability is what makes subsequent policy fine-tuning feasible. It transforms "data scarcity" — a longstanding challenge in robot learning — into an engineering problem that can be solved through automated batch production.

A Two-Stage Fine-Tuning Strategy

Splat2Act fine-tunes $π_{0.5}$ in two stages, each building on the last:

Stage 1: Adapting to Public Object Assets

The first stage adapts the policy to hundreds of public object assets. These assets can come from two sources: either from the target deployment scene alone, or from all 101 scenes combined. The goal of this stage is to give the policy a broad understanding of scene structure and object categories.

Stage 2: Personalizing to the User's Own Objects

The second stage is where true personalization happens — the policy is fine-tuned to work with the user's own objects and optimized for grasping tasks on specific receptacles within the deployment scene.

This staged design reflects a progressive philosophy moving from "general adaptation" to "bespoke personalization," preserving generalization ability while precisely aligning with the user's actual needs.

Vision-Language-Action Models (VLA) are a new generation of manipulation policy frameworks that end-to-end combine large vision-language models (such as GPT-4V, PaLI, etc.) with robot action outputs. Their inputs are typically camera images and natural language instructions; their outputs are robot joint angles or end-effector motion commands. $π_{0.5}$ is a general-purpose VLA released by Physical Intelligence, pretrained on thousands of hours of cross-robot, cross-task demonstration data, with strong zero-shot manipulation capabilities. However, VLA generality comes from the breadth of training data — which also means the model has no "memory" of any specific user's environment. Fine-tuning is the standard method for bridging this gap: using a small amount of domain-specific data to steer the model's behavior toward the target scene, while retaining the general knowledge encoded in the pretrained weights, without training from scratch.

Validation in Both Simulation and the Real World

The results are impressive. After the full pipeline, the strongest personalized policy achieved an overall 62.5% grasping success rate in the target scene — a qualitative leap from the baseline of 0%.

More critically, the team validated the policy on a real Stretch 3 robot, achieving a 63.9% success rate on a subset of 12 tasks, compared to a 66.7% success rate on those same tasks in simulation.

The gap between simulation success (66.7%) and real-world success (63.9%) is remarkably small, indicating that the sim-to-real transfer gap has been effectively bridged. This is precisely what the real-to-sim-to-real paradigm set out to prove: that data generated solely by a simulation oracle can train a personalized manipulation policy that works in the real world.

The sim-to-real gap is a long-standing core challenge in robot learning: policies trained in simulation often cannot be directly deployed on real robots, due to differences in visual appearance (materials, lighting, rendering noise), physical dynamics (friction coefficients, joint compliance), and sensor noise model mismatch. Traditional mitigation approaches include domain randomization (randomly perturbing textures, lighting, and physical parameters in simulation) and domain adaptation (additional training of visual encoders on real image distributions). Splat2Act compresses the visual domain gap from the ground up by using 3DGS reconstruction to obtain visual inputs nearly identical in distribution to the real scene — this is the primary reason the simulation and real-world success rates differ by less than 3 percentage points, and it reflects a more precise technical approach: "replace randomization with photorealistic reconstruction of the actual scene."

Implications for Personalized Robot Deployment

The value of Splat2Act lies not just in a few specific success rate numbers, but in validating a scalable technical pathway: with just a smartphone, users can adapt a general-purpose robot to their own home.

This integrates several cutting-edge technologies — photorealistic reconstruction via 3D Gaussian Splatting, general manipulation capability from VLA foundation models, and automated task construction with simulation-based demonstration generation. Together, they push "personalized robotics" — a challenge that once required professional engineers — toward something accessible to everyday users.

Of course, a 62.5% success rate is still some distance from the reliability a truly consumer-ready home robot would require. But the leap from 0% to 62.5% already offers a compelling proof-of-concept for deploying general-purpose manipulation policies in the real world.