Splat2Act
An automated real-to-sim-to-real pipeline for personalizing robot manipulation policies. It uses 3D Gaussian Splatting to construct a digital twin, automatically generates demonstration data through simulation, and adapts pre-trained vision-language-action (VLA) policies to user-specific scenes and objects.
Timeline (last 90 days)
The general-purpose policy model π_{0.5} has a grasping success rate of 0% in real deployment scenarios without adaptation.
Splat2Act is an automated pipeline whose core concept is real-to-sim-to-real, allowing users to scan a scene with a phone and photograph objects in order to personalize a pre-trained VLA policy to a specific scene and set of objects.
Splat2Act directly handles building-scale indoor environments, whereas prior real-to-sim methods mostly reconstruct only small-scale scenes such as individual tabletops.
Splat2Act uses 3D Gaussian splatting to construct photorealistic digital twins while also generating collision geometry for physical interaction in simulation.
The research team applied the pipeline to 101 scanned campus environments, with a median navigable area of 104.87 square meters and an average of 7 receptacles per environment.
The Splat2Act pipeline produced 852 receptacles and more than 68,000 grasping demonstration data entries.
Splat2Act's fine-tuning of π_{0.5} is divided into two stages: the first stage adapts to hundreds of common object assets, and the second stage personalizes to the user's own objects and optimizes for the specific receptacle grasping task.
After training through the complete pipeline, the strongest personalized policy achieves an overall grasping success rate of 62.5% in the target scene.
The research team deployed the policy on a real Stretch 3 robot, achieving a 63.9% success rate on a subset of 12 tasks, compared to a simulation success rate of 66.7% on those same tasks.
The difference between Splat2Act's simulation success rate and real-robot success rate is less than 3 percentage points, primarily because the 3DGS reconstruction produces visual inputs with nearly the same distribution as the real scene, fundamentally compressing the visual domain gap.
1 more timeline events
All Facts (11)
The general-purpose policy model π_{0.5} has a grasping success rate of 0% in real deployment scenarios without adaptation.
50%UnverifiedSplat2Act is an automated pipeline whose core concept is real-to-sim-to-real, allowing users to scan a scene with a phone and photograph objects in order to personalize a pre-trained VLA policy to a specific scene and set of objects.
50%UnverifiedSplat2Act directly handles building-scale indoor environments, whereas prior real-to-sim methods mostly reconstruct only small-scale scenes such as individual tabletops.
50%UnverifiedSplat2Act uses 3D Gaussian splatting to construct photorealistic digital twins while also generating collision geometry for physical interaction in simulation.
50%UnverifiedThe research team applied the pipeline to 101 scanned campus environments, with a median navigable area of 104.87 square meters and an average of 7 receptacles per environment.
50%UnverifiedThe Splat2Act pipeline produced 852 receptacles and more than 68,000 grasping demonstration data entries.
50%UnverifiedSplat2Act's fine-tuning of π_{0.5} is divided into two stages: the first stage adapts to hundreds of common object assets, and the second stage personalizes to the user's own objects and optimizes for the specific receptacle grasping task.
50%UnverifiedAfter training through the complete pipeline, the strongest personalized policy achieves an overall grasping success rate of 62.5% in the target scene.
50%UnverifiedThe research team deployed the policy on a real Stretch 3 robot, achieving a 63.9% success rate on a subset of 12 tasks, compared to a simulation success rate of 66.7% on those same tasks.
50%UnverifiedThe difference between Splat2Act's simulation success rate and real-robot success rate is less than 3 percentage points, primarily because the 3DGS reconstruction produces visual inputs with nearly the same distribution as the real scene, fundamentally compressing the visual domain gap.
50%UnverifiedTraditional methods for mitigating the sim-to-real gap include domain randomization and domain adaptation; Splat2Act adopts a technical approach of replacing randomization with photorealistic reconstruction of the real scene.
50%