GRASE: Turning Videos into Editable 3D Scenes with an Agentic Closed Loop

Converting a video of a real environment into an editable, simulatable 3D scene has long been a core need in robotics, autonomous driving, and virtual content production. But this "Real-to-sim" path is far from easy—traditional approaches tend to be one-way pipelines, where each step only receives limited information from the previous one, errors accumulate layer by layer, and the final scene ends up distorted in both geometry and layout. A paper titled 《GRASE: Geometry-native Real-to-sim Agentic Scene Modeling from Video》 proposes a new idea: using an agentic framework to turn the entire modeling process into a closed feedback loop, keeping the original video as a constant "reference" that constrains the output of every step.

Why One-Way Pipelines Fail Easily

Most existing Real-to-sim methods follow a "one-shot, loosely coupled" paradigm. Depth estimation, layout inference, asset generation, and geometry optimization each operate independently, with each stage only receiving the output of the previous one and unable to question or correct prior results. Once a single link deviates—say, a depth map fails on transparent surfaces or weakly textured regions—the error propagates like a snowball, ultimately producing problems such as misaligned walls, floating objects, and distorted proportions in the final scene.

What's worse, such methods typically "lock in" intermediate estimates. Once the system commits to a layout assumption, subsequent steps can only work under that assumption, with no opportunity for global correction. For simulation tasks requiring precise geometric consistency, this rigid structure is fatal.

Real-to-sim refers to converting real-world observations (video, point clouds, depth maps, etc.) into digital scenes that can drive simulation, widely used in robot training, autonomous driving testing, and virtual content production. Its core challenge is that real observations are 2D, sparse, and noisy, whereas simulation scenes require 3D, dense, and geometrically self-consistent representations.

Error accumulation is an inherent flaw of one-way pipelines. Depth estimation, layout inference, asset generation, and geometry optimization execute in sequence; small upstream deviations (such as missing depth caused by transparent surfaces) are treated as established facts by downstream stages and eventually amplified into significant errors in geometry and layout. The value of closed-loop feedback lies precisely in using the original observations to continuously verify the output of each step, suppressing errors near their source rather than exposing them only at the end.

GRASE's Core Idea: Treating Video as a "Continuous Reference"

The keywords of GRASE are "agentic" and "geometry-native." It no longer treats multi-view video as a one-time input, but as a reference benchmark that runs through the entire modeling process. The system does not rush to finalize intermediate results, but continuously maintains a current hypothesis and repeatedly compares rendered evidence against the original video.

Once the rendered result disagrees with the observation, GRASE locates which component is at fault and modifies only that part, rather than starting from scratch. This "fix only what's wrong" strategy ensures cross-view consistency while avoiding unnecessary global perturbations. In other words, it turns modeling from "one-shot solving" into "closed-loop control with continuous correction."

How the Three-Stage Feedback Loop Works

GRASE's pipeline can be broken down into three interlocking stages, each of which can receive correction signals from subsequent feedback.

Stage One: Fitting the Building Shell and Openings

The system first fits the building's structural shell from depth cues and multi-view information, including walls, floors, ceilings, and openings such as doors and windows. This step establishes the scene's "skeleton" and provides a unified coordinate system for subsequent asset placement.

Building shell refers to structural surfaces such as walls, floors, and ceilings in a scene, while openings include holes such as doors and windows that connect the interior and exterior. The significance of fitting the shell and openings lies in establishing a unified global coordinate system: only after the structural skeleton is determined can subsequently generated assets like sofas and tables be placed into a consistent spatial framework, avoiding floating objects, wall clipping, or distorted proportions. This step typically relies on depth cues and multi-view geometric constraints, and is a key link in Real-to-sim that determines overall layout quality.

Stage Two: Placing Assets in a Shared Coordinate System

Next, the agent is responsible for placing the generated 3D assets into the same coordinate framework and checking their spatial relationships—such as whether the sofa is against the wall or whether the table blocks the door. This step emphasizes layout plausibility rather than the fine detail of individual objects.

Stage Three: Iteratively Refining Geometry and Position

Finally, the geometry and placement of each asset are iteratively optimized, with the goal of making the rendered result match the original video as closely as possible across multiple observation viewpoints. Feedback truly forms a closed loop here: rendered evidence drives corrections, and corrections generate new rendered evidence.

Comparison Results with Methods Such as GPT6-Astra

Experiments in the paper show that GRASE significantly outperforms existing state-of-the-art methods on multiple metrics. One notable comparison is with a scheme that directly uses GPT6-Astra for real-to-sim conversion. Although large models perform excellently in semantic understanding and asset generation, in scenarios requiring strict geometric consistency, a single-pass conversion without a feedback loop still struggles to guarantee precise cross-view alignment.

GRASE's advantage comes precisely from not relying on the "luck" of a single generation, but instead suppressing errors through iterative correction. For applications requiring physical simulation, robot training, or digital twins, this corrigible and traceable modeling approach is more reliable than "one-shot perfection."

What It Means for the Real-to-sim Field

GRASE embodies an emerging trend: bringing agents into 3D modeling not to make models "guess more cleverly," but to make them "revise more systematically." Video is no longer just input, but a criterion running throughout. This idea has direct value for robot simulation, AR/VR content creation, and industries that need to rapidly build digital replicas from real scenes.

Of course, closed-loop feedback means more iterative computation. How to control cost while maintaining accuracy will be the next hurdle for such methods on the path to practicality. But at least GRASE proves that giving the modeling process the ability to "go back and revise" is more realistic than pursuing one-shot perfection.

One-line Summary

GRASE uses an agentic closed-loop feedback to turn video real-to-sim modeling from a one-way pipeline into continuously corrected closed-loop control.

Paragraph Summary

GRASE proposes a geometry-native agentic framework to solve the error accumulation problem of traditional Real-to-sim one-way pipelines. It treats multi-view video as a continuous reference throughout the process rather than a one-time input, operating through a three-stage closed loop: first fitting the building shell and door/window openings to establish a skeleton, then having the agent place 3D assets in a shared coordinate system and check spatial relationships, and finally iteratively refining geometry and position until the rendered result matches the original video across multiple viewpoints. The feedback mechanism allows the system to locate problematic components and make local corrections rather than starting over. Experiments show that GRASE significantly outperforms existing state-of-the-art methods, including schemes that directly use GPT6-Astra for single-pass conversion, with clear advantages in cross-view geometric consistency, making it suitable for applications such as robot simulation and digital twins.