GRASE
A real-to-sim scene modeling framework based on closed-loop agent feedback. It uses multi-view video as a continuous reference and performs three-stage iterative correction to generate geometrically consistent 3D simulatable scenes.
Timeline (last 90 days)
GRASE is a Real-to-sim method based on an agent framework that converts multi-view video into editable, simulatable 3D scenes through closed-loop feedback.
GRASE adopts a 'fix only what's wrong' strategy: when the rendered result does not match the original video, only the problematic component is modified rather than starting over from scratch.
GRASE's process is divided into three interlocking stages: fitting the building shell and openings, placing assets in a shared coordinate system, and iteratively refining geometry and position.
In GRASE's stage one, the building structural shell (walls, floor, ceiling) and openings (doors, windows) are fitted from depth cues and multi-view information, establishing a unified coordinate system for subsequent asset placement.
In GRASE's stage two, the agent places the generated 3D assets into the same coordinate frame and checks the reasonableness of spatial relationships, such as whether a sofa is against the wall or whether a table blocks a door.
In GRASE's stage three, the geometry and placement position of each asset are iteratively optimized so that the rendered result matches the original video as closely as possible across multiple observation viewpoints, forming a feedback loop.
The paper's experiments show that GRASE significantly outperforms existing state-of-the-art methods on multiple metrics.
GRASE's closed-loop feedback approach is more reliable than single-pass generation and is suitable for applications requiring physical simulation, robot training, or digital twins.
Closed-loop feedback means more iterative computation, and how to control cost while maintaining accuracy is the next hurdle for such methods on the path to practical adoption.
All Facts (9)
GRASE is a Real-to-sim method based on an agent framework that converts multi-view video into editable, simulatable 3D scenes through closed-loop feedback.
50%UnverifiedGRASE adopts a 'fix only what's wrong' strategy: when the rendered result does not match the original video, only the problematic component is modified rather than starting over from scratch.
50%UnverifiedGRASE's process is divided into three interlocking stages: fitting the building shell and openings, placing assets in a shared coordinate system, and iteratively refining geometry and position.
50%UnverifiedIn GRASE's stage one, the building structural shell (walls, floor, ceiling) and openings (doors, windows) are fitted from depth cues and multi-view information, establishing a unified coordinate system for subsequent asset placement.
50%UnverifiedIn GRASE's stage two, the agent places the generated 3D assets into the same coordinate frame and checks the reasonableness of spatial relationships, such as whether a sofa is against the wall or whether a table blocks a door.
50%UnverifiedIn GRASE's stage three, the geometry and placement position of each asset are iteratively optimized so that the rendered result matches the original video as closely as possible across multiple observation viewpoints, forming a feedback loop.
50%UnverifiedThe paper's experiments show that GRASE significantly outperforms existing state-of-the-art methods on multiple metrics.
50%UnverifiedGRASE's closed-loop feedback approach is more reliable than single-pass generation and is suitable for applications requiring physical simulation, robot training, or digital twins.
50%UnverifiedClosed-loop feedback means more iterative computation, and how to control cost while maintaining accuracy is the next hurdle for such methods on the path to practical adoption.
50%