StateCraft

StateCraft represents a shift in thinking: rather than letting the model implicitly memorize everything, it gives it an explicit, structured state representation, thereby improving performance and controllability and making the inner workings of the world model more transparent and intervenable.

StateCraft surpasses existing baselines in all three dimensions: generation quality, memory retention and controllability.

StateCraft's core architecture consists of two components: an abstract state responsible for tracking entity attributes and changes, and a visual state generated by a video diffusion model that represents the current viewpoint's frame.

In StateCraft, a language model is responsible for maintaining and updating the abstract state, and the video diffusion model transforms the abstract state into visual output, with the generation of the visual state conditioned on the abstract state.

StateCraft chooses generative interfaces as its primary evaluation scenario.

StateCraft's abstract state is viewpoint-independent, and even if certain changes have never been observed in the training data, the model can respond correctly by editing the abstract state.

What supports StateCraft is an automated data collection pipeline that can generate information-dense, richly annotated trajectory data at scale without human or agent supervision.

StateCraft's accompanying evaluation benchmark measures model performance across three dimensions: generation quality, memory capability and controllability.

On the accompanying evaluation benchmark, StateCraft comprehensively surpasses existing state-of-the-art methods and maintains high self-consistency and stable frame quality in long-trajectory generation.

StateCraft's abstract state is explicit, and users can directly modify it, with the generated results following these edits even if the edited state has never appeared in the training data.

StateCraft was proposed by researchers at insait-institute and addresses the long-range consistency problem of video world models by decoupling a persistent, updatable abstract state from the visual state.

StateCraft's 3D experiments are currently at the proof-of-concept stage and have not yet been extended to a complete application.

The research team demonstrated a 3D proof-of-concept experiment: an action changes the lighting conditions of an unobserved room, and the model needs to correctly infer this change and reflect it in subsequent generation.