ATAS: A World-State-Centric Framework for Agentic Video Generation

From Script Text to Consistent Shots: A New Challenge in Video Generation

Generating long-form, multi-shot, film-grade content is becoming the next battleground for generative AI. Yet current mainstream approaches struggle with cross-shot consistency: when you want the model to "return" to the same scene from a different angle, or to re-present an event that occurred in an earlier shot during a later one, simply passing prompts or the last few frames fails to establish a unified spatial reference—and there is no time-indexed, replayable record to draw on.

This is precisely the core problem the paper Any Time, Any Space sets out to solve. The authors propose a framework called ATAS (Any Time, Any Space) that upgrades video generation from "shot-by-shot stitching" to "scheduling around world state."

Cross-shot consistency refers to the ability to keep scene layout, object appearance, lighting atmosphere, and event logic aligned across multiple independently generated video clips. It is a fundamentally different problem from intra-shot temporal consistency (no flicker between adjacent frames, smooth motion): the latter only requires modeling over a window of seconds, whereas the former demands that the model maintain a stable implicit world representation across minutes or longer, and through the viewpoint jumps caused by shot transitions. Traditional approaches rely on first/last-frame conditioning, reference image injection, or long-context windows, but these essentially compress "memory" into tokens or latent variables—once the gap grows longer, the spatial reference drifts, and precise retrieval of a specific historical moment becomes impossible.

ATAS's Core Idea: Compiling a Script into Retrievable Shot Requests

The key design of ATAS is to "compile" a script into shot requests with explicit references. Each shot request clearly annotates three types of information: location, camera, and event-time. With these structured references, non-contiguous shots can share the same spatial and temporal coordinates.

This compilation layer turns video generation into essentially a "retrieve–generate–register" loop, rather than a black-box reasoning process that relies solely on context.

In film production, a shot request typically includes framing, camera position, motion trajectory, and timecode—it is the smallest executable unit of communication between a director and a cinematographer. ATAS borrows this industry convention and formalizes it into a structured object that a model can parse: location provides the spatial anchor, camera describes the viewpoint transformation for observing that space, and event-time gives the shot's position on the story timeline. Together, the three amount to assigning each shot a set of "query conditions in a world coordinate system," so that even if two shots are not adjacent in generation order, they can be located on the same space and the same timeline—avoiding the model having to guess scene associations purely by semantic similarity.

Pre-Generation Retrieval: Scene Memory and Event Memory

Before executing each shot request, ATAS retrieves relevant evidence from two independently maintained memories:

• Scene memory: stores registered static geometry, providing a reference for spatial reproduction;
• Event memory: dynamic observation records indexed by scene, event, and time, supporting replay-style reconstruction of earlier events.

This dual-memory structure of "static geometry + dynamic observations" essentially builds a queryable state table for the video world. The model no longer merely "sees" the previous frame—it can "look up" what this world was previously confirmed to look like and what has happened in it.

Retrieval-augmented generation (RAG) is already fairly mature in large language models; its idea is to have the model retrieve evidence relevant to the current query before generating, rather than relying entirely on knowledge frozen in parameters. ATAS migrates this paradigm to the video domain: pre-generation retrieval is equivalent to preparing a "set description" and an "event archive" for the current shot. The difference is that the unit of text retrieval is a paragraph, whereas the unit of video retrieval is geometric structure and dynamic observation—the former determines what the frame should look like, the latter determines what should be happening within it. This separation allows the two dimensions to evolve independently: the scene can remain still while events advance, or the scene can change while the event continues.

Post-Generation Registration: Only Approved Observations Enter Memory

After generation completes, ATAS reconstructs and registers the clip. An important detail: only when both review and state checks pass is the observation written to memory. This prevents erroneous frames from contaminating the world state and being continuously amplified in subsequent shots.

The authors also emphasize that film delivery and memory admission are decoupled—even if a clip is not accepted into memory, the final output is still completed independently. This design balances the stability of the production pipeline with the reliability of the memory store.

Experimental Results: Comprehensive Improvement in Cross-Shot Consistency

The numbers in the paper are quite convincing:

• On four different video generators, ATAS improves all reported cross-shot consistency metrics;
• For every backbone model, ATAS achieves the highest overall benchmark score;
• On the spatial reproduction task for target images, it outperforms the compared agent-based and 3D-guided methods;
• It leads on all three criteria for event recapture;
• It receives the highest scores on all six human evaluation criteria.

These results cross-validate the same conclusion across multiple independent dimensions: organizing the generation process around world state maintains spatiotemporal consistency better than mere prompt or frame passing.

Why This Represents a Paradigm Shift

The most valuable aspect of ATAS is treating video generation as a "world state management" problem rather than a "next-frame prediction" problem. It explicitly models space and time as retrievable, reviewable, and registrable state, allowing the agent to maintain the same worldview across non-contiguous requests.

For film-grade content generation that requires long duration, multiple scenes, and revisability, this path offers an approach closer to real production workflows—establish the world first, then schedule the shots. The project page and code are publicly available at https://lhmd.top/atas/.

A world model typically refers to an internal representation capable of predicting the future state of an environment, originally used for planning in reinforcement learning. The "world state" proposed by ATAS is related but leans more toward a manageable, explicit archive: it does not require the model to learn complete environment dynamics, but instead solidifies verified observations into queryable records for subsequent decision-making. This approach mitigates an inherent problem of pure world models—once a prediction goes wrong, the error accumulates autoregressively. Through the separation of review gating and memory admission, ATAS keeps erroneous frames contained within a single generation rather than letting them propagate along the timeline, which is especially critical for long-duration content generation.

One-line Summary

ATAS reframes video generation as a retrieve–generate–register loop around world state, significantly improving cross-shot spatiotemporal consistency.

Paragraph Summary

Targeting the difficulty of maintaining cross-shot consistency in long-form, multi-shot generation, ATAS proposes a world-state-centric framework. It compiles a script into structured shot requests with references to location, camera parameters, and event time, so that non-contiguous shots share the same spatiotemporal coordinates; before generating each shot, it retrieves evidence from scene memory (static geometry) and event memory (dynamic observations indexed by scene, event, and time), and after generation it reconstructs and registers the clip, with only approved observations written to memory—decoupling film delivery from memory admission. Experiments show that ATAS improves all cross-shot consistency metrics across four video generators, and leads compared methods in spatial reproduction, event recapture, and six human evaluation criteria.