Hybrid Cinematography: Previsualizing Hallucination Risk in Generative Video Reshooting

Generative video technology is transforming the filmmaking process. In traditional shoots, camera movement is locked in on set; generative video reshooting allows directors to change camera movement in post-production—at the cost of "hallucinating" content that was never recorded. This risk often only surfaces after leaving the set, leaving creators with few options. A study called Hybrid Cinematography proposes a workflow that bridges physical shooting and generative reshooting, enabling filmmakers to previsualize and manage hallucination risk right on set.

The Problem: Hallucination Risk in Post-Production Reshooting

Generative video models can synthesize new viewpoints and camera movements from existing footage, opening up possibilities for low-cost pickups and reframing. But the model can only "imagine" unrecorded content based on its training data—when camera movement deviates from the actual shooting coverage, the generated results may diverge from physical reality, producing inconsistencies in space, lighting, or objects.

The paper points out that this hallucination risk is typically delayed: filmmakers on set cannot immediately judge whether a given camera movement can be safely generated, and only realize in post that the footage doesn't support the target viewpoint. At that point, reassembling actors and rebuilding sets is extremely costly, and the project may be forced to compromise.

Generative video reshooting builds on video diffusion models. These models are trained on large-scale video-text pairs, learning to progressively denoise from noise into coherent frame sequences. Given existing footage and conditioning signals (such as target camera trajectories, depth maps, or optical flow), the model can synthesize new viewpoints through "inpainting" or "novel view synthesis" mechanisms. The key limitation is that the model can only express visual patterns within its training distribution. When the target viewpoint lies far beyond the recorded coverage, the model lacks sufficient conditioning signal constraints and relies on priors to "fabricate" content—this is the technical source of "hallucination" in the paper. Understanding this helps explain why the risk is delayed: hallucination is not random noise, but plausible-looking details that contradict physical reality, often only noticed after post-production compositing.

The Solution: A Hybrid Cinematography Workflow

The core idea of Hybrid Cinematography is to bring an editable 3D shot plan and a proxy of the live-action footage onto the set. Through this previsualization system, the team can assess the hallucination risk of target camera movements in real time.

The workflow includes several key capabilities:

• Real-time risk assessment: The system compares planned camera movements against the coverage of already-shot footage, flagging viewpoint regions where the model may lack data support.
• Iterative plan adjustment: After seeing risk areas, filmmakers can modify the shot plan to find a motion scheme that balances physical shooting and generative synthesis.
• Guided pickups: For high-risk areas, the system can guide the team to shoot targeted pickups to fill data gaps.
• Consciously accepting hallucination: In certain creative scenarios, the director may choose to accept a certain degree of generative hallucination in exchange for a more ideal visual effect—provided the risk has been clearly identified.

Editable 3D shot plan refers to reconstructing the camera positions, focal lengths, and motion trajectories of the live-action shoot using a virtual camera in three-dimensional space, typically obtained through camera tracking or visual SLAM. Live-action proxy refers to low-resolution or simplified representations of the footage, used for fast playback and spatial alignment rather than final image quality. Combined, the system can compute the coverage relationship between the target virtual camera trajectory and recorded viewpoints: which regions have real pixel support and which require model extrapolation. This coverage analysis is similar to visibility determination in novel view synthesis and is the core mechanism for assessing hallucination risk. By running this analysis synchronously on set, the team can adjust plans while actors and sets are still in place, moving what would have been post-production decisions earlier.

Tool Implementation: Mobile AR and Offline Pipeline

The research team implemented this workflow as two complementary tools. The first is a mobile augmented reality (AR) application for on-set use, supporting on-site planning, shooting, and review. The AR interface lets filmmakers intuitively see the potential results of generative reshooting and make decisions during the shoot.

The second is an offline pipeline for processing existing video footage. This means the workflow applies not only to on-set shooting for new projects, but also to risk assessment of generative reshooting for completed footage.

Mobile augmented reality (AR) application plays a role of spatial anchoring in this scenario: through the rear camera and inertial measurement unit (IMU) of a phone or tablet, the device can estimate its own six-degree-of-freedom pose on set and overlay virtual camera trajectories and risk-area overlays onto the live view. This lets filmmakers "see" the possible results of generative reshooting much like watching a monitor. The offline pipeline typically reconstructs scene geometry and camera parameters from existing video using structure from motion (SfM) or multi-view stereo (MVS), then assesses data coverage for any target trajectory. The two have different tech stacks but share the same coverage analysis and risk assessment logic, enabling the workflow to serve both on-set shooting and retrospective processing of completed footage.

Research Findings: How Risk Previsualization Affects Decisions

The research team conducted a study with experienced filmmakers to observe how previsualized risk information affects their cinematographic decisions. The results showed that when hallucination risk was visualized in advance, filmmakers adjusted their camera movement plans, tending to choose shooting paths supported by live-action data.

The study also revealed the tension between creative intent and generative hallucination: on one hand, generative tools expand creative freedom; on the other, avoiding hallucination risk may constrain the visual effect the director originally wanted. This tension is not a technical flaw but a creative trade-off that needs to be explicitly managed in the workflow.

Implications for Film and Television Production Pipelines

The significance of Hybrid Cinematography lies in moving the unpredictability of generative video from post-production to the shooting set. In traditional pipelines, physical shooting and post-production are separate stages; this workflow lets the two converse on set.

For film and television teams exploring generative video tools, this research offers a practical framework: rather than simply replacing shooting with AI, establish an awareness of generative capabilities' boundaries during the shooting stage, turning technical risk into a decidable, negotiable variable.

One-line Summary

Hybrid Cinematography moves generative reshooting's hallucination risk to the set, using 3D shot plans and AR tools to aid decisions.

Paragraph Summary

Generative video reshooting lets directors change camera movement in post, but the model can only "hallucinate" unrecorded content, and the risk often surfaces after leaving set. Hybrid Cinematography proposes a hybrid workflow: introducing an editable 3D shot plan and live-action proxy on set to compare target movement with existing coverage in real time and flag viewpoints that may lack data support. The system supports iterative plan adjustment, targeted pickups, or consciously accepting hallucination. Implementation includes a mobile AR on-set app and an offline video processing pipeline. A user study shows risk previsualization prompts filmmakers to adjust shot plans while exposing the tension between creative intent and generative hallucination, offering film teams a practical framework to turn technical risk into a decidable variable.