POV Generation: A New Frontier for AI Image Models and the Open-Source Challenge

What Is POV Generation

AI image generation models have recently unlocked a fascinating new capability: re-generating an image from the perspective of a specific person within a scene (Point of View, or POV). In plain terms, you feed the model a photo containing multiple people, and it outputs what the scene would look like through the eyes of one particular person — as if you were standing in their exact position, looking outward.

This is fundamentally different from traditional image editing. It requires the model not only to understand the content of the image, but also to reason spatially — determining where the target person is in the scene, which direction they're facing, and then inferring what objects, people, and backgrounds would appear in their field of view. In essence, this is an implicit 3D scene reconstruction and viewpoint transformation task.

According to a developer's hands-on report shared on Reddit, Meta's Muse Image handles this type of task impressively. Given an image containing a man in a scene, the model can generate a new image from that man's point of view. Nano Banana, another widely discussed model in the community, has also been confirmed to perform well on this task.

reddit source: POV Generation

Commercial Models Can Do It — But at a Steep Price

Currently, the ability to reliably perform POV generation is limited to a handful of closed-source commercial models. Both Meta Muse Image and Nano Banana have been shown to have this capability, but a critical problem follows: the inference costs of these models are prohibitively high for large-scale deployment.

For developers who want to integrate POV generation into their products or need to process images in bulk, paying per-image through an expensive commercial API is clearly not a sustainable solution. This is precisely why the developer turned to open-source alternatives — could an open-source model that's self-hostable and cost-controllable replicate similar results?

Three Attempts, Three Failures: The Open-Source Journey

The developer systematically tested three technical approaches. None of them worked — and the results are instructive for anyone else attempting this.

Path 1: Use Open-Source Image Editing Models Directly

He tested several well-regarded open-source image models, including Qwen Image Edit, FLUX.2 9B Base, and Hunyuan Image 3.0 Instruct. These models have solid reputations for general image editing tasks, but they all failed on the specific task of POV generation — none could generate a scene as viewed from a particular person's perspective within the image.

This reveals that current open-source image models lack targeted optimization for spatial reasoning tasks like viewpoint transformation. They excel at local editing, style transfer, and content replacement, but struggle with higher-order tasks that require understanding the geometric relationships within a scene.

Path 2: Generate a Text Description of What the Person Sees First

The second approach was a multi-step pipeline: first use a language model to describe "what this person is looking at," then feed that description to an image generation model. He used the Gemma model for the description step.

The result: the model tended to drift toward describing the person's own appearance rather than what they see. This exposes a common weakness in multimodal models — when it comes to coreference resolution and perspective switching, models easily confuse "describing the subject" with "describing what the subject sees," failing to accurately imagine the scene from a third-party spatial standpoint.

Path 3: Use a Video Model to Switch Camera Perspective

The third approach used the video generation model Wan 2.2 TI2V, using instructions to shift the camera to the target person's viewpoint. This attempt failed entirely as well. The developer acknowledged, however, that he hadn't yet tested larger-scale video models, so whether this path is truly a dead end remains to be seen.

Wan 2.2 TI2V (Text-Image to Video) is a multimodal generation model that takes an image and text as input and generates a sequence of continuous video frames. The core advantage of the TI2V paradigm is that video models are naturally exposed to large amounts of camera motion sequences during training, giving them some prior knowledge of viewpoint changes like "the camera moving from point A to point B." However, there is a fundamental difference in task complexity between "camera pan/dolly" and "fully switching to the first-person perspective of another person in the scene." The former is a continuous camera movement; the latter requires the model to understand the specific spatial position and orientation of another subject in the scene and reconstruct their monocular field of view — placing far greater demands on the video model's spatial reasoning capabilities. This explains why even video models with some viewpoint priors still fail at precise POV switching.

The Technical Core of the Problem

Looking across all three failures, the fundamental difficulty of POV generation comes down to a lack of spatial-geometric understanding and viewpoint transformation capability. This task implicitly requires the model to:

Locate the target person's 3D position within the scene
Infer the orientation of their head/body to determine their viewing direction
Reconstruct scene content that is occluded by the person's body and invisible in the original image
Maintain spatial consistency of other elements in the scene

The reason closed-source models like Muse Image and Nano Banana can accomplish this is very likely because their training data and objectives included large amounts of multi-view, 3D-consistent samples. General-purpose open-source models lack this specialized training and naturally struggle to generalize to such tasks.

NeRF (Neural Radiance Field) and 3D Gaussian Splatting are the two most mainstream implicit 3D scene reconstruction methods today. NeRF trains a neural network that maps spatial coordinates and viewing directions to color and volume density, enabling rendering from any new viewpoint — though its training and inference speeds are relatively slow. 3D Gaussian Splatting represents a scene using millions of 3D Gaussian ellipsoids with position, color, and opacity attributes, dramatically improving rendering speed and becoming the dominant alternative for novel view synthesis in recent years. The core advantage of both methods is that they separate "scene geometry" from "appearance" into distinct models, making it a deterministic geometric operation to render a new view from any specified position and orientation — rather than relying on a model's implicit spatial imagination. Incorporating these methods into a POV generation pipeline could theoretically bypass the fundamental spatial consistency limitations of end-to-end generative models.

Promising Directions to Explore

While the original post hasn't found a mature open-source solution yet, a few directions seem worth exploring based on the nature of the task:

Introduce explicit 3D intermediate representations: First use monocular depth estimation or scene reconstruction models (such as NeRF or 3D Gaussian Splatting approaches) to recover scene geometry, then render a new viewpoint from the target person's position, and finally use an image model to fill in details. This "geometry first, generation second" pipeline may be more controllable than end-to-end approaches.

More precise prompt engineering: In the text description step, using structured prompts that explicitly instruct the model to "not describe this person, only describe the objects, directions, and distances in front of them" may help correct the drifting behavior seen with Gemma-class models.

Test larger video models: Video models inherently carry priors around camera motion and continuous viewpoint change. Larger-parameter video models may perform better at camera perspective switching — a direction the original author also acknowledged as worth validating.

Task-specific fine-tuning: If sufficient multi-view paired data is available, applying LoRA or full-parameter fine-tuning to an open-source image editing model specifically to train viewpoint transformation capability could be the most direct and effective route.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method that works by injecting low-rank decomposition matrices alongside the weight matrices of a pretrained model, training only these newly added parameters (typically just 0.1%–1% of the original model's parameter count), thereby adapting a large model to a specific task at very low computational cost. For a task like POV viewpoint transformation that requires specialized capability, the feasibility of LoRA fine-tuning depends heavily on the quality and quantity of training data. Ideal paired data would include "image pairs of the same scene from different people's perspectives" — constructed, for example, through 3D game engines, synthetic data pipelines, or multi-camera capture systems. If such data is difficult to obtain, another option is to first pre-train on synthetic scenes (such as indoor/outdoor environments rendered in Blender) to generate large-scale viewpoint pair data, then fine-tune with a small amount of real imagery — a common data construction strategy in current multi-view generation research.

Closing Thoughts

POV generation represents a microcosm of the broader evolution of image generation from "2D content manipulation" to "3D spatial understanding." Commercial models have already cracked this problem, but at high cost; the open-source ecosystem is still in the exploration phase, with mainstream models unable to handle the task directly. For developers tracking the frontier of generative AI, this represents both an unfilled gap and a genuine technical opportunity worth pursuing.