Depth Map and OpenPose Extraction Workflow: A Practical Guide to Controllable AI Video Generation

Starting with a Football Meme

Recently, a Reddit creator shared a Depth Map and OpenPose pose extraction workflow they built. The origin story is quite interesting: the creator is a fan of Norwegian football star Haaland, and since the algorithm kept pushing related AI memes their way, they decided to build one themselves.

Behind this seemingly casual creation lies an increasingly important trend in AI video generation—how to precisely control the camera movement and character poses of generated videos through structured reference information. The creator open-sourced the workflow, so anyone can simply import a video and obtain a clean depth video and an OpenPose pose video.

What Are Depth and OpenPose Extraction

Depth Map

A depth map uses grayscale values to represent the distance from each pixel in the frame to the camera. In AI video generation, depth information helps the model understand three-dimensional spatial structure and camera motion trajectories, ensuring the generated frames maintain consistent perspective relationships and a coherent sense of camera movement.

The technology for generating depth maps has undergone a profound evolution from hardware sensors to purely algorithmic estimation. Early depth information relied primarily on RGB-D cameras (such as Intel RealSense and Microsoft Kinect), which measured distance directly through structured light or Time-of-Flight (ToF) principles—structured light approaches project a known pattern via an infrared emitter and calculate distance based on the degree of deformation, while ToF approaches measure the round-trip time of light pulses. However, such hardware solutions are costly, highly susceptible to ambient light interference, and difficult to use for extracting depth from existing video footage in post-production.

In AI video generation scenarios, algorithmic approaches based on Monocular Depth Estimation are more commonly used—representative models include Intel's MiDaS, Meta's DPT (Dense Prediction Transformer), and the recently prominent Depth Anything series (jointly released by ByteDance and the University of Hong Kong). These models learn to infer pixel-level relative depth relationships from a single RGB image through self-supervised or cross-dataset joint training on millions of images, requiring no dedicated hardware. The extracted depth map plays the role of a "spatial anchor" in video generation: the generation model interprets grayscale gradients to understand the layered relationship between foreground and background, then maintains consistent perspective distortion, depth-of-field blur, and camera push-pull rhythm when synthesizing new frames—effectively avoiding issues such as object clipping or unnatural camera movement.

OpenPose Pose Recognition

OpenPose is a mainstream human keypoint detection solution that can identify skeletal joints, hands, and facial keypoints, presenting them as a line-based skeleton. When used as reference input, AI models can precisely reconstruct the movements and poses of characters in the original video, greatly improving the coherence of character animation.

OpenPose was proposed by the Perceptual Computing Lab at Carnegie Mellon University around 2017. It was the first open-source framework capable of real-time multi-person keypoint detection, and its paper, published at CVPR 2017, quickly became the benchmark reference in this field. Its core technical innovation lies in Part Affinity Fields (PAF)—a two-dimensional vector field that encodes the orientation and position of limbs. Traditional top-down methods require first detecting each person's bounding box and then analyzing them one by one, and in crowded scenes the computational cost grows linearly with the number of people. In contrast, the PAF approach uses a single forward pass through a convolutional neural network to simultaneously predict heatmaps for all keypoints and the directional vector field of limb connections, then combines keypoints into complete skeletons through a bipartite graph matching algorithm—achieving a dual breakthrough in both computational efficiency and accuracy. The standard OpenPose model can output 18 to 25 body keypoints (including the neck, shoulders, elbows, wrists, hips, knees, ankles, etc.), as well as 21 keypoints for the hands and 70 keypoints for the face. The widespread adoption of the ControlNet framework in 2023 further cemented the status of the OpenPose skeleton diagram—in this framework, the generation model no longer needs to understand complex human texture appearances but instead directly extracts motion information from concise, colored skeleton lines, greatly reducing the visual noise interference of the control signal and making pose-guided generation more precise and stable.

Using both types of information together allows you to lock down both "how the camera moves" and "how the characters move," significantly improving the overall controllability of the generated video.

Using It Together with Seedance 2

The creator specifically noted that the extracted depth video and OpenPose video can be directly used as reference videos for Seedance 2, effectively preserving the camera movement and character poses of the original video.

Seedance 2 is a video generation model from ByteDance and a representative product of the current "Video-to-Video" guided generation paradigm. To understand this paradigm, one must first grasp its underlying conditional generation mechanism in Diffusion Models: the training process of diffusion models consists of two stages—"noising" and "denoising." The former gradually adds Gaussian noise to the data until it becomes completely random, while the latter trains the network to learn to inversely predict the noise at each step under given conditions. During inference, the "condition" is no longer limited to text prompts—additional visual signals (such as reference videos, depth maps, and skeleton diagrams) can be introduced to constrain the direction of denoising. A reference video (whether the original video, a depth video, or a pose video) technically serves as a Structural Prior—it uses explicit spatial geometry or motion information to tell the model "how the three-dimensional relationships should be" or "what the motion trajectory should be," while high-frequency information such as texture details, lighting style, and character appearance is freely filled in by the text prompt and the model's own visual knowledge accumulated during pretraining. This generation approach of "structural constraint + stylistic freedom" is precisely the core advantage of structured reference videos over pure text descriptions.

Seedance 2 supports guiding the generation direction through reference material. Compared with the limitations of pure text prompts, which struggle to describe complex camera movements, structured reference videos provide a "visual constraint" that makes the generated results more stable and controllable.

The workflow of "extract structural information → use it as reference input → guide generation" is becoming the mainstream paradigm for AI video creation. Compared with the early "blind box" generation that relied entirely on text prompts, creators now have far greater control over the final image.

The Practical Value of the Workflow

Lowering the Barrier to Creation

Building a complete pipeline for depth extraction and pose recognition from scratch is not easy—it involves chaining together multiple model nodes and tuning parameters. Node-based workflow editors like ComfyUI have greatly lowered this barrier—creators don't need to write code; by dragging and dropping nodes, they can chain depth estimation models, pose detection models, and video preprocessing nodes into a complete pipeline, then package and share it in JSON format. After the creator packaged and shared the workflow, users only need to import a video to obtain both types of clean reference material with a single click, saving the tedious setup process.

Maintaining Consistency in Camera Movement and Pose

One of the most common pain points in AI video creation is Temporal Inconsistency—jumpy character movements and chaotic camera motion. The technical root of this problem lies in the fact that early image diffusion models processed each frame as an independent unit, lacking explicit modeling of continuity along the time dimension: the denoising process for each frame was independent of the others, causing subtle yet cumulatively noticeable deviations in character pose and background texture between adjacent frames. Mainstream solutions have evolved along two parallel technical paths: at the architectural level, by inserting Temporal Attention layers into U-Nets or Transformers, or by extending 2D convolutions into 3D convolutions, the model can perceive the features of adjacent frames when generating each frame; at the inference level, by introducing external structural signals to provide explicit motion constraints—which is precisely where depth maps and OpenPose reference videos come into play. Depth map sequences provide the model with a frame-consistent spatial hierarchy reference, significantly reducing camera shake and abrupt perspective changes, while continuous OpenPose skeleton sequences primarily constrain the temporal reasonableness of joint motion trajectories, preventing issues such as limb misalignment, sudden changes in joint angles, or motion "ghosting" (a typical artifact where motion blur and residual images overlap). Working together, the two make generated videos noticeably more coherent and natural along the time dimension.

The Community Value of Open-Source Sharing

A small project driven by fan interest, delivered as a tool that benefits a broader group of creators, is a textbook example of bottom-up innovation in the AI open-source community. This culture of sharing "reproducible workflows" has given rise to numerous specialized tools for vertical scenarios. Communities such as Reddit's r/StableDiffusion and r/ComfyUI, along with platforms like Civitai, have become the main channels for distributing such workflows, allowing the technical accumulation of individual creators to quickly benefit the global community and to be continuously refined through iterative feedback.

From "Generation" to "Control"

This case reflects the direction in which AI video generation technology is evolving: creators' focus is shifting from "can it be generated" to "can it be generated the way I want."

The depth map and OpenPose extraction workflow essentially "translates" a reference video into a structured language that the model can more easily understand, then hands it to the generation model to fill in the details. This process is highly consistent with the concept of the Animatic in traditional film and television production—animators first use a rough sequence of line-art sketches to determine camera blocking, character actions, and rhythm before moving on to the fine-production stage. The depth map corresponds to the spatial composition and camera movement design in the animatic, the OpenPose skeleton sequence corresponds to the character's action choreography, and the generation model ultimately plays the role of "rendering the storyboard into the finished footage." As more intermediate-layer tools emerge, AI video creation will increasingly resemble traditional film and television production—with storyboards, blocking, and precise action design, rather than random gacha-style pulls.

Readers who want to try it out can select a short video with clear camera movement and character actions, extract reference information through such a workflow, then feed it into a model that supports reference videos, such as Seedance 2, to personally experience the efficiency gains brought by "controllable generation."

Key Takeaways