DiffusionGemma Character Motion Transfer in Practice: A ComfyUI Workflow Breakdown

An Intriguing Character Motion Transfer Experiment

In the field of AI video generation, combining a static character image with real motion footage has long been a direction that creators are eager to explore. Recently, a Reddit developer shared his character motion transfer experiment based on a DiffusionGemma custom node, demonstrating the possibility of "swapping the face but keeping the motion" within a ComfyUI workflow.

The concept is remarkably straightforward: given a static reference image (a girl) and a video of a different person in motion, a single prompt—"put the girl from the image into the video"—allows the system to replace the subject's identity while preserving the motion from the original video.

Breaking Down the Tech Stack: DiffusionGemma + LTX 2.3

Workflow Composition

The technical combination in this experiment offers considerable reference value. The author used DiffusionGemma's custom node in ComfyUI, paired with the LTX 2.3 video generation model. The inputs to the entire pipeline are extremely minimal:

• A static reference image (source of character identity)
• A reference video containing the target motion (source of motion)
• A single natural-language prompt

About DiffusionGemma: DiffusionGemma is a hybrid framework that combines Google's Gemma language model architecture with diffusion models. To understand the significance of this architecture, one must first understand its technical lineage: Gemma is a family of lightweight open-source language models released by Google in 2024, with parameter sizes ranging from 2B to 7B, designed with inference efficiency in mind. The core motivation for combining it with diffusion models lies in the fact that traditional diffusion models typically rely on text encoders from contrastive learning models like CLIP for text conditioning injection. However, CLIP's training objective is image-text alignment rather than semantic understanding, which creates clear limitations when handling complex multi-step instructions or descriptions of multi-entity relationships. Introducing a large language model as the conditioning encoder allows full use of its semantic reasoning capabilities learned from massive text corpora, enabling more precise parsing of compound instructions like "put the girl from the image into the video" that involve entity reference and cross-modal operations.

The core idea of diffusion models is to generate high-quality images or videos from random noise through a step-by-step denoising process. By incorporating the semantic understanding capabilities of large language models into this process, the model can more precisely understand the mapping between natural-language instructions and multi-modal inputs. This fusion architecture has a natural advantage when handling complex multi-condition generation tasks—the language model handles semantic parsing and condition alignment, while the diffusion model handles high-quality content generation. Distributed as a custom node for ComfyUI, DiffusionGemma lowers the barrier to entry for researchers and creators, allowing them to invoke its multi-modal reference generation capabilities without building a training environment from scratch.

About LTX 2.3: The LTX (Latent Text-to-Video) series of models is an open-source video generation model developed by Lightricks. Its core design is based on the DiT (Diffusion Transformer) architecture, performing diffusion modeling on video sequences in latent space. DiT was proposed by Peebles and Xie in 2022, and the core advantage of introducing the Transformer architecture into the diffusion process is that the self-attention mechanism naturally supports global information interaction across sequences of arbitrary length.

Looking at the evolution of the technical architecture, UNet was originally proposed by Ronneberger et al. in 2015 for medical image segmentation tasks. Its encoder-decoder structure with skip connections achieved notable success in image generation and served as the core architecture for the Stable Diffusion 1.x/2.x series. The fundamental difference between the two lies in the receptive field: the receptive field of convolutional operations expands only to a limited extent with layer depth, and modeling long-range dependencies relies on stacking more layers; the computational complexity of self-attention is O(n²), but each layer can establish direct dependencies between any two positions in the sequence. Compared to earlier UNet-based video generation models, the convolutional layers of UNet are essentially local receptive field operations that rely on temporal-dimension convolutions to progressively propagate inter-frame information. DiT, on the other hand, allows frame 1 and frame 16 to interact directly through attention, modeling temporal relationships spanning dozens of frames directly in a single forward pass, thereby maintaining longer-range temporal consistency and being less prone to inter-frame drift when handling complex motion. Notably, recent mainstream video generation models such as Sora and CogVideoX have also widely adopted DiT or its variant architectures, marking the transition of mainstream video generation technology from the UNet era to the Transformer era.

Furthermore, LTX's design choice to model in latent space also carries important engineering significance: the pixel space of raw video has extremely high dimensionality. By compressing video into a low-dimensional latent variable space using a VAE (Variational Autoencoder) before running the diffusion process, computational cost can be reduced by orders of magnitude. The way a VAE works is: the encoder compresses high-dimensional pixel-space video frames into low-dimensional continuous latent variables, the diffusion process runs in this compressed space, and the decoder then restores the denoised latents to pixel-level output. A 512×512 resolution, 24-frame video has raw data volume of about 18MB; after VAE compression to a 64×64×4 latent space, the data volume can be reduced to about 1/96 of the original, making it possible to run video generation models on consumer-grade GPUs. The LTX series has specifically designed a 3D VAE with spatiotemporal joint compression tailored to video characteristics, capable of compressing along both spatial and temporal dimensions simultaneously. Typically, after compressing by 8 to 16 times before running the diffusion process, computational cost can be reduced by orders of magnitude—this paradigm has become the standard design for mainstream video generation models. LTX 2.3 is a more recent version in this series, with improvements in generation resolution, inference speed, and temporal coherence. Its open-source nature has also made it a popular base model for video generation in the ComfyUI community.

About ComfyUI: ComfyUI is one of the most influential open-source graphical workflow tools in the field of AI image and video generation. Its core design philosophy is to break down the generation pipeline into a visual "node graph." Each node represents an independent functional module—such as model loading, condition encoding, samplers, and decoders—and users can build complex generation pipelines simply by connecting the data flow between nodes, without writing any code. This modular design has fostered the rapid emergence of numerous cutting-edge experiments, and the DiffusionGemma node is a typical product of this community innovation model.

This "image + video" dual-reference mode is a typical paradigm of current reference-based video generation. It's worth noting that the development of this research direction can be traced back to early work in the field of Image Animation: around 2019, keypoint-driven methods (such as the First Order Motion Model) were the first to enable animation generation from a single image, though generation quality was limited by the training instability of the GAN architecture. After the rise of diffusion models, IP-Adapter (2023) proposed a lightweight image prompt adapter that injects the semantic features of the reference image into the diffusion process through decoupled cross-attention layers, becoming a foundational component for much subsequent work and laying the technical groundwork for the multi-modal reference generation paradigm used in this experiment. The system needs to decouple "identity features" from "motion features" during generation—the former extracted from the static image, the latter preserved from the video.

It's worth noting that the complete decoupling of appearance and motion is mathematically an ill-posed problem—a type of problem where the number of unknowns exceeds the number of constraint equations, resulting in non-unique solutions. This concept originates from the three conditions of a "well-posed problem" proposed by Jacques Hadamard in 1902: the solution exists, the solution is unique, and the solution depends continuously on the initial data. The reason decoupling character appearance and motion constitutes an ill-posed problem is fundamentally this: from a single static reference image, we cannot know the complete appearance manifold of the character under different lighting, different viewpoints, and occlusion states; and from the driving video, we cannot be sure how much motion detail should be preserved and how much identity information should be replaced.

This dilemma can be understood quantitatively at the level of information content: a 512×512 reference image contains about 786K pixels of appearance information, but the complete appearance description required for a continuously posed person sequence in the target video theoretically needs to cover infinitely many combinations of viewpoints and poses, forming an infinite-dimensional appearance manifold. Existing methods essentially fit a local approximation of this infinite-dimensional manifold with limited reference information, and their generalization error is unavoidable. Any solution must introduce additional assumptions—for example, assuming that the textures in the reference image can be transferred to all poses, or assuming that the motion structure in the driving video is independent of identity. These assumptions clearly fail when there are significant body shape differences or severe occlusion, and this is also the theoretical ceiling shared by all current reference-driven video generation approaches.

Representative works in academia include: Animate Anyone (Alibaba, 2023), which first systematically introduced ReferenceNet—a reference encoder structurally symmetric to the main UNet—injecting the appearance features of the reference image into the generation process of each frame through a cross-frame attention mechanism. The core innovation of ReferenceNet lies in the cross-frame attention mechanism: in the self-attention computation of each layer of the main network, not only the features of the current frame serve as Key and Value in the computation, but the features from the corresponding layer of the reference image are also concatenated in, so that the generation process of each frame can "see" the appearance information of the reference image. Compared to simple condition injection (such as CLIP image embeddings), this mechanism preserves finer-grained appearance detail, at the cost of increased memory usage and inference time. MagicAnimate (ByteDance, 2023), which appeared almost concurrently with Animate Anyone, was also based on temporal consistency modeling, but adopted a different temporal attention design strategy. The rapid iteration in this field reflects the enormous potential of diffusion models in conditional generation control.

Champ (2024) built on this by introducing the SMPL 3D human parametric model as the driving signal, making cross-body-shape motion transfer possible. SMPL (Skinned Multi-Person Linear Model) is a parametric 3D human shape and pose model proposed by the Max Planck Institute, decomposing the human body shape into shape parameters β (controlling body build—height and weight, typically 10 dimensions) and pose parameters θ (controlling the rotation of 72 joints). Even when the person in the reference image differs significantly in body shape from the person in the driving video, the driving pose can be "adapted" to the target body shape by adjusting the β parameters, thereby avoiding motion distortion caused by body shape mismatch. MimicMotion, on the other hand, shifted its focus to optimizing inference efficiency and temporal smoothness. Commercial products such as Runway's Act One and Kling's character consistency features are also exploring the same direction. This experiment is precisely a concrete implementation of this research direction within the open-source ecosystem.

Complete Preservation of Downstream Control Signals

The author particularly emphasized that while completing the identity replacement, the workflow still preserved the integrity of downstream control signals such as pose, depth, and canny (edges). This means that ControlNet-type conditional control was not disrupted throughout the transfer process, and the generated result remained structurally consistent with the original video.

How ControlNet and Multiple Control Signals Work: ControlNet is a conditional control network proposed by Lvmin Zhang in 2023. Its core innovation lies in injecting additional structural conditions into the generation process of a pretrained diffusion model, enabling precise structural control without disrupting the original model weights. Its architectural design borrows from the "twin network" concept: an encoder with an identical structure but independent weights runs in parallel with the main generation network, extracting intermediate features from the control signal, and then injecting these features into the corresponding layers of the main network through zero-initialized convolutional layers (zero convolution).

The design of zero-initialized layers is key to architectural stability: in the early stages of training, if control signal features are directly injected into a pretrained diffusion model, the randomly initialized injection layers would produce large random gradients, disrupting the carefully trained weight distribution of the original model and causing a sharp drop in generation quality or even training collapse. Zero-initialized layers (with both weights and biases initialized to zero) contribute nothing to the injected signal at the start of training, ensuring that the model behaves exactly as it would without control signals in the early stages of training; as training progresses, the weights of these layers slowly learn from zero, gradually introducing control constraints and achieving a "progressive" injection of capabilities. This design philosophy is philosophically consistent with the strategy in LoRA (Low-Rank Adaptation) of initializing the incremental weight matrix to zero, and it is also an important reason why ControlNet can quickly converge to high-quality control effects on a relatively small amount of paired conditioning data.

Pose control typically uses OpenPose or DWPose to extract human skeleton keypoints, constraining body motion trajectories; depth control estimates the 3D structure of the scene using models like MiDaS or Depth Anything, locking spatial positional relationships; Canny edges extract the contour line information of the image, constraining the overall shape outline. The stacked use of these three control signals forms a multi-layered structural locking mechanism, which is key to preventing "motion drift" or "structural collapse" after character replacement.

In an actual ComfyUI workflow, the stacking of multiple ControlNet signals imposes constraints of varying strength on the intermediate feature layers of the diffusion model through their respective conditioning_scale parameters. Overly high control weights will cause the generated result to "over-conform" to the control signals, losing the naturalness of appearance details; overly low weights render the control signals ineffective, causing body structure to collapse. In character motion transfer scenarios, a common practical strategy is: set the pose signal weight relatively high (0.8–1.2) to lock the key skeletal motion trajectories, set the depth signal weight in the middle range (0.5–0.8) to maintain spatial hierarchy relationships, and set the canny signal weight relatively conservatively (0.3–0.6) as an auxiliary contour constraint only, to avoid over-restricting the generation freedom of texture details.

This point is particularly critical in practical applications. The biggest challenge in motion transfer is often not "swapping the face," but how to avoid motion distortion, structural collapse, or temporal jitter after replacing the subject. The ability to maintain multiple control signals indicates that this workflow performs robustly in structural constraints.

Why This Experiment Is Worth Watching

Extremely Low Interaction Barrier

Traditional video character replacement often requires complex keyframe annotation, frame-by-frame masking, or dedicated face-swapping models. This experiment, however, accomplished the separation and recombination of identity and motion with just a single natural-language prompt. This "prompt-driven" interaction method dramatically lowers the barrier to creation.

The Evolution Direction of Multi-Modal Reference Generation

From a more macro perspective, such experiments reflect that AI video generation is evolving from "text-to-video" to "multi-modal reference generation." Pure text-to-video struggles to precisely control motion and identity—because natural language is essentially a discrete symbolic system that has difficulty accurately describing continuous motion trajectories and fine-grained appearance features. Introducing images and video as references gives creators stronger control over results, using high-dimensional visual information directly as generation conditions and bypassing the bottleneck of linguistic description. This is also the direction being jointly explored by commercial products like Runway and Pika as well as numerous open-source workflows.

Current State and Limitations

The author frankly points out that these custom nodes are still in early development ("still very much a work in progress"). The main current limitations include:

• Stability and generation consistency are not yet reliable
• Performance on complex motion or multi-person scenes remains to be verified
• Metrics such as temporal coherence still need further optimization

These limitations are common in the field of reference-driven video generation: when the character in the reference image differs significantly from the person in the driving video in body proportions, clothing style, or range of motion, there is often an inherent tension between the stable injection of identity features and the precise alignment of motion structure—this is essentially a concrete manifestation of the aforementioned ill-posed problem. Understood from an information-theoretic perspective, the appearance prior information provided by a single reference image is far from sufficient to deterministically cover the appearance states under all target poses. This can be understood through a simple analogy: from a single frontal photo, we cannot deterministically predict the details of a person's back texture while doing a cartwheel—the model must make a trade-off between limited priors and generation diversity, and pose combinations outside the training distribution will inevitably cause quality degradation. This is a challenge faced jointly by both academia and industry. Therefore, this is more of a proof of concept than a mature solution that can be directly put into production. The author also actively sought feedback in the community, asking whether there were better methods for reference-based video processing, reflecting the collaborative spirit that is a hallmark of the open-source community.

Practical Advice for Creators

For creators hoping to try similar workflows, this experiment offers several ideas worth drawing on:

1. Modular Thinking: Leveraging ComfyUI's node-based design, flexibly combining different models and control signals allows rapid iteration of experimental directions. The visual nature of the node graph also makes it easy to troubleshoot problematic nodes, reducing debugging costs.
2. Control Signal Stacking: The combined use of ControlNet conditions such as pose, depth, and canny is the core means of ensuring structural stability in motion transfer. It's recommended to adjust the weight ratios of each control signal according to the specific scenario—for scenes emphasizing body motion, appropriately increase the weight of the pose signal (in the 0.8–1.2 range), while lowering depth and canny according to scene complexity to avoid over-constraining and affecting appearance naturalness. The three signals correspond to constraints in three dimensions—skeletal motion, spatial hierarchy, and shape contour, respectively—and understanding their respective semantic meanings helps in tuning parameters in a targeted way.
3. Continuously Track Emerging Nodes: While nodes like DiffusionGemma are not yet mature, they represent the cutting-edge exploration of the community and are worth watching for future iterations. Similar projects worth watching also include the various ComfyUI custom node ecosystems centered around video consistency and character control.

Conclusion

This DiffusionGemma-based character motion transfer experiment, though a preliminary attempt, clearly demonstrates the vitality of the open-source community in reference-based video generation. As the capabilities of the underlying models (such as the LTX series) continue to improve—whether it's the enhancement of temporal modeling capabilities by the DiT architecture (thanks to Transformer's global attention capturing long-sequence inter-frame relationships better), the refinement of multi-modal condition injection mechanisms, or the optimization of video compression efficiency by 3D VAE—and as custom nodes gradually mature, the vision of achieving high-quality character replacement with just "one image and one video" may not be far off. For developers and content creators interested in AI video creation, such experiments are worth continued attention.