Agent + ComfyUI in Practice: Building Seamless AI Long-Video Stitching Nodes from Scratch

Title

Agent + ComfyUI in Practice: Building Seamless AI Long-Video Stitching Nodes from Scratch

Meta Description

A Bilibili creator shares hands-on experience using an Agent with ComfyUI to achieve seamless AI long-video stitching — covering the 22-frame Latent reuse principle, custom node development, VRAM optimization strategies, and how to distill Skills through dialogue with AI to build a personal video workflow.

Article Content

In the field of AI video generation, consistency and coherence in long videos have always been core challenges. Through a detailed hands-on video, a Bilibili creator demonstrated how to leverage an Agent (AI agent) to understand and modify ComfyUI's long-video stitching technology from scratch, ultimately building a custom node tailored to personal work habits. The entire process is not just a technical breakdown but also a methodology demonstration of "collaborating with AI."

The Core of Long-Video Stitching: Clever Reuse of 22-Frame Latent

At the start of the video, the creator showed an impressive result: two video segments seamlessly joined, with consistent color tone and space, no obvious degradation, and the character's motion dynamics perfectly inherited. The principle isn't complicated — extract the 22-frame Latent from the end of the previous video segment and feed it directly as the reference condition input for sampling the next segment, forcing these 22 frames to appear 1:1 in the first 22 frames of the new video. After generating multiple segments, simply remove the overlapping 22 frames to achieve perfect stitching. The creator admits that the so-called "context" technique is essentially the staged transfer of Latent in the latent space — not some deep magic.

The key to this mechanism is that it bypasses the pain points of traditional "last-frame stitching." Traditional methods use only a single image as the starting frame for the next segment; the model cannot inherit the momentum from the end of the previous video, and because the reference condition must be converted to Latent through an R2V node, color and image quality are inevitably polluted, causing the video to degrade segment by segment and produce a jarring "hiccup" effect. By contrast, feeding 22 consecutive Latent frames directly eliminates the conversion process, fundamentally solving the problems of color shift and lost dynamics.

Latent (latent space representation) is the compressed data form that diffusion models use internally. Models like Stable Diffusion do not generate images or videos directly in pixel space; instead, they first encode the frame into a low-dimensional latent tensor via a VAE (variational autoencoder), perform the diffusion and denoising process in this latent space, and finally decode back to pixel space. Because the latent space has far fewer dimensions than pixel space, computation is greatly reduced while preserving semantic and motion information. The term "22-frame Latent" refers to the latent sequence obtained by encoding 22 consecutive frames — it naturally carries the motion state and temporal relationships between frames. It is precisely because this Latent segment is reused directly as a condition input that the model can inherit the momentum from the end of the previous video, unlike a single image which loses dynamic information. Understanding this intermediate latent representation is a prerequisite for understanding the entire long-video stitching solution.

The R2V (Reference-to-Video) node is a component in ComfyUI workflows used to convert reference images or reference frames into video generation conditions. Its role is to encode externally provided reference material into Latent conditions acceptable to the model, then feed them into the sampler. The problem is that the reference image must first undergo encoding conversion, a process that introduces color shift and quality loss; when stitching a long video segment by segment, each segment goes through such a conversion, and errors accumulate progressively, ultimately manifesting as color drift and a jarring "hiccup." The 22-frame Latent approach lets the continuous latent space bypass the R2V encoding step and go straight into the sampler, avoiding degradation from repeated conversion at the source — this is the fundamental engineering difference from traditional last-frame stitching.

From "Newbie" to Building Custom Nodes: How the Agent Decomposes Complex Workflows

The creator was initially intimidated by long-video technology because the workflows shared online were densely packed with nodes and had a high learning curve. It wasn't until he saw another creator's first-person long take of a mecha climbing a building made with this technique that he resolved to master it. He downloaded a workflow from a foreign author on Civitai, but found its design logic seriously incompatible with his own habits — the foreign author still retained first/last-frame thinking, used numerous node parameters, and provided no detailed tutorial.

So he turned to an Agent, threw the workflow page at the AI, and asked it to interpret the core technique and extract the key parts to adapt to his own pipeline. The Agent quickly read through the webpage, sorted out the implementation logic, and helped scan for locally missing plugins and models, automatically downloading them from GitHub and Hugging Face (the creator emphasized the necessity of "magic" internet access). Along the way, the Agent also revealed a tendency to be "too clever for its own good": for example, the creator only asked for a small node with a frame-count selection option, but the Agent generated three nodes, one of which had confused logic and wasted a lot of debugging time. This led to the key principle for collaborating with Agents — the user must clearly define requirements, control the pace, and know how to validate results.

Ultimately, with the Agent's help, the creator successfully added a Latent input interface to the R2V reference condition input and added a frame-count dropdown option (default 22 frames, with 73 frames etc. selectable), automatically aligning audio and video. The core nodes were streamlined to just two: reference condition processing and sampling output, with all redundant nodes removed.

VRAM Optimization and Segmentation Strategy: The Key to Advanced Usage

The creator specifically noted that while the 22-frame Latent approach is great, it's not a silver bullet. Because each segment must carry the previous segment's Latent as reference, VRAM usage rises with the number of segments, so each segment's duration must be shorter than an independently generated one. He recommends shortening individual segment duration but increasing resolution (e.g., from 1.4x to 1.6x or even 2.0x), trading duration for quality — especially friendly for users with limited VRAM.

He also questioned whether 22 frames (less than a second) is enough to fully understand the previous video's content. The Agent explained that these 22 frames are consecutive with no frame dropping and can inherit the motion state. But in testing, the creator still chose 73 frames (about 3 seconds) to get more sufficient context, even though a 16G GPU might struggle. He emphasized that the real advanced approach is segment-based control: each segment's reference conditions, sampling steps, and duration should be dynamically adjusted based on actual generation results, rather than locking all parameters at once like the original author did. "You might generate four segments and all four could be wrong — one bad apple spoils the whole barrel." He warned that the H3 model has spatial errors and gacha-style degradation, so quality must be ensured through segment-by-segment retries and corrections.

In node optimization, the creator also solved the tiling problem during latent upscaling. The original node defaulted to a tile size of 32, which on a 16G GPU still pushed VRAM usage above 98%, causing the GPU to idle while waiting for memory swapping and actually slowing things down. He expanded the tiling parameter to options like 24, 16, and 8; testing showed 24 avoids the VRAM threshold and runs faster. He recommends users "probe downward" based on their own VRAM to find the balance point.

Latent Upscale and Tiling are common techniques for increasing resolution in video generation: first complete sampling at low resolution, then upscale the result in latent space and add details. But the upscaling stage must process the latent tensor of the entire video at once, requiring extremely high VRAM. Tiling splits the frame into small blocks processed one at a time, trading time for VRAM; smaller tiles mean fewer tensors processed per pass and lower peak VRAM, but greater seam handling and scheduling overhead between tiles. In the original article, changing the tile size from the default 32 to 24 is exactly finding the balance on a 16G GPU between "VRAM not overflowing" and "acceptable processing speed" — too large a tile maxes out VRAM and causes the GPU to idle waiting for memory swapping, while too small a tile causes frequent scheduling that slows things down. Users should probe downward based on their own VRAM to find their own threshold.

The Art of Conversing with an Agent: Distilling Skills So AI Adapts to Your Habits

The Agents the creator used include DeepSeek^Ad V4.1 (free/low-cost), WorkBuild, and local small models (such as 1825B Flashnex). He complained that DeepSeek is "terrible" at writing prompts — the period-drama female assassin transported to a campus revenge plot it generated had no polish at all, even featuring floating actions like "flying through the air," completely unusable for testing stitching effects. But for node design and code comprehension, DeepSeek performed excellently.

He summarized a set of methods for collaborating with Agents: after each task, have the Agent summarize the thinking constraints and validation methods from that session into a "Skill" to serve as mandatory rules for subsequent conversations, avoiding repeated mistakes. As Skills accumulate, they need periodic consolidation to prevent slowdowns. "Every time you do this kind of task, summarize these skills, debate its ideas, make it follow your thinking and habits — gradually it becomes branded into its body like a mark."

Finally, the creator showcased his optimized save node, adding a "delete first N frames" option (default: no deletion), corresponding to the frame-count selection, for easier post-production compositing. The entire workflow can now generate and stitch two video segments; going forward he plans to further simplify the process, create a dedicated Skill for writing prompts, and explore nodes for managing multiple video segments simultaneously.

The value of this video lies not only in implementing a long-video technique but also in demonstrating how "an ordinary person + Agent" can decompose a black box and customize tools. As the creator said: "The hardest technical problems have already been solved by the core project's authors. What you need to do is understand the principles, then let AI help you adapt them to your own habits."

Skill (skill distillation) here refers to a context-engineering practice for AI agents: solidifying the constraints, thinking steps, and acceptance criteria validated in a given task into reusable text rules, loaded as system prompts or mandatory instructions in subsequent conversations. Its value lies in countering large language models' "context forgetting" and "repeated mistakes" — the model itself has no persistent memory across sessions, and without explicit distillation, similar errors will recur in every new task. As Skill entries grow, prompts get longer and inference slows, so periodic consolidation and deduplication are needed to retain high-value rules. This practice essentially uses a manually maintained rule base to compensate for the model's lack of memory, gradually aligning AI output with the user's personal habits and workflow instead of starting from scratch each time.

One-line Summary

A Bilibili creator uses an Agent to modify ComfyUI, achieving seamless long-video stitching via 22-frame Latent reuse and building custom nodes.

Paragraph Summary

The article documents a Bilibili creator's full process of using an AI agent to master ComfyUI's long-video stitching technique from scratch. The core principle is reusing the continuous Latent of the last 22 frames of the previous segment as the reference condition for the next, bypassing R2V conversion and fundamentally solving the color shift and lost momentum of traditional last-frame stitching. Faced with a complex workflow from a foreign author, the creator had the Agent interpret the logic, scan for missing plugins and models, and download them automatically, ultimately streamlining the nodes to just reference condition processing and sampling output, plus a frame-count dropdown. The article also discusses VRAM optimization strategies — shortening segment duration, increasing resolution, and adjusting latent upscale tiling from 32 to 24 to avoid the VRAM threshold — as well as ensuring quality through segment-based dynamic parameter adjustment. Finally, it summarizes the methodology for collaborating with Agents: define requirements clearly, control the pace, and distill Skills so AI adapts to personal habits.