Building a Video Generation Agent: Lessons Learned from Brief to Storyboard

A Seemingly Simple Goal

Automatically turning a one-line brief into a complete video is the ultimate dream many creators have for AI Agents. Recently, a developer shared their hands-on experience building such a pipeline on Reddit. The goal was straightforward: take a one-sentence brief like "30-second morning vibe promo for a coffee brand" and have the Agent automatically output a structured list of shots, then call a video generation model for each one.

The overall design was clear and typical: a planning model breaks down the brief into a script, outputting a JSON array of shots — each containing fields like characters, actions, camera movement, duration, and mood. Each shot is then passed to a media generation tool for rendering. This is a classic Agent workflow pattern: separation of the planning layer and execution layer.

The theoretical roots of this architecture trace back to the classical AI concept of "task planning and execution separation," widely adopted in the LLM era across Agent frameworks like ReAct (Reasoning + Acting) and Plan-and-Execute. The planning layer is typically handled by a more capable reasoning model responsible for understanding goals, decomposing tasks, and generating execution plans; the execution layer is handled by specialized tools or smaller models performing the actual operations. The advantage of this architecture is that both layers can be independently iterated and upgraded — but its limitations are equally clear. Errors in the planning layer cascade into the execution layer, and if the data format contract between the two layers (such as a JSON Schema) is inconsistent, the entire pipeline collapses.

The developer expected the planning model to return "boringly strict" structured data. For the coffee promo example, one shot might be described as: "Young founder holding coffee, pushing open the café door at sunrise, camera slowly pushing in from outside the window, 4 seconds, warm and quiet atmosphere" — and another as: "Barista pouring a latte, steam rising, close-up of cup being set down, slight handheld shake, 3 seconds, cozy and premium atmosphere."

Reality: Malformed Output Is the Biggest Enemy

The vision was appealing, but as the developer wryly noted, "the first things to break were completely predictable." These are classic pitfalls that nearly everyone building LLM Agents encounters:

• Returning prose instead of valid JSON: Models naturally tend to "speak human," rather than emit machine-parseable structures.
• Overly verbose shot descriptions: Individual shot descriptions turned into full paragraphs, completely unusable as video generation prompts.
• Missing required fields: For example, the critical duration field would be omitted.
• Hallucinated parameters: The Agent would "invent" tool parameters that didn't exist, attempting to call nonexistent interfaces.

These issues reveal a core tension: Large language models are naturally inclined to generate fluent text, while Agent workflows require strict, predictable, machine-readable structured output. When we ask a model to be both creative (come up with good shots) and rule-abiding (strictly follow a schema), conflict is inevitable.

Engineering Discipline to Tame the Model

Faced with this chaos, the developer applied a series of pragmatic engineering constraints — approaches worth borrowing for any Agent developer.

Strict JSON Schema

The most critical move was using a strict JSON Schema with additionalProperties: false.

JSON Schema is a declarative standard for describing and validating JSON data structures (based on IETF draft specifications). In the world of LLM structured output, it acts as a "contract" — precisely defining the field names, data types, value ranges, and required fields that model output must satisfy. The additionalProperties: false constraint rejects any fields not explicitly declared in the schema, directly preventing the model from "inventing" extra fields — any property outside the schema causes validation to fail. This constraint is especially important for LLMs, which have a tendency toward "creative divergence" and may spontaneously add seemingly reasonable extra properties (like a self-invented mood_score field), causing downstream parsers to error out. It's worth noting that major model APIs including OpenAI and Anthropic now support passing a JSON Schema at call time to constrain output format. Some implementations — such as OpenAI's Structured Outputs — even enforce Schema constraints at the token sampling level, fundamentally eliminating format errors. This is an effective way to constrain LLM structured output at the source.

Hard Limits on Count and Length

The developer set a hard cap of 6 shots, preventing the model from expanding the script without bound. Each field was also limited to approximately 15 words, with validation occurring before any tool calls. This detail is particularly important — it ensures shot descriptions always remain concise enough to serve as video prompts, rather than sprawling prose paragraphs.

Retry Loop with Feedback

Finally, a retry loop with a maximum of 3 attempts. When validation fails, the system feeds the specific error messages back to the model, allowing it to regenerate based on the error context.

This mechanism is essentially a form of "supervised self-correction," grounded theoretically in reinforcement learning's feedback signal concept and recent LLM self-improvement research paradigms like Self-Refine and Reflexion. Unlike uninformed retries, "diagnostic feedback" injects specific errors (e.g., "duration field missing," "description exceeds 15 words") as new context into the next conversation turn, enabling targeted corrections with a significantly higher success rate. Setting a limit of 3 attempts is an engineering pragmatism: it avoids infinite loops burning excessive tokens while covering most format fluctuations caused by randomness — typically 1–2 retries are sufficient. Failures beyond 3 attempts usually indicate deeper prompt design issues requiring human investigation. This "error-feedback-driven self-correction" mechanism is the mainstream approach in current Agent engineering for handling format issues, capable of resolving the vast majority of structural failures.

MCP Protocol: Unified Tool Abstraction for the Media Layer

On the media generation side, the developer's architectural choice was notably forward-thinking. Instead of writing separate API wrappers for each video/image model, he used Atlas Cloud's MCP (Model Context Protocol) server to expose image and video models as uniformly callable tools.

MCP is an open protocol released by Anthropic in late 2024, designed to standardize how AI models interact with external tools and data sources. Before MCP, each AI application needed to write its own adapter layer for different tools, creating enormous amounts of duplicated engineering work. MCP's core design follows a client-server architecture: the MCP Server exposes domain-specific tools and resources (such as video generation APIs, database queries), while the MCP Client (typically an Agent framework) calls these tools through a unified protocol without needing to know about underlying implementation details. This design is analogous to an operating system's "driver" abstraction — application layers don't need to write dedicated code for each piece of hardware.

This is precisely where the MCP protocol's value lies — it provides Agents with a unified tool-calling interface, freeing them from the specific API differences of each underlying model. In a video generation pipeline, an MCP Server can wrap the video generation APIs of different vendors like Runway, Kling, and Sora as standard tools, and the Agent simply calls a generic interface like generate_video(prompt, duration) — model switching at the underlying level is completely transparent to the Agent. When a pipeline needs to integrate multiple different generation models, this abstraction layer dramatically reduces maintenance costs and integration complexity.

The developer also referenced open-source repositories like awesome-agent-skills and AtlasCloudAI/atlas-cloud-skills to study how others encapsulate reusable Agent workflows. This reflects a broader industry trend: Agent development is shifting from "reinventing the wheel" toward "reusing standardized skill modules."

An Unsolved Challenge: Pacing

Despite schema validation and retry loops resolving most format issues, the developer candidly acknowledged one stubborn remaining challenge: the model's judgment on pacing is often off. For instance, it might allocate 8 seconds to a shot that should only be 3 seconds.

This problem is highly representative. Video pacing is a judgment that depends heavily on industry experience — professional editors instinctively know that "establishing mood shots typically run 3–5 seconds, while climactic action sequences can be cut down to 1–2 second flashes." This kind of tacit knowledge appears infrequently in professional film production texts and is difficult for language models to fully internalize. Technically, this is an "out-of-distribution generalization" challenge — the model's training data may cover the syntactic rules of video duration (duration is an integer number of seconds), but lacks sufficient contextualized judgment examples (specific content types corresponding to specific duration ranges). Structural errors can be solved with engineering constraints, but semantic-level "taste" and "judgment" remain difficult to correct with simple rules. Shot pacing involves a holistic assessment of content importance, visual impact, and narrative tension — exactly where current LLMs fall short in professional creative domains. Potential engineering solutions include injecting "recommended duration ranges per shot type" into the prompt as prior constraints, introducing a professional editing rules library, or using few-shot examples to demonstrate well-paced storyboards to the model — but all of these require explicit encoding of professional knowledge and cannot fully replace human intuition. A validator can ensure the duration field exists and contains a valid number, but it cannot judge whether that number is actually reasonable.

Four Takeaways for Agent Developers

This case, though small in scope, distills the core lessons of building a practical video generation Agent pipeline:

First, never trust the model to spontaneously output structured data. Strict JSON Schema, additionalProperties: false, field-level validation — these constraints are not optional. They are requirements for any production-grade Agent.

Second, retry loops with error feedback are the most cost-effective solution for handling format instability. Rather than chasing a perfect first attempt, design a loop that can self-correct.

Third, use standard protocols like MCP for tool-layer abstraction. This makes your Agent architecture cleaner and more extensible, avoiding repeated effort on multi-model integration.

Fourth, maintain a clear-eyed perspective. Current technology can reliably solve "format correctness," but "content quality" and "professional judgment" still require human involvement or more refined guidance. Agents can help you scaffold the structure — but decisions that require professional intuition still need humans, at least for now.

Key Takeaways