Multi-Agent Systems Keep Losing the Plot? Write the Plan to a File

A developer who is both a father of two and an AI engineer has been building a team of AI agents in his spare time to create soothing videos for young children. His experience reveals a subtle but critical problem in multi-agent systems: when the plan only lives in the chat context, agents quietly forget the details. His solution, in his own words, is almost embarrassingly boring — write the plan to a file.

A Side Project Exposes a Real Pain Point

His motivation was simple: most content made for toddlers is too loud, too fast, and overstimulating. He wanted to make something different for his two kids — slow-paced songs, soft colors, and three consistent characters (a felt hedgehog, a rabbit, and a little duck).

Instead of animating things by hand, he assembled a small team of AI agents, each responsible for exactly one thing:

• One plans the story beats
• One generates static images
• One reviews every static image before paying to convert it into video
• One stitches the clips into a finished episode

This kind of single-responsibility decomposition is a sound multi-agent design pattern. But the real trouble surfaced in how they shared the plan.

Keeping the Plan in Chat Was the First Mistake

In the early weeks, the entire plan "lived" in the conversation context. The author candidly admits this was a mistake.

The root cause lies in how large language models work: agents summarize older messages, quietly dropping details in the process. The result was a series of strange "drifts":

• A second rabbit that should never exist suddenly appeared on a shelf
• Props that had been explicitly banned crept back in
• Scene order rewrote itself between sessions

Worse, he often didn't notice these errors until after paying to generate video — which happens to be the most expensive step in the entire pipeline. In other words, context forgetting wasn't just a quality problem; it translated directly into money lost.

This points to a systemic weakness in current multi-agent systems: chat context is not a reliable state store. It gets truncated, summarized, and reinterpreted — it cannot serve as the authoritative source of truth for a long-running project.

When processing long conversations, LLMs do not retain all historical messages indefinitely. As the context length approaches the model's context window limit, the system typically auto-compresses or summarizes earlier messages to make room for new content. This is inherently lossy compression: summaries preserve the gist but drop specifics — like "only one rabbit on the shelf" or "no balloon props allowed." The problem is even more severe in multi-agent scenarios, because each agent has its own independent context window and may be reading a different "history," leading to divergent interpretations of the same plan. This phenomenon is commonly referred to as "context drift" or "instruction forgetting" in the industry, and is one of the primary engineering challenges in LLM-based long-horizon task systems.

The Boring but Effective Fix: Make Files the Contract

The author's fix involves zero cleverness. Now, each episode ships with a few "boring files":

• Who the characters are and what they look like
• The order of scenes
• What is allowed and forbidden on screen
• A checklist that the QA agent runs against every static image

The core rule is a single sentence: agents read the files, not the chat. Chat is for communication; the files are the contract.

This shift produced several immediate improvements:

QA became actionable. The review agent can now say things like "that second rabbit on the shelf must go" — a concrete instruction rather than a vague "something feels off." The standard shifted from subjective impression to objective verification against a file.

Failures get caught at the cheap stage. Any static image must match the file before it can enter the expensive video generation step. Errors are caught before money is spent.

New agents are plug-and-play. A newly added agent can get up to speed immediately just by reading the files, without having to scroll back through 40 messages of chat history.

This approach maps directly to a well-established software engineering concept: Single Source of Truth (SSOT). In traditional software development, configuration files, database schemas, or API contracts play this role — all modules read rules from one authoritative source rather than each maintaining their own copy. Writing the plan to files is essentially transplanting this mature engineering principle into AI agent collaboration. A related idea is Infrastructure as Code: making agreements that previously lived in someone's head or in verbal communication explicit, version-controllable, reviewable, and rollback-able text files. For AI systems, files carry an additional advantage: they are not subject to reinterpretation due to a model's sampling randomness. Every read returns exactly the same content — they are deterministic.

Lessons for Multi-Agent Engineering

This case comes from a hobbyist children's video project, but the principles it surfaces apply to any long-horizon, multi-agent collaboration.

Separate state from conversation. Keeping "the plan" and "the communication" distinct is the key to avoiding context drift. Conversation is inherently ephemeral and lossy; project constraints need a persistent, lossless, verifiable carrier.

Constraints should be verifiable, not just felt. Writing out an explicit allow/deny list transforms QA from fuzzy judgment into checklist verification — essentially introducing deterministic "unit tests" into an agent system.

Set your gates at the cheapest stage. Placing verification at the image stage rather than the video stage is a cost-aware pipeline design: the more expensive the operation, the more rigorous the upstream checks should be.

The author closes with a question worth asking across the industry: If you're running multiple agents on a long-term project, where do you store the plan? Files, a database — or do you just trust the chat memory?

Based on the mistakes he made, the answer is clear: don't trust chat memory. Giving your agent team a readable, verifiable, persistent "contract" is almost always more reliable than any sophisticated memory mechanism.

The author's point about "setting gates at the cheapest stage" echoes a classic principle from software quality assurance: the later a defect is found, the more expensive it is to fix. In traditional software, this is called Shift-Left Testing — moving testing and validation as early as possible in the development lifecycle. The same logic applies to AI generation pipelines: text validation < image validation < video generation; the further downstream, the higher the unit cost, so errors missed at one stage get amplified at the next. The author's decision to anchor the QA step at the static image stage is an explicit application of this principle. For any AI pipeline involving API call fees, cloud compute, or external services, identifying the relative cost of each stage and deploying checks at low-cost nodes is foundational engineering practice for controlling overall operational costs.