Production-Grade Multi-Agent Systems: When AIs Contradict Each Other

Most multi-agent system tutorials only teach you how to build agents. Almost none tell you how to watch them run. This article comes from a developer who has run multi-agent pipelines in production for six months, covering orchestration patterns, failure modes, and the distributed systems traps that frameworks never mention.

A Real Failure Scene

Last Tuesday, two components of a submit-verify automation pipeline contradicted each other. The submission agent reported a successful form submission—record number REF-2847, confirmation number APP-2847, status COMPLETE. The verification agent queried the external portal API and returned zero records, status NOT_SUBMITTED.

Both agents had evidence, both were describing the same event, only 60 seconds apart. The truth: the external portal had a 60-second processing delay between form submission and API visibility. The submission agent was correct at T+0, and the verification agent was also correct at T+5 seconds.

This isn't a hallucination problem—it's a distributed state problem layered with a confidence-presentation bug. Multi-agent systems are essentially distributed systems—processes passing messages, shared state observable at different moments, external APIs that can fail or lag, retry logic that can amplify a local failure into a total disaster. Every failure mode of microservices applies here, and frameworks won't defend you against any of them.

Defining a Production-Grade Agent System

The author's definition of a production-grade agent system: failures are contained, visible, and recoverable without human intervention 95% of the time. The remaining 5% that need human attention must surface immediately with enough context for an operator to resolve in 10 minutes.

The point isn't "successful agents," it's "agents that fail gracefully."

The production-readiness checklist includes four hard requirements: external calls must have circuit breakers (retry at most 2 times, then stop and alert); agents pass structured data between each other (IDs, timestamps, status codes—not natural language); irreversible operations must be followed by verification steps; every invocation needs a complete, queryable execution log.

Circuit Breaker is a classic distributed systems pattern for preventing cascading failures, originating from Michael Nygard's Release It!. It works like an electrical fuse: when an external dependency fails repeatedly past a threshold, the circuit breaker "trips" and subsequent requests fail fast without actually making the call, avoiding a retry storm that exhausts resources. Circuit breakers typically have three states—closed (normal pass-through), open (reject outright), and half-open (allow a few requests to probe recovery). The author implements a simplified version with a Postgres table: two consecutive failures stop that step and alert, until a human acknowledges and resets. In a multi-agent pipeline, the value of a circuit breaker is cutting the amplification loop of "failure → retry → worse failure," keeping local faults local.

Idempotency means that executing the same operation once or many times produces the same result. In distributed systems, due to network timeouts, lost acknowledgments, and similar issues, callers often can't confirm whether an operation succeeded and can only retry—and retries can cause duplicate side effects (such as duplicate form submissions or double charges). The standard solution is an idempotency key: the caller generates a unique fingerprint for each logical operation, the server first checks whether that fingerprint has already been processed, and if so returns the cached result without re-executing. Payment gateways (such as Stripe's Idempotency-Key request header) widely adopt this mechanism. For AI agent operations with side effects, idempotency key checks are almost the only means of preventing "duplicate submission."

How the Three Orchestration Patterns Actually Perform

Sequential mode suits scenarios involving irreversible operations—form submissions, email sends, database writes. The author's pipeline takes about 90 seconds per record. Slow, but n8n's execution logs provide a clean, ordered record of every step. When a submission silently fails, opening the execution history reveals the failed node and the exact error.

Parallel mode fits the ingestion phase—hitting five external sources simultaneously. The failure happened when the author tried to parallelize the submit-verify step for speed, directly producing the conflict described earlier. The rule now: parallel is safe only when subtasks are truly independent and the merge contract is defined before the first line of code is written.

Hierarchical mode hasn't been used at scale yet. The author's orchestrator, n8n, treats the entire workflow as a flat graph. The takeaway: hierarchical orchestration only makes sense when there's genuine ambiguity that requires judgment. If the logic can be expressed as a decision tree, use sequential mode with branches. And don't introduce hierarchical mode before you have strong observability—debugging a conversation between a supervisor agent and three sub-agents, with one agent drifting out of context midway, is not something you want to do at midnight.

The Circuit Breaker That Saved the Pipeline

After the conflict incident, the author added circuit breakers to every step touching an external system. The implementation is just a Postgres table recording workflow_id, step_id, fail_count, last_fail, cleared_by, and cleared_at.

Before each node runs, it queries the failure count. If two consecutive failures have occurred, it skips execution, sends an ntfy alert, and stops. On success, the count resets; on failure, it increments. After a human reviews the alert and confirms it's safe, they manually reset.

Two consecutive failures trigger shutdown. Since being built, this mechanism has fired four times, each time catching an issue that would have gotten worse with more retries: a rate limit that needed a 10-minute backoff, a malformed target URL that crashed browser automation, the portal processing delay described earlier, and a network timeout masking a home-server VPN disconnection.

The Key Frameworks Skip: Idempotency

If an agent submits a form but the orchestrator doesn't receive confirmation (network error, timeout), it retries—and then you submit twice. Every operation with side effects needs an idempotency key check before execution.

The author's approach: the submission agent writes a fingerprint (record_id + entity_id + date) to Postgres before attempting. If the fingerprint already exists, skip the submission and return the cached result. This is standard practice for payment systems, but it's almost never discussed in AI agent architecture articles.

Structured Data Passing is the core means of mitigating context drift. Natural language summaries lose detail and introduce ambiguity each time they pass through an agent, and after multiple hops the original observation may be unrecognizable. The alternative is to pass typed fields between agents—record ID, entity ID, timestamp, status code, confidence—rather than prose description. This aligns with the distributed systems philosophy of "pass events, not state": messages should be unambiguous, verifiable, replayable carriers of fact. The author's assertion that "the database is the source of truth and agent output is merely candidate fact" is an extension of this principle. When an agent says "submission succeeded," that's only candidate fact; it becomes truth only when the corresponding record is written to the database with a timestamp. Downstream agents should query the database rather than trust an upstream natural-language assertion.

What Breaks First

Context drift comes first. Agent A passes a summary to B, B summarizes further, and by C the nuances of A's original observation are gone. The mitigation is to pass structured data; the database is the source of truth and agent output is only candidate fact.

Retry amplification follows closely. An agent fails, the orchestrator retries, the agent fails differently because the world has changed. By the third attempt the error is completely different. The circuit breaker stops at the second, and the alert includes the context of the first failure.

Confident wrong answers rank third. Local models state wrong conclusions in the same format and tone as correct ones, with no built-in uncertainty signal. The author addresses this with verification steps after consequential operations, requiring an explicit confidence field in structured output, and routing low confidence to human review. The agent is never the last gate.

Agents are not sources of truth. They are producers of candidate facts. The database is the source of truth.

The Observability Tutorials Never Cover

Every multi-agent tutorial teaches you to build agents. Almost none teach you how to watch them run. The author uses two layers of tooling: LangFuse for LLM call tracing (prompt, response, latency, token count for every call to Ollama), and n8n's built-in execution history for workflow-level visibility.

When the conflict incident occurred, opening the n8n execution record showed both agents' responses side by side with their timestamps. The 58-second gap was right there in the logs. Without it, the author might have spent an hour hypothesizing; with it, the root cause was located in four minutes.

Without complete timestamped execution logs, you're not running a production system—you're running a demo that hasn't failed yet.

Reflections After Six Months

The portal incident eventually resolved itself—the submission had been there all along, the verification agent just queried too soon. The orchestrator now waits 60 seconds before the verification step for operations involving external portal submissions. It's a patch, but it works. Most production systems are full of such patches, each encoding a specific failure mode someone hit at a bad moment.

The pipeline currently running isn't elegant, but it's auditable. Every decision is in the logs, every failure has a circuit breaker, every irreversible operation has an idempotency check and verification step. It doesn't cascade-fail when one part fails, and it doesn't silently succeed when it actually failed.

Build observability first. Add circuit breakers before production. Define merge contracts before writing parallel steps. Agent prompts are the easy part. The plumbing of distributed systems is where multi-agent pipelines truly live or die—and almost nobody writes about it.

One-line Summary

Multi-agent systems are essentially distributed systems, and production readiness hinges on circuit breakers, idempotency, and observability.

Paragraph Summary

This article shares the experience of a developer who has run multi-agent pipelines in production for six months. Starting from a scene where submission and verification agents contradicted each other, the author argues that multi-agent systems are essentially distributed systems: every microservice failure mode applies and frameworks offer no defense. The article sets the standard for production-grade agents as "failing gracefully"—95% requiring no human intervention, 5% resolvable within ten minutes—and lays out four hard requirements: circuit breakers, structured data passing, verification steps after irreversible operations, and complete queryable execution logs. The author also compares the real-world performance of sequential, parallel, and hierarchical orchestration, shares concrete implementations of circuit breakers and idempotency keys using a Postgres table, and summarizes three major failure modes: context drift, retry amplification, and confident wrong answers. The core conclusion: agent prompts are the easy part—the plumbing of distributed systems is where the real battle lies.