Three-Agent System in Production: Architecture Breakdown Behind a 78% Reduction in Database Diagnosis Time

Cornerstone OnDemand transformed their database operations from reactive firefighting to proactive automation using a three-agent system called Orion AI. A team of three engineers, over six months, leveraged Amazon Bedrock and the Strands Agents framework to compress average database diagnosis time from 45 minutes down to 10 minutes — a 78% reduction.

More importantly, this isn't a demo. It's a system running in the production environment of a major SaaS provider, handling real database incidents. For teams focused on shipping multi-agent systems in production, this case offers a rare and grounded reference point.

Orion AI Three-Agent System Architecture

The Problem: Manual Database Diagnosis at Scale

Cornerstone's database operations team was caught in a familiar loop: alert fires → engineer manually queries logs and metrics → writes diagnostic SQL → interprets results → decides on a fix → executes → documents the incident.

Each cycle averaged 45 minutes. On a multi-tenant SaaS platform fielding dozens of alerts per day, engineers were spending most of their time on repetitive diagnosis rather than meaningful architectural work.

What the team needed wasn't just alerting or monitoring tooling — they needed an automated solution that covered the entire loop from alert to remediation. That framing turned out to be the foundation of the project's success: the scope was narrow, and the boundaries were clear.

Architecture: Three Specialized Agents with Clear Boundaries

Orion AI decomposes database operations into three narrowly scoped agents, each responsible for exactly one thing.

Diagnosis Agent

Ingests alert and metrics data, identifies problem patterns, determines which database components need inspection, and hands off context to the query agent.

Query Generation Agent

Receives diagnostic context, generates safe read-only SQL queries, validates syntax before execution, and returns structured results to the diagnosis agent.

Remediation Agent

Receives confirmed diagnostic and query results, proposes remediation actions, requires mandatory human approval for destructive operations, executes approved fixes, and logs all actions for audit.

There's a core design decision worth highlighting: no agent directly calls another agent. All handoffs are managed by a state machine maintained by Strands Agents, with an orchestrator preserving context and audit logs across the workflow. This design avoids the two most common failure modes in multi-agent systems — circular dependencies and state conflicts.

Strands Agents is an open-source agent orchestration toolkit from Amazon, designed for building multi-step AI workflows on Amazon Bedrock. Its core mechanism abstracts each agent's behavior as "tool call + state transition," allowing developers to declaratively define which tools an agent can call and what format its outputs take — while the framework handles context passing and error retries across steps. Compared to chaining LLM API calls directly, Strands Agents' value lies in its built-in state persistence and orchestration logic. Engineers don't need to hand-write control flow like "if A succeeds, call B; otherwise roll back" — the framework automatically tracks completion state for each phase via workflow ID. This is the underlying mechanism that enables Orion AI to "resume from the last successful stage": state isn't stored in a particular agent's process memory, but persisted to external storage, so any crashed stage can be restarted and continue from where it left off.

Orchestration Flow and State Management

The workflow has three phases: Diagnosis → Query Generation & Execution → Remediation Proposal. Each agent receives structured input and returns structured output; the orchestrator links all artifacts together via a single workflow ID.

Phase	Input	Output	Stored State
Diagnosis	Alert data, metrics	Problem classification, affected components	Alert ID, timestamp, severity, component ID
Query Generation	Diagnostic context	SQL queries, expected schema	Query text, validation status, execution plan
Remediation	Diagnosis + query results	Action plan, approval requirements	Proposed actions, approval status, execution log

This state management approach has a practical benefit: if an agent fails or times out, the system can resume from the last successful phase rather than re-running expensive diagnostic or query operations. For pay-per-use LLM inference, that saves both cost and time.

Security Boundaries and Failure Modes

The biggest risk in multi-agent systems is error cascading. Orion AI has three layers of protection.

Query safety: All generated SQL passes through a parser that rejects DML/DDL statements; query timeouts prevent runaway operations; results are truncated at a configurable row limit; connection pooling protects against database overload.

Remediation gates: Destructive operations require explicit human approval; approval requests include full context and predicted impact; approved actions execute within transaction boundaries; rollback procedures are generated alongside remediation plans.

Observability: Every agent invocation logs inputs, outputs, and latency; failed calls trigger alerts; query execution plans are recorded before execution; remediation actions are written to a separate audit database.

The most common failure mode is the diagnosis agent misclassifying a problem, leading to irrelevant queries being generated. The system's response is elegant: the query agent is allowed to return a "insufficient data" result, which triggers a fallback to human diagnosis rather than blindly executing an incorrect remediation.

Error cascading in multi-agent systems refers to one agent's erroneous output being accepted as trusted input by downstream agents, ultimately producing consequences far more severe than a single point of failure. A typical scenario: the diagnosis agent misclassifies a high-load issue as a connection leak; the query agent generates cleanup SQL targeting the connection table; the remediation agent then proposes forcibly terminating connections. Each step looks locally "reasonable," but the entire chain is built on a false premise — and executing it could directly interrupt normal traffic. Preventing error cascading isn't just about validation at each layer; it's about allowing downstream agents to actively refuse to accept uncertain inputs. The mechanism in Orion AI where the query agent returns "insufficient data" and triggers a human fallback is exactly this philosophy in action: better to interrupt the automated flow than to pass an uncertain conclusion to a remediation agent with write access.

Deployment: Designed for Bursty Load

Cornerstone runs Orion AI on AWS infrastructure: Amazon Bedrock handles LLM inference (using Claude models), Strands Agents manages orchestration and state, Amazon RDS stores audit logs and workflow state, AWS Lambda executes the agents, and Amazon EventBridge handles alert ingestion.

The team's rationale for choosing Lambda was straightforward — database incidents are bursty. The system sits idle most of the time, and when an alert fires, Lambda can scale to handle multiple concurrent workflows without pre-provisioned capacity.

Permission isolation is also thorough: each agent runs in its own Lambda function with its own IAM role. The diagnosis agent has read-only access to metrics; the query agent has read-only database credentials; the remediation agent has write permissions but only acts after approval. This principle of least privilege is exactly what production-grade systems should look like.

Amazon Bedrock is AWS's fully managed foundation model service, letting developers call models from multiple providers — including Anthropic Claude, Meta Llama, and others — through a unified API, without deploying their own inference infrastructure. For production systems like Orion AI, Bedrock's core advantages are: pay-per-call pricing (zero cost at idle, consistent with Lambda's design philosophy), native integration with AWS IAM, and built-in model invocation logging for auditing. Cornerstone chose Claude models for diagnostic reasoning and SQL generation tasks, with Claude's strong performance on structured output and complex instruction-following being a key consideration. Worth noting: the article flags a "model dependency" risk — a Bedrock outage degrades Orion AI entirely to manual diagnosis. This is an inherent trade-off of managed services, and the team mitigated it with runbooks rather than technical failover. That's a pragmatic engineering call.

Results and Trade-offs

Six months after launch, the numbers are compelling: diagnosis time dropped from 45 minutes to 10 minutes (78% reduction); 60% of incidents now resolve without human intervention; engineers are freed from repetitive diagnosis work; the system automatically handles 40+ alerts per day.

But the team was candid about the trade-offs they accepted:

Latency for safety: The system takes 10 minutes instead of 2 because it includes validation steps, approval gates, and structured handoffs. The team explicitly chose correctness over speed.

Limited remediation scope: The remediation agent only handles a predefined set of actions (restart services, flush caches, adjust connection pools). Complex fixes still require a human engineer.

Model dependency: The system depends on Bedrock's availability. If Bedrock goes down, Orion AI falls back to manual diagnosis. The team maintains health monitoring and manual fallback runbooks for exactly this scenario.

Technical Judgment: When to Use This Pattern

This case draws a clear picture of when this pattern fits.

Good fit: You have repetitive operational workflows that follow predictable patterns; you can draw clear boundaries between diagnosis, analysis, and remediation; you need audit trails and approval gates for compliance; your team is small but needs to scale operational capacity.

Poor fit: Your incidents are highly variable and demand creative problem-solving; you can't tolerate 10-minute diagnosis latency; your database operations require complex multi-step transactions; you lack the infrastructure to run orchestrated agent workflows.

Cornerstone's case shows that multi-agent systems can work in production — provided you design for failure modes, maintain clear agent boundaries, and accept the trade-off between speed and safety. A three-person team was able to ship this in six months because they stayed focused on a narrow problem domain and leveraged existing frameworks (Bedrock, Strands Agents) rather than building an orchestration system from scratch. That may be the most valuable lesson this case offers to any engineering team thinking about production multi-agent systems.