5 Things You Must Do Before Deploying AI Agents to Production (With a Banking POC Case Study)

Why Most AI Agent Projects Die on the Road from Demo to Production

Sandy, Data and AI Technical Lead at Databricks, shared a battle-tested AI implementation methodology at the AI Engineer conference. Having previously spent five years as Principal Data and AI Architect at AWS, over the past two years he has been deeply involved in AI Agent deployment projects across heavily regulated industries such as B2B software and financial services—witnessing countless projects go from dazzling demos to failed, non-deployable ventures.

He observed a pattern that plays out at almost every enterprise: intense pressure from the top demanding that teams "do something with AI," so every discussion begins with "which model should we choose"—GPT or Claude? Teams argue fiercely for weeks over this. Once a model is selected, they hastily build features and produce a polished demo in a controlled environment. Leadership is satisfied, signs off, and pushes it to production. Then, weeks later, the problems erupt: "What is the AI actually doing? Why doesn't it answer like it did in the demo?" In the end, not only is there no ROI, but money and manpower are lost.

Sandy attributes the root causes of failure to three major gaps: the observability gap (inability to trace every AI decision), the evaluation gap (never having defined quantifiable metrics that truly matter to the business), and the governance gap (no one to hold accountable and no one managing data assets when AI fails in production). These three insights gave rise to the five-pillar framework he has repeatedly validated.

The Five Pillars: Things You Should Figure Out Before Writing Any Code

Sandy emphasizes that these five pillars must be considered before a project even kicks off, ideally built out step by step in sequence. They are: Evaluation, Observability, Data Foundation, Orchestration, and Governance.

The most counterintuitive point is this: model selection is not among these five pillars at all. In the banking case he shares later, the model wasn't chosen until week seven of an eight-week POC. This stands in stark contrast to how most teams "argue about the model from day one."

Pillar One: Evaluation—The AI System's "Requirements Specification"

The essence of evaluation is defining success—and it must be defined with numbers, not vague talk of "accuracy." Take a banking chatbot as an example: one of the core goals is "deflection"—having AI handle simple queries to reduce the burden on human agents. You need to be clear: Is 85% accuracy acceptable? How many false positives are tolerable? What's the target deflection rate?

Next comes building test cases—the so-called "golden dataset." The method is to communicate with domain experts and collect the actual responses human agents give to customer questions in real-world scenarios, especially those ambiguous edge cases. Then automate the entire evaluation pipeline: the AI provides answers, the system automatically compares and scores them against the test set, and monitors AI performance in production in real time.

Sandy breaks the evaluation architecture into three layers—an architectural decision that must be made upfront:

• Deterministic Layer: Format validation, email/phone regex, PII detection, intent classification, and other classic ML tasks. Cheap and mature, these should be tackled first.

• Semantic Layer (Non-deterministic): This involves groundedness (factual consistency), with the core technique being "LLM as Judge"—using an independent judge LLM to evaluate the main model's output across dimensions such as safety, relevance, and factual consistency.

  LLM as Judge Technical Background: This is a technical paradigm that uses a large language model to automatically evaluate the quality of another language model's output. Traditional NLP evaluation metrics (such as BLEU and ROUGE) rely on exact string matching and struggle to capture semantic-level accuracy and fluency. Since 2023, academia and industry have gradually reached consensus: for open-domain generation tasks, aligned large models exhibit high agreement with human experts in judging output quality. A typical implementation involves constructing a "judge prompt" that feeds the original question, reference answer, and the answer to be evaluated together into the judge model, asking it to score across dimensions like relevance, factual consistency, and safety compliance, along with reasoning. The main risks of this approach include the judge model's own biases (such as favoring longer responses), blind spots caused by homology with the model being evaluated, and high inference costs. Therefore, production practice typically combines it with a deterministic rule layer rather than relying on it alone. Databricks' MLflow can already run custom LLM judges automatically on trace data.

• Behavioral Layer: This is the most easily overlooked layer, focusing on whether tool calls are correct and whether the agent gets stuck in loops. For example: a user asks "What's my account balance?" and the AI gets the amount right—the first two layers both pass. But the behavioral layer would discover that the AI actually made three redundant calls to the database. It doesn't matter in a demo, but in production with thousands of queries per day, each making redundant calls, it becomes a costly expense.

Pillar Two: Observability—No Tracing Means No Production System

The core of observability is tracing—recording every decision the agent makes.

Sandy uses a real retail banking case to explain: a customer complains about being charged an overdraft fee and requests a waiver. With tracing enabled, you can clearly see the agent's complete decision chain: intent classification (latency, confidence) → connecting to the customer database API to fetch account details → retrieving the overdraft policy document from the RAG vector store to determine whether the complaint is valid → reasoning to generate a response → final guardrail check → replying to the customer.

RAG Vector Store Technical Background: Retrieval-Augmented Generation (RAG) is the mainstream architecture for addressing the "hallucination" and knowledge cutoff problems of large language models. Its core idea is: before generating an answer, first retrieve the most relevant document fragments from an external knowledge base and inject them into the prompt as context, so the model's answers are grounded and verifiable. The vector database is RAG's key infrastructure—it chunks documents, converts them into high-dimensional vectors via an embedding model, and builds an approximate nearest neighbor (ANN) index to support millisecond-level semantic similarity retrieval. Common vector databases include Pinecone, Weaviate, Milvus, and PostgreSQL's pgvector extension. The banking case failure mentioned later in the article—where a new policy document wasn't updated in the vector store, causing the AI to give outdated answers—is precisely the classic "knowledge base drift" problem in RAG system operations, revealing the necessity of continuous monitoring of the knowledge base update pipeline.

Without this visualized tracing system, when a customer raises a dispute, you have absolutely no way to verify what the AI did, and you're forced to "give the customer a discount to make the problem go away." This is exactly why heavily regulated industries like finance set tracing and observability as prerequisites for going to production—without them, there is no true production system.

Going further, tracing capabilities can be used for online monitoring: when redundant or failed calls are detected in production, fallback strategies can be applied automatically, such as "retry up to three times, then escalate to a human."

Pillar Three: Data Foundation—The Decisive Factor That Takes Up 60% of the Time

Sandy candidly admits that in a typical project, he spends 60% of his time on the data foundation. The reason runs deep: data has always been built for humans, and humans are forgiving—if a report has wrong data, you just ask someone to fix it. But agents are unforgiving; they will confidently give wrong answers while you remain completely unaware.

He divides data into two categories: problem data (the data needed to serve AI outputs, such as pre-training/post-training data and data accessed via APIs) and trace data (observability data). Especially when an enterprise runs hundreds of agents, trace data requires a complete strategy—how to design the schema, how to store it, and how to serve auditing and online monitoring.

In the Databricks tech stack, this data foundation is composed of application layers like Delta Lake + Unity Catalog + Mosaic AI.

Delta Lake and Unity Catalog Technical Background: Delta Lake is a storage layer format open-sourced by Databricks, built on top of Apache Parquet. It brings database features such as ACID transactions, schema evolution, time travel, and efficient DML operations (UPDATE/DELETE/MERGE) to the data lake, solving the reliability problem of traditional data lakes being "chaotic on write." It is the core foundation of Databricks' Lakehouse architecture. Unity Catalog is Databricks' unified data and AI governance layer, providing fine-grained access control across workspaces (column-level and row-level permissions), data lineage tracking, metadata management, and integration with external cloud storage. In AI Agent scenarios, the key value of Unity Catalog lies in: after adding business semantic descriptions to tables and columns, Text-to-SQL agents can understand data meaning when querying, significantly improving the accuracy of generated SQL; meanwhile, automatic tagging of PII columns can drive downstream data masking and access auditing workflows, providing a technical foundation for compliance governance.

A key value of Unity Catalog is this: when you add descriptions to tables and columns and tag PII columns, the AI gains rich context when querying these tables. Enterprises often use multiple frameworks like CrewAI and LangChain across multiple cloud platforms simultaneously, so they need a centralized layer to aggregate trace data, serving multiple scenarios such as operations dashboards, frontline support, and drift monitoring.

Pillar Four: Multi-Agent Orchestration—An Exponential Leap in Complexity

When a single agent runs well, you don't need to think about orchestration. But when you connect five agents, complexity rises exponentially—agents need to communicate in various ways and wait for each other's responses. Sandy introduces three core patterns:

• Orchestrator-Worker Pattern: A central orchestrator controls and dispatches tasks to agents based on their specialties. All requests pass through the orchestrator, providing central control—when something goes wrong, you just check the orchestrator logs.
• Choreographic Pattern: Each agent is autonomous and independent, listening for events it cares about via a message bus and able to run in parallel. Take a mortgage application: one agent looks up customer information, another looks up approval information—they process in parallel without depending on each other, significantly reducing latency because messages don't need to make round trips through an orchestrator.
• Human-in-the-loop: When an agent's confidence falls below a threshold, a human is brought into the workflow to review and decide.

Sandy notes that in real deployment, you also need to seriously consider state management, fault tolerance (failure recovery), and enterprise-scale scaling—including the Saga pattern, compensation pattern, and circuit breaker pattern.

Saga Pattern and Circuit Breaker Pattern Technical Background: The Saga pattern originates from the consistency problem of cross-service long transactions in distributed systems. In microservice architectures, the traditional two-phase commit (2PC) is costly, so the Saga pattern breaks a long transaction into a series of local transactions—each local transaction publishes an event upon completion to trigger the next step; if a step fails, a series of "compensating transactions" are executed to gradually roll back completed operations. In multi-agent orchestration scenarios, the Saga pattern is used to handle complex cross-agent business process rollbacks—for example, in a mortgage application, if the approval agent fails, the pre-approval record already written by the customer information agent needs to be undone. The Circuit Breaker pattern borrows from electrical engineering concepts: when a downstream service (such as an external API or model endpoint) fails consecutively beyond a threshold, the circuit breaker enters an "open" state, directly returning a degraded response without making further requests to prevent cascading failures from spreading; after a cooldown period, it enters a "half-open" state to probe for recovery. Both patterns are standard tools for ensuring system resilience in finance-grade AI agents.

Pillar Five: Governance—Who Do You Call When the System Crashes at 3 AM

What's discussed here is not data governance (that's a necessary prerequisite), but rather oversight and accountability from an AI perspective. Key points include: a complete audit trail (recording every action, connection, and request) and pre-validation of personal information (using named entity recognition to intercept PII)—in Sandy's case, this layer alone detected 47 PII leaks during the testing phase.

There are also two governance points that are often overlooked:

• Prompt Version Management: Prompts must be treated like code, incorporated into a formal change management process, rather than casually edited and committed to Git.
• Model Change Management: Model providers continuously upgrade their models, but public leaderboard benchmarks are often meaningless in your enterprise context. You must test new models with your own evaluation dataset and maintain the flexibility to switch models at any time, avoiding the risk of over-reliance on a single model.

Case Retrospective: How an Eight-Week POC Rescued a Failing Project

A retail bank built a chatbot 18 months ago, handling about 20,000 customer inquiries per month, of which 60% were simple queries (such as checking balances or asking about overdrafts). They spent 6 months and $85,000 on a POC that ultimately failed—precisely because they fell into the three gaps: no one knew why it failed in production, no one could measure it, and no one was accountable.

After Sandy's team took over, they set clear business goals: have AI handle 60% of simple queries at 85% accuracy, while meeting operational metrics like latency. The execution rhythm of the eight-week POC is highly instructive:

• Weeks 1-2: Build the evaluation layer. Collect 200 real human agent response cases, define success metrics, and set up an automated evaluation pipeline.
• Weeks 3-6: Build the data foundation layer. Validate API connections, tracing, distributed storage, and security—it was during this testing that the redundant API call problem was caught.
• Weeks 7-8: Only now begin selecting the model. Because there was an evaluation dataset, comparisons could be run quickly across different models and accuracy calculated, making the model decision fast and rational.

Six weeks after launch, a real failure validated the framework's value: the bank adjusted its interest rate policy and sent notices to customers, but customers couldn't get correct answers from the chatbot and started clicking "thumbs down." Because a measurement system was in place, the CSAT drop was detected immediately, and root cause analysis revealed: the new policy document hadn't been updated in the vector store, the embeddings hadn't taken effect, causing the AI to give outdated answers. With this system in place, the problem was quickly located and fixed. This failure scenario also confirms the importance of monitoring the knowledge base synchronization pipeline in RAG architecture—additions, deletions, and modifications to documents must have corresponding automated trigger mechanisms to ensure the vector store stays consistent with the source data.

Production Incident Runbook: Actions You Can Start Tomorrow

Sandy particularly emphasized an often-overlooked "production incident runbook" that defines the standard handling process when failures occur: Detect using the evaluation dashboard → Diagnose using tracing → Contain through prompt version rollback or handoff to a human → Fix using the test case library, then solidify new problems into a continuously growing evaluation suite.

In production, this runbook also needs to integrate with the enterprise's existing ITSM system to enable precise alerting.

ITSM System Integration Background: IT Service Management (ITSM) is a standardized management framework for enterprise IT operations, with ITIL (Information Technology Infrastructure Library) as the mainstream methodology, covering processes like incident management, problem management, change management, and the service desk. ServiceNow, Jira Service Management, and BMC Remedy are mainstream ITSM platforms. In the context of AI Agent productionization, integrating the incident runbook with ITSM means: AI system alerts are no longer just a Slack message, but automatically create tickets in ServiceNow with priorities, assignees, and SLA constraints, triggering standardized incident response processes and writing root cause analysis results and change records (such as prompt rollbacks) into the knowledge base as references for the next similar incident. This integration is especially critical for heavily regulated industries—regulators require enterprises to provide a complete AI decision audit trail and incident handling records, and the timestamps and operation logs of ITSM systems are an important part of this compliance evidence chain.

He also shared three practical lessons: First, the test case library is a continuously growing "living system" that requires dedicated governance and categorization (such as security, login, and other categories). Second, prompt Git commit messages must record in detail "why it was changed and what problem it fixed," otherwise it can't be traced. Third, behavioral layer evaluation is expensive, so you can test only a small subset in the CI pipeline and run the full set only when merging to the main branch, in order to control costs.

The first step you can take tomorrow: Define success from a business perspective (not a technical one), collect a few examples of "good answers" to build a dataset, and use simple Python code to set up an automated comparison pipeline. This is the correct starting point for AI Agents to truly move into production.

Key Takeaways