Data Analysis Agent in Production: A Complete Guide from Demo to Controlled Deployment

From Magic Demo to Production Line

Every Monday morning, generating a sales weekly report means manually shuttling data between databases, spreadsheets, charts, and documents — a massive drain on human effort. Plenty of AI Agent demos look impressive — buttons get clicked automatically, results materialize on their own — but when it comes to real production deployment, they fall apart. This article follows a data analysis Agent called Report031 to show how it completes the full cycle of "plan, invoke tools, pause for approval, and deliver a traceable report" — rather than stacking up a pile of "technically impressive-looking building blocks."

The core idea is straightforward: first define what questions the report needs to answer and what counts as valid evidence, then think about state graphs and tools. At the entry point, Report031 is decomposed into five deliverables — data scope, metric definitions, conclusions, charts, and approval — each with a clear completion criterion. This "results-first" design philosophy is the first line that separates toy demos from production-grade Agent systems.

State Graph Design: Confining the LLM's Freedom Within Inspectable Boundaries

The LLM core will always produce non-deterministic outputs, but the outer track must be deterministic: which nodes are allowed, how much budget can be spent in one pass, when to stop, and where to go on failure. Here's a precise insight: the state graph doesn't think for the model — it confines the model's degrees of freedom within inspectable boundaries.

With every step forward, Report031 leaves behind three pieces of information: current state, input content, and next destination — preventing the entire chain from becoming an undebuggable black box. The pipeline is divided into five layers:

• FastAPI layer: receives incoming requests
• LangGraph layer: orchestrates state transitions
• Memory & planning layer: preserves context and the execution blueprint
• MCP tool layer: provides data capabilities
• Model service layer: handles understanding and interpretation

Observability probes run through every layer. The value of layering isn't drawing pretty architecture diagrams — it's ensuring that permissions, versions, errors, and replacement boundaries all have clear ownership.

MCP Tool Contracts and the Principle of Least Privilege

Many people mistakenly think tools are just "plugin buttons for the model," but in practice, every tool — querying, metric calculation, charting, report generation — must declare its structured inputs, structured outputs, purpose, and error boundaries. The model first discovers the contract, then submits parameters.

MCP (Model Context Protocol) provides a discoverable tool protocol, but it doesn't automatically achieve business decoupling. Real decoupling still depends on whether deployment, permissions, version compatibility, and failure handling are clearly designed.

A classic example is data access permissions: when Report031 reaches the data gate, it only receives a "read-only key." The query tool also validates parameters, restricts allowed fields, caps maximum row counts, enforces execution timeouts, records an audit trail, and minimizes exposure of sensitive information. Write operations, deletions, and arbitrary SQL are simply not made available. Least privilege isn't a security slogan — it's a verifiable capability boundary within the tool contract — a lesson many teams only learn after a production incident.

Planning and Execution: Expose Gaps Before Taking Action

The Planner begins by spreading out the "blueprint" on the workbench: what's in scope this week, which fields the metrics depend on, what the tool budget is, and what counts as a completed result. The key here is that a plan isn't a nicely written natural-language description — it's a set of executable steps and stopping conditions.

This design allows the Agent to surface gaps before taking any action, and to re-plan or stop early based on new evidence. Once execution begins, the analysis Agent can only invoke tools along the approved route: query results first enter an "evidence tray," after which metrics are calculated, charts are generated, and explanations are written.

The model cannot write facts that don't exist in the database. All invocation parameters, result summaries, durations, and errors go into the trace log. This means every conclusion Report031 produces carries its source from the moment it's generated.

Tiered Approval Mechanism: Finding the Balance Between Speed and Control

Not every step requires a human to click approve. The risk-tiered delegation strategy looks like this:

• Read-only queries, deterministic calculations, charting → auto-approved
• Conclusions involving business judgment → enter the approval gate, where the reviewer can see the input, evidence, and the action that will be executed
• High-risk capabilities like write operations and deletions → simply not provided

Approval nodes are implemented via Interrupt calls, with checkpoints saved. On resume, the same Thread ID is used, Command Resume sends back the approval result, and a persistent Check Pointer is configured. There's an important pitfall worth highlighting: in-memory checkpoints cannot guarantee recovery after a restart, and switching threads opens a completely new state.

The Idempotency Trap in Interrupt Recovery

An even subtler issue: nodes containing interrupts may restart from the beginning when resumed, meaning code before Interrupt runs again. If that code already wrote a report, sent a message, or deducted a quota, it will execute twice.

There are two solutions: use a business key for idempotency protection, or move one-time side effects to a safe task boundary. Otherwise, a single approval could produce two results — this is one of the easiest traps to fall into in production.

Traceable Reports: Every Conclusion Can Be Traced Back to Evidence

Once approval passes, trend charts, comparison charts, and composition charts all grow from the same batch of data. The role of charts is defined pragmatically: charts are not decoration — trends show change over time, comparisons show differences across categories, and compositions show share breakdowns. Report031 keeps only the views that support its conclusions, firmly avoiding cramming six-panel dashboards and complex multi-axis charts into a weekly report that no one can read quickly.

More importantly, there's the "lineage chain" design: every conclusion in the report can trace back along the evidence line to the metric definition, query result, data time range, and corresponding chart. When the data version changes, the system can determine which conclusions need to be recalculated. This traceability lets reviewers ask "what's the basis for this?" rather than passively trusting the model's tone.

Controlled Failure and the Five Production Switches

Real production pipelines will fail — but failures must be controlled. Each error type has a corresponding handling path:

• Timeout → enter limited retry
• Empty result → return to scope confirmation
• Parameter error → request additional input
• Full pipeline error → terminate immediately and escalate to human

Every error type has a retry budget and stopping condition — infinite loops are never used to mask failures. Report031 retains the last valid checkpoint so that fixes can resume from a known state.

Switching from Demo to production requires enabling five switches: persistent state, least privilege, MCP's Streamable HTTP, full-chain observability, and fixed Golden Set regression tests. Local processes can still use stdio, but you also need to rehearse interrupt recovery, full pipeline rejection, and dependency degradation — proving that when something goes wrong, the system stops at the right position.

Evaluating Agent Quality with Six Metrics

One final insight worth remembering: don't summarize an Agent with "accuracy." Here are six evaluation metrics that better reflect engineering reality:

1. Task completion rate
2. Tool success rate
3. Human approval pass rate
4. End-to-end latency
5. Token usage and cost
6. Regression test case pass rate

Once Report031 has been stamped with all three marks — "controllable, recoverable, observable" — Module 2 officially wraps up. The next stop is MLOps, where versioning, evaluation, and releases form a closed loop.

Conclusion

The greatest value of this practical framework isn't that it demonstrates a working product — it's that it clearly draws the engineering boundary between a "toy demo" and a "production-grade Agent": results defined upfront, states inspectable, permissions minimized, approvals risk-tiered, conclusions traceable, and failures controlled. For any team genuinely trying to deploy AI Agents in a business context, these principles are far more valuable than chasing the latest model.