Production AI Agent Operations: A Three-Step Framework for Observing, Evaluating, and Optimizing

Title

Production AI Agent Operations: A Three-Step Framework for Observing, Evaluating, and Optimizing

Meta Description

Azure Cosmos DB team proposes a three-step framework for operating production AI Agents: Observe trace tracking, Evaluate with LLM as a judge and regression testing, Optimize cost across RU, retrieval, memory, and LLM layers. This article analyzes the core points and applicable boundaries of this methodology.

Article Content

Building an AI Agent that can run is not difficult; the hard part is making it run stably, efficiently, and predictably in production. In the latest episode of the "Code with Azure Cosmos DB" series, the Azure Cosmos DB team presented a clear operational framework: Observe, Evaluate, Optimize. The value of this methodology lies not in introducing some new product, but in treating AI Agents as real software systems—first understand behavior, then verify quality, and finally control cost.

Why Production Agents Need an Operations Framework

Traditional software has logs, monitoring, and regression tests after launch, while many AI Agent projects perform well in the demo stage but run into frequent problems once in production: user preferences are forgotten, routing decisions drift, and a certain change quietly breaks functionality that was originally working. According to hosts Marko Haari and Madeline Egan in the video, the root cause is often not insufficient model capability, but the team's lack of visibility into the Agent's actual behavior.

The framework they propose follows a simple logic: before fixing any problem, you must first know what the Agent is actually doing. These three stages are not a one-time process but a continuous loop—production systems keep evolving, so the actions of observing, evaluating, and optimizing must also continue.

Observe: Make Every Step of the Agent Traceable

The core of the observation stage is the trace. Every Agent request generates a complete trace that records key actions such as tool calls, routing decisions, and memory retrieval. When the Agent reads and writes information from Azure Cosmos DB, these interactions are also incorporated into the same trace and fully presented in LangSmith.

The design thinking here is worth noting: because both memory data and application data are stored in Cosmos DB, the team can obtain a complete end-to-end view in one place without having to piece together clues across multiple systems. Microsoft Agent Framework Tracing is responsible for showing how requests flow through the system, including routing, handoffs, and tool calls; Microsoft Foundry provides observability capabilities for production AI workloads. The entire solution is based on OpenTelemetry, and traces can flow directly into the team's existing observability stack.

One detail that is easy to overlook: observability itself does not create value; what it creates is "understanding." Only by first seeing clearly how the Agent currently behaves can subsequent evaluation and optimization have a basis.

trace and span: In distributed tracing systems, a complete request is called a trace, and a trace consists of several spans, each representing a unit of operation (such as a tool call, a database query, or a model inference). OpenTelemetry is currently the industry-standard observability standard; it defines collection and export protocols for three types of signals—trace, metric, and log—so that the same tracing data can flow into different backends such as Jaeger, Datadog, and LangSmith without repeatedly instrumenting for each observability platform. This is precisely the technical premise behind the article's emphasis that "traces can flow directly into the team's existing observability stack."

LangSmith: An LLM application observability and evaluation platform launched by the LangChain team, supporting recording of Agent call chains, viewing each LLM input and output, building datasets, and running evaluations. In this article, it plays the role of combining Cosmos DB interactions with application-side actions in a single presentation.

Evaluate: Test Agents Like Software

Observability tells you what happened; evaluation answers "is this correct?" In the video, Madeline Egan emphasizes that Agents should be tested like software—from individual routing decisions and tool calls to end-to-end overall performance, everything can be included in the scope of evaluation.

The specific methods include three categories. The first is LLM as a judge, using models to automatically perform large-scale quality checks and avoid manual review of responses one by one. The second is regression testing, used to catch unexpected side effects and ensure that new changes do not break functionality that is already working. The third is embedding evaluation into the CI/CD pipeline so that every change is verified before entering production.

For Cosmos DB, there is also a highly representative dedicated evaluation: verifying whether cross-session memory recall is accurate. In the third episode of the series, the user explicitly says they do not like red-eye flights; then in a brand-new session, does the Agent remember and comply with this preference? Because Cosmos DB is the only persistent storage for this memory, evaluation can directly target the persisted user state without cross-system data verification.

This leads to the most pragmatic sentence in the entire framework: the one who should discover that the Agent has a problem is your evaluation suite, not your users.

LLM as a judge: An automated evaluation paradigm in which a large model acts as the scorer. The approach is to provide the original input, the Agent output, and a scoring rubric, and let the judging model output a score or rationale. Compared with traditional metrics such as exact match or BLEU, it is better able to handle quality judgments for open-ended answers and is suitable for large-scale sampling-based quality inspection. Its limitation is that the judging model itself has bias and instability, so human-labeled samples are usually needed for calibration, and consistency checks should be performed on scoring results.

Regression testing and CI/CD: Regression testing means rerunning the existing test set after every change to confirm that previously passing behavior has not been broken; CI/CD is the pipeline that automates build, test, and deployment. Embedding Agent evaluation into CI/CD means that every change to prompts, tool definitions, or retrieval configuration triggers a rerun of the evaluation set, and failure to meet the standard blocks merging or release.

Optimize: Four-Layer Synergy, Not Single-Point Squeezing

After confirming that the Agent works properly, the next question is how to make it run faster and cheaper. Egan's judgment is direct: the most expensive Agent is rarely the optimal choice. Optimization should narrow the gap between "actual spend" and "user experience."

They break optimization into four layers, and these four layers are built on the same data foundation:

• RU optimization: At the Cosmos DB layer, get indexing strategy, partition model, and query patterns right to avoid paying for throughput you do not need. This is the foundation of the entire system.
• Retrieval optimization: Tune vector search and hybrid search so the Agent can find the most relevant context faster while avoiding over-fetching data.
• Memory optimization: Instead of carrying the full conversation history indefinitely, retain only key information through summarization and compression, reducing the number of round trips to Cosmos DB.
• LLM layer optimization: Reduce token consumption through more compact prompts, smaller context windows, and fewer model calls.

These four layers do not exist in isolation. Better retrieval means there is less content for the LLM to process, and better memory summarization means fewer database round trips. As stated in the video, improvements at the database layer not only reduce RU consumption but also improve the efficiency of the entire application—this is exactly the advantage brought by the four layers sharing the same data foundation.

It is worth emphasizing that optimization is not a step temporarily added before launch. In the first episode of the series, the coding assistant was already teaching Cosmos DB best practices when writing the first line of code. Good architectural decisions are made early in development and continue to pay returns in production.

RU (Request Unit): The throughput and cost unit of Azure Cosmos DB, normalizing the CPU, IO, and memory consumed by read and write operations into a single value. The number of RUs available per second (RU/s) determines the database's processing capacity, and charges are based on provisioned or autoscaling RU/s. Therefore, the essence of "RU optimization" is to make each query consume fewer RUs: a reasonable partition key avoids cross-partition queries, an appropriate indexing strategy reduces unnecessary index writes and scans, and queries retrieve only the needed fields rather than the full document. These adjustments map directly to the bill amount, which is also why the article calls this the foundation of the entire optimization system.

Vector search and hybrid search: Vector search retrieves based on the similarity of embedding vectors; hybrid search fuses vector similarity with keyword scoring (such as BM25) for ranking, balancing semantic recall and exact matching. Tuning directions include controlling the number of returned results top-k, setting similarity thresholds, and choosing appropriate partition and index types to avoid retrieving too much irrelevant context at once and wasting RU and tokens.

Applicable Boundaries of This Framework

What needs to be viewed objectively is that this episode is essentially product evangelism for Azure Cosmos DB, and the three stages of observation, evaluation, and optimization are deeply tied to its own or partner ecosystem components such as LangSmith, Microsoft Agent Framework, and Microsoft Foundry. For teams that do not use the Azure technology stack, the specific toolchain cannot be copied directly.

But the methodology itself is universal: before an Agent goes live, there must be trace-level visibility; quality verification must be automated and enter CI/CD; and cost optimization must start simultaneously from the data layer, retrieval layer, memory layer, and model layer. Treating AI Agents as software systems to be operated, rather than as black boxes to pray to—this is the real insight of this framework.

One-line Summary

Azure Cosmos DB team proposes a three-stage loop of observe, evaluate, and optimize, treating AI Agents as production software systems to be operated.

Paragraph Summary

The article introduces the AI Agent operations framework proposed by the Azure Cosmos DB team in the "Code with Azure Cosmos DB" series: observe, evaluate, and optimize. In the observation stage, OpenTelemetry-based traces record tool calls, routing decisions, and memory retrieval, and because memory and application data are both stored in Cosmos DB, an end-to-end view is obtained; the evaluation stage advocates testing Agents like software, using LLM as a judge for large-scale quality checks, regression testing to prevent side effects, embedding evaluation into CI/CD, and using cross-session memory recall as a dedicated use case; the optimization stage is divided into four layers—RU optimization, retrieval optimization, memory optimization, and LLM layer optimization—which share the same data foundation and reinforce one another. The article points out that the framework's toolchain is deeply tied to the Azure ecosystem, but the methodology is universal.