Complete Guide to Agent Observability: Tracing, Evaluation, and Red Teaming in Practice

In the journey from prototype to production for AI Agents, one core challenge consistently plagues developers: How do you bridge the gap between an Agent's expected behavior and its actual performance? At the AI Engineer conference, Amy and Nitya from the Microsoft Foundry team presented a comprehensive solution for Agent observability, covering the full lifecycle management of tracing, evaluation, security protection, and AI-assisted optimization.

Non-Determinism: The Core Challenge Facing AI Agents

Regardless of which tech stack you use to build Agents, an inescapable reality remains: Agents are inherently non-deterministic. This isn't just a demo-stage issue—it's a core challenge in production environments. When you deliver an Agent to real users, you must manage the risks that come with this non-determinism.

To understand this, we need to return to the underlying mechanics of large language models. In traditional software engineering, functions are typically deterministic—the same parameters always return the same results. But LLMs, with their probability distribution-based token sampling mechanisms (such as randomness controlled by the temperature parameter), dynamic changes in context windows, and silent model version updates, are inherently non-deterministic. In Agent scenarios, this non-determinism is further amplified: Agents don't just generate text—they make tool-calling decisions and plan multi-step tasks, where minor deviations at each step can lead to significant differences in final outcomes. This is why traditional unit testing and integration testing methods are severely insufficient for Agent systems, and statistical evaluation frameworks must be introduced.

The speakers summarized the response strategies across three dimensions:

1. Evaluation: Check performance, quality, and safety to ensure the Agent works as expected
2. Monitoring: Continuously track Agent performance in production, responding to model changes, user behavior shifts, and environmental changes
3. Optimization: Systematically improve Agent performance based on collected data

An elegant analogy: evaluation is like a building inspector checking whether your house meets code; while security protection (Red Teaming) is like hiring someone to try to break into your house, proving that defenses actually work.

Building an Agent Prototype from Scratch: Getting Started with Microsoft Foundry

Faced with the selection dilemma of over 2 million available models and 11,000+ models in the Azure catalog, developers often don't know where to begin. Microsoft Foundry provides a fast path forward.

Rapid Prototyping via Portal

On ai.azure.com, developers can complete the following in just minutes:

• Create a project with automatic resource configuration
• The system automatically recommends GPT-4.1 as a starting model
• Add system prompts, tools (like Bing Search), and knowledge bases
• Built-in App Insights for instant tracing

Crucially, the Portal isn't just a toy environment. Every interaction with the Agent automatically generates trace records, including token consumption, response time, and AI quality and safety metrics. Developers can select evaluation metrics directly in the Playground without needing to memorize various metric names.

Deep Development via SDK

After validating concepts in the Portal, developers can seamlessly switch to the SDK for more complex development. The workshop provides a complete notebook path:

1. Lab 1: Environment connection verification, ensuring Codespaces correctly interfaces with the Azure backend
2. Lab 2: Create a basic Agent with code (model + prompt)
3. Lab 3: Add function tools (flight queries, hotel bookings, car rental services)
4. Lab 4: Build a multi-Agent collaboration system (Orchestrator Agent)
5. Lab 5: Configure tracing (local + Azure Monitor)
6. Lab 6: Run evaluations and Red Teaming

Multi-Agent Collaboration Tracing and Cost Optimization in Practice

Foundry introduces the concept of Team Agent, allowing developers to split a monolithic Agent into multiple specialized sub-Agents coordinated by an Orchestrator.

Multi-Agent Systems (MAS) represent an important evolution in current AI Agent architecture. Unlike monolithic Agents that concentrate all capabilities in a single prompt and toolset, multi-Agent architectures decompose complex tasks into multiple specialized sub-Agents, each with independent system prompts, toolsets, and potentially different underlying models. The orchestrator handles task distribution, result aggregation, and error handling. The advantages of this architecture include: separation of concerns makes each sub-Agent easier to debug and optimize; different sub-Agents can use models at different cost tiers (e.g., small models for simple queries, large models for complex reasoning); and sub-Agents can iterate independently without affecting the overall system. However, this also introduces new challenges: latency accumulation from inter-Agent communication, complexity of orchestration logic, and cross-Agent tracing and attribution problems—precisely the core issues that Foundry Team Agent and its tracing system aim to solve.

At the tracing level, the value of this architecture is particularly evident. When a user submits a travel planning request, the tracing system clearly shows:

• How the orchestrator distributes tasks to the flight Agent, hotel Agent, and car rental Agent
• The call chain, duration, and token consumption of each sub-Agent
• Which steps succeeded and which encountered problems

Regarding cost optimization, the speakers highlighted two key strategies: first, model substitution (e.g., switching from GPT-4.1 to GPT-4 Mini to reduce token costs), but regression evaluations must be run after each substitution to confirm quality hasn't degraded; second, trace analysis, identifying bottlenecks by observing duration at each stage (e.g., if knowledge retrieval takes too long, consider caching or data preprocessing).

A notable detail: Foundry's tracing is built on the OpenTelemetry standard. OpenTelemetry (OTel) is an open-source observability framework hosted by the Cloud Native Computing Foundation (CNCF) that unifies collection standards for three major signals: distributed Traces, Metrics, and Logs. Adopting OpenTelemetry in the AI Agent domain is significant: it means Agent call chains can be seamlessly correlated with traditional microservice tracing data. For example, the complete path of a user request—from frontend to API gateway, to Agent orchestrator, and finally to the LLM inference endpoint—can all be linked through a unified trace ID. This prevents AI systems from becoming observability "black boxes" and allows existing monitoring infrastructure (such as Jaeger, Grafana, Azure Monitor) to directly consume AI-related telemetry data without building entirely new monitoring systems. Additionally, tracing data can be simultaneously pushed to both the Foundry Portal and Azure Monitor, enabling IT teams to see AI-related telemetry on their familiar monitoring platforms.

Agent Evaluation System: From Quality Metrics to Security Testing

Quality Evaluation Metrics Explained

Foundry provides a rich set of built-in evaluators covering:

• Relevance: Whether the answer addresses the question
• Fluency: Whether the language is natural
• Groundedness: Whether the answer is factually supported
• Agent behavior evaluation: Whether tool calls are correct, task completion rate, etc.

Evaluation results clearly indicate failure reasons. For example, in one test case, the Agent returned "August 24, 2024" instead of "2025," and was flagged as a groundedness failure—it wasn't truly answering based on facts.

Red Teaming Security Testing in Practice

Red Teaming is the most challenging aspect of security protection. The core idea is to use an attacker Agent to proactively probe the target Agent for vulnerabilities.

Developers can define risk categories (such as data leakage, unauthorized operations, etc.), select attack strategies (from simple direct attacks to complex Crescendo attacks), and then let the system automatically generate attack cases and evaluate defense effectiveness.

Crescendo attacks deserve particular attention. This is a multi-turn conversational jailbreak technique published by the Microsoft research team in 2024. Unlike traditional single-turn prompt injection, Crescendo exploits the LLM's in-context learning ability by gradually building context through a series of seemingly innocuous conversations, ultimately guiding the model to breach safety boundaries. For example, an attacker might first ask "What are some famous chemical discoveries in history?" and then gradually steer the topic toward synthesis methods for dangerous chemicals. Since each conversational turn appears individually benign, traditional input filters struggle to detect this type of attack. In Agent scenarios, Crescendo attacks are even more dangerous because Agents have tool-calling capabilities—an attacker could potentially induce the Agent through progressive conversation to execute unauthorized API calls or data access operations. Defending against Crescendo attacks requires security evaluation at the conversation level rather than the individual turn level.

For Agent scenarios, there's also a special class of capability abuse attacks—attempting to get the Agent to perform actions it shouldn't (such as stealing passwords or leaking data), which is particularly dangerous in multi-tool Agents.

Observe Skill: Using AI to Automatically Evaluate and Optimize AI Agents

The most exciting part of the presentation was the demonstration of Foundry Skills, particularly the Observe skill. This represents an entirely new paradigm: letting an AI Agent help you evaluate and optimize your AI Agent.

Automated Evaluation and Optimization Workflow

1. Developers only need to provide a basic Agent (model + prompt + endpoint)
2. The Observe skill automatically analyzes the code and discovers the lack of evaluation datasets
3. Automatically generates test datasets
4. Selects appropriate evaluators and runs baseline evaluations
5. Returns a results report highlighting issues
6. Automatically runs the Prompt Optimizer, improving prompts and re-evaluating
7. Iterates across multiple rounds, recording improvements and regressions in each round
8. The human ultimately decides which version to adopt

The Prompt Optimizer is a technique that uses automated methods to systematically improve LLM prompts. Its core approach borrows from hyperparameter tuning in machine learning: treating the prompt as a tunable parameter, the evaluation metric as the objective function, and finding the optimal prompt through iterative search. Common optimization strategies include: automatic rewriting based on evaluation feedback (having another LLM improve the prompt based on failure cases), discrete optimization based on gradient approximation (methods like APE, OPRO, etc.), and prompt mutation and selection based on evolutionary algorithms. In Foundry's implementation, the optimizer analyzes the specific reasons for evaluation failures (such as insufficient groundedness, incorrect tool calls), then makes targeted adjustments to instructions, constraints, and examples in the system prompt, followed by re-evaluation to verify improvement.

In the demonstration, the Observe skill progressively improved the Agent's evaluation scores from initial levels through multiple iterations. Crucially, it doesn't just optimize prompts—it also suggests deeper improvements like model substitution and tracing configuration.

The Critical Role of Human-in-the-Loop

In this process, human-AI collaboration is essential. AI may exhibit a "seesaw effect" during optimization—improving one metric while degrading another. Human judgment is needed to determine when to stop iterating and choose the best balance point. In the demo, after multiple attempts, the system found scores fluctuating between 5 and 8, and ultimately the developer decided to adopt the version with a score of 5 as the optimal solution.

Developer Toolchain: Building a Complete Observability Loop

Beyond the SDK and Portal, Foundry also provides multi-layered developer support:

• AI Toolkit Extension: Access Foundry features directly in VS Code
• Ask AI: A Portal-embedded AI assistant that understands your project context
• Azure Monitor Natural Language Queries: Use natural language instead of complex KQL queries to analyze trace logs
• MCP Server: Call Foundry skills directly through GitHub Copilot

The significance of MCP Server deserves elaboration. MCP (Model Context Protocol) is an open protocol released by Anthropic in late 2024, designed to standardize interactions between LLMs and external tools and data sources. MCP uses a client-server architecture: MCP Servers expose tools and resources, while MCP Clients (such as GitHub Copilot, Claude Desktop, and other AI assistants) call these capabilities through a standard protocol. Foundry providing an MCP Server means developers can directly invoke Foundry's evaluation, tracing, and optimization capabilities within their preferred AI coding assistant, without switching to the Portal or manually writing SDK code. This reflects an industry trend where AI toolchains are moving from "platform lock-in" toward "protocol interoperability," where developers can orchestrate AI capabilities from different platforms within a unified conversational interface.

The entire toolchain's design philosophy is: regardless of which development stage you're in, there are corresponding tools to help you bridge observability gaps. From the Portal's rapid validation, to the SDK's fine-grained control, to AI Skills' automated optimization, it forms a complete closed loop.

Summary: Bridging the Three Gaps in Agent Observability

The gaps in Agent observability manifest at three levels:

1. Behavioral gap: Deviations between expected and actual Agent behavior, which need to be discovered through tracing and evaluation
2. Response speed: Time from detecting issues to diagnosing and fixing them, which needs to be shortened through correlated tracing and evaluation
3. Security perimeter: Not only guarding against issues during normal use, but proactively testing against malicious attack scenarios

The key to bridging these gaps lies in: transforming observability from reactive remediation into a built-in capability of the development process, and leveraging AI coding assistants to accelerate the entire evaluation-optimization loop. Foundry Skills is still in a very early stage (released only two weeks ago), but the direction it demonstrates—using AI to manage AI quality—is undoubtedly an important trend in Agent engineering.

Key Takeaways

• Agent non-determinism is the core challenge in production environments, requiring management across three dimensions: evaluation, monitoring, and optimization
• Microsoft Foundry provides a complete path from Portal rapid prototyping to SDK deep development, with built-in OpenTelemetry-based tracing
• Multi-Agent collaboration architecture (Team Agent) combined with tracing systems can precisely locate performance bottlenecks and cost consumption for each sub-Agent
• Red Teaming proactively probes security vulnerabilities through attacker Agents; Crescendo attacks and capability abuse attacks are threats unique to Agent scenarios
• Foundry Observe skill implements an AI-driven automated evaluation and optimization loop, including automatic test data generation, evaluation execution, prompt optimization, and iterative improvement