6 Must-Do Tasks Before Deploying Enterprise AI Agents: A Complete Guide to Observability + Evaluation

The New Bar for AI Agent Roles

As large model applications move from the lab to enterprise production environments, hiring requirements for AI engineers are rapidly escalating. According to insights shared by tech content creators, interview requirements for Agent-related positions have noticeably increased—simply knowing how to use AI tools is no longer enough. The real differentiator among candidates is whether they can develop customized agents and solve the engineering challenges that come after deployment.

Distilled from the original sharing, an enterprise-grade Agent typically needs to solve six core categories of problems before going live:

• Streaming output interruption: When a user closes the page mid-stream, the streaming output can stop, but the complete conversation record must be fully preserved without data loss. Streaming Output is a core interaction pattern in large model applications, implemented via Server-Sent Events (SSE) or WebSocket protocols. Unlike traditional request-response patterns, streaming output allows the server to push generated tokens to the client one by one or in batches, enabling users to see real-time generated content without waiting for the complete response. This pattern greatly improves user experience but introduces engineering complexity: when a user closes the browser or navigates away mid-stream, the HTTP connection breaks, and the server needs to handle this interruption gracefully—both stopping unnecessary compute resource consumption and ensuring that already-generated partial content is fully persisted to the database for subsequent conversation context assembly and audit trails.
• High concurrency: How to enable a single Agent to handle 500 to 1,000, or even 5,000+ concurrent accesses.
• Multi-tenant isolation: When different users (e.g., User A and User B) use the same Agent simultaneously, their generated temporary files, data analysis reports, etc., must be fully isolated and invisible to each other. Multi-tenancy Isolation is a foundational architectural requirement for SaaS and enterprise applications. In AI Agent scenarios, multi-tenant isolation involves not only traditional database row-level or schema-level isolation, but also temporary files generated during Agent runtime (such as CSVs and charts from data analysis), sandbox execution environments (such as the file system of a code interpreter), and knowledge base partitions in vector databases. Common implementation approaches include: container-based process-level isolation, namespace-based file system isolation, and logical isolation based on tenant IDs. The choice typically depends on the trade-off between security requirements and performance overhead.
• LLM gateway routing: Models like DeepSeek and Kimi K2 carry rate-limiting risks. The gateway needs to automatically switch to backup models upon timeout or call failure to prevent Agent downtime.
• Observability and log tracing: The ability to trace every step of Agent execution—knowing how many model calls were made, how many tokens were consumed, and how much money was spent.
• Evaluation system: Providing a closed loop of human annotation + LLM-based evaluation to continuously improve accuracy and reduce hallucinations.

This article focuses on the three hottest areas—tracing, observability, and evaluation—with Langfuse as the core open-source platform.

Why Enterprises Choose Langfuse Over LangSmith

In the domain of agent tracing, observability, and evaluation, there are currently two mainstream options: Langfuse and LangSmith provided by the official LangChain team. Based on feedback collected from frontline engineers, domestic enterprises have almost universally adopted Langfuse.

Data Privacy and Compliance

The first and most critical reason is data privacy and compliance. Langfuse uses the MIT open-source license (the same tier as DeepSeek's open-source license), making it nearly fully open-source with no copyright concerns. The MIT license is one of the most permissive open-source licenses, created by the Massachusetts Institute of Technology. It allows anyone to freely use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the software, with the sole condition of including the copyright notice and permission notice in all copies. Unlike the GPL license, the MIT license does not require derivative works to also be open-sourced, which allows enterprises to develop proprietary modifications based on MIT-licensed projects for closed-source commercial use—this is the fundamental reason domestic enterprises favor MIT-licensed projects. More importantly, all Trace and Token data in Langfuse can be kept entirely within the enterprise intranet, with no communication to external networks and no license callback required.

In contrast, LangSmith, even when deployed on enterprise intranet servers, still requires periodic internet connectivity to verify licenses, and some metadata must be reported to LangChain's servers. This is nearly unacceptable for domestic enterprises—especially in industries like finance, government, and healthcare where strict data sovereignty regulations apply. Any form of data leaving the network could violate relevant provisions of data security and personal information protection laws.

Differences in Business Models

The second reason lies in business models. The LangChain team's frameworks—LangChain, LangGraph, etc.—are all open-source, while LangSmith is precisely their sole major revenue source—its core computation engine is closed-source, primarily generating revenue through SaaS cloud services for overseas enterprises. This inherently makes it difficult to meet domestic enterprises' rigid requirements for privatized, intranet deployment.

It's worth noting that Langfuse has reached millions of monthly downloads and installations on GitHub, holding an absolute leading position among similar open-source solutions.

Architectural Decoupling Between Langfuse and Agents

The first step to using Langfuse well is understanding the decoupled relationship between it and Agent projects.

In the overall architecture, the enterprise's self-developed Agent project (an intelligent agent built for a specific business line) is an independent service, and Langfuse is another independently deployed server. The two communicate via the Langfuse SDK, with the core data being Traces. This architectural design follows the "separation of concerns" principle in microservices—the Agent focuses on business logic execution, Langfuse focuses on observation and evaluation, neither intrudes on the other, and a failure in either one won't affect the other's normal operation.

Langfuse itself comprises a complete set of components: a Web interface for project administrators, PostgreSQL for data storage, Redis for caching, ClickHouse for analytics, and a dedicated Langfuse Worker for asynchronous evaluation. ClickHouse is an open-source columnar database management system developed by Yandex, designed specifically for Online Analytical Processing (OLAP) scenarios. It can achieve sub-second query responses on billions of rows, making it ideal for storing and analyzing the massive Trace data generated during Agent operation. Compared to traditional row-oriented databases, ClickHouse can deliver performance improvements of tens to hundreds of times in aggregation queries and time-series analysis, enabling administrators to perform multi-dimensional real-time data analysis on Agent performance—such as tracking Token consumption trends by time period or comparing accuracy changes across model versions.

The key point here is that Agent evaluation is not performed inside the Agent itself, but executed asynchronously by independent Workers within Langfuse. After scoring, results are stored in ClickHouse and ultimately presented to project leads through analytics dashboards, guiding continuous Agent optimization.

The presenter actually rented a server on Alibaba Cloud (~250 RMB/month) and deployed a complete environment via Docker Compose containing six containers including Langfuse Web, Worker, PostgreSQL, and Redis, validating the feasibility of the entire workflow.

Trace, Observation, and Core Concept Breakdown

To use Langfuse effectively, you must first clarify several foundational concepts.

Trace: A Complete Conversation Request

Trace represents a single complete request of an intelligent agent—one full round of Q&A—containing user input, final output, session, and version information. Each data entry visible in the Langfuse interface is a Trace. This concept aligns with distributed tracing thinking—just as Jaeger or Zipkin traces the complete call chain of an HTTP request in a microservices architecture, a Trace in Langfuse tracks the complete processing chain that a user question undergoes within the Agent.

Observation: The Observable Abstraction Layer

Observation is the observable abstraction of the entire working process within a Trace. It breaks down into several types, the most common being:

• Generation: A single model call, recording the system prompt, user input, model output, and call duration.
• Span: A user-defined segment of a general operation—for example, marking a sub-agent's execution as a Span, with start and end times.
• Tool: A single tool usage.
• Event: An event node.

Every model call, tool usage, and Span execution generates data, which is collected into Langfuse via the SDK, enabling complete tracing and observability. In the financial analysis Agent demonstrated, you can clearly see: the request first performs task planning, then calls the model (taking 6.53 seconds), then invokes various tools (each tool's duration clearly labeled), all presented in a tree-structure diagram showing the complete execution chain of every step.

Token Statistics and Cost Accounting

One of Langfuse's most practical values is cost visibility. By pre-configuring unit prices for each model in the settings interface (e.g., price per million input/output tokens), the system automatically tallies the number of tokens consumed at each step and for each request, converting them into actual monetary costs. For example, a single request consuming 7,163 input tokens and 640 output tokens totals approximately ¥0.006369—precise to each step. This feature is critically important for enterprises: when an Agent processes tens of thousands of requests daily, the cost differences between different prompt strategies and model choices can escalate from a few hundred to tens of thousands of RMB per month. Precise token-level cost accounting enables engineering teams to quantify the ROI of every optimization, providing data support for technical decisions.

Prompt Governance and the Four Stages of Evaluation

Centralized Prompt Management

The presenter emphasized an enterprise best practice: host prompts in Langfuse rather than hardcoding them in project code.

Langfuse provides a dedicated Prompt Manager. During runtime, the Agent dynamically calls a loading function to pull the latest prompt from Langfuse by Label and name. The code logic includes a fail-safe: if Langfuse is enabled and the service is available, prompts are pulled from the platform; if the Langfuse service goes down, it falls back to local default prompts.

The benefits are extremely clear:

• Iterate prompts without changing code, with modifications taking effect immediately;
• Version rollback capability—if evaluation scores or response times degrade after a modification, you can roll back to a previous version with one click;
• Observe score and latency changes before and after prompt adjustments through Trace data, forming a data-driven optimization loop.

This philosophy of separating configuration from code is essentially an extension of the "Configuration as Code" principle from software engineering into the AI domain. In traditional software development, we manage application configurations through config centers like Apollo or Nacos; in AI Agent development, prompts are the most critical "configuration"—they determine the Agent's behavioral patterns, output quality, and response style, far exceeding the importance of ordinary application parameter configurations.

The Four Stages of Evaluation

The evaluation system for enterprise-grade Agents progresses through four stages:

1. Scoring standardization: Define evaluation rules and user feedback rules in Langfuse's Score Config (the presenter's example included eight rules).
2. Sample upload: Upload evaluation sample data (Golden Data) for human correction.
3. Human annotation closed loop: Annotate specific Traces (adding annotations via the interface's Annotation function and adding them to datasets).
4. Continuous online evaluation: A dedicated evaluation LLM (Judge/referee model) automatically evaluates based on the aforementioned human rules, annotations, and golden answers, feeding results back to the Agent.

A common misconception needs clarification here: evaluation absolutely does not mean letting the Agent review itself. The entire evaluation relies on extensive human involvement—humans need to define evaluation rules in Langfuse, provide error samples, offer corrections, and perform annotations. This "human-machine collaborative" evaluation model draws from the "Human-in-the-Loop" philosophy in machine learning: human experts provide high-quality judgment criteria and edge cases, while the LLM scales these standards across massive Trace data. As the Agent runs longer and humans add more rules and regression datasets, the LLM evaluation becomes increasingly accurate, forming a positive feedback loop of "the longer it runs, the more accurate it becomes."

Engineering Approach to High-Concurrency Architecture

As an extension, the presenter also demonstrated the complete high-concurrency architecture from the same project. The approach is essentially horizontal scaling through microservices:

• The front layer uses a load balancer (Alibaba Cloud ALB or Nginx) to distribute traffic;
• Traffic enters multiple SSE gateways, which record request origins and write user requests into a Redis message queue;
• After dequeuing requests, rate limiting, idempotency checks, and cache judgment are performed. Idempotency means that executing the same operation once produces exactly the same effect as executing it multiple times—this is especially critical in Agent systems: due to network jitter and timeout retries, the same user request may be submitted repeatedly. If the Agent's tool calls (such as transfers, sending emails, or data writes) lack idempotency, duplicate execution can lead to severe consequences. Common idempotency implementations include deduplication based on unique request IDs, database unique constraints, and distributed locks;
• Multiple Agent Workers (worker1, worker2...workerN) act as consumers, pulling requests from the queue;
• All Workers share the same PostgreSQL instance, storing user data, Trace mappings, LangGraph Checkpoints, sessions, and long-term memory. LangGraph's Checkpoint mechanism is a core feature for Agent state persistence—it automatically saves a state snapshot to external storage after each node in the Agent's execution Graph completes. This mechanism enables different Workers to pick up and continue processing different stages of the same request, achieving horizontal scaling of stateless Workers. It also allows recovery from the most recent Checkpoint upon exceptions without restarting from scratch;
• Results are returned to users via SSE event streams on one path, while entering an evaluation queue on another path, where Evaluation Workers perform asynchronous scoring.

The scaling logic is straightforward: a single Worker handles approximately 200 concurrent connections, so 10 Workers can support 2,000 concurrent users. Theoretically, increasing Worker count linearly increases concurrency capacity. Similarly, Langfuse itself is distributed and can scale by adding Langfuse Worker nodes. It's important to note that this linear scaling capability assumes downstream dependencies (database, Redis, LLM APIs) don't become bottlenecks—in actual production, database read-write separation, Redis clustering, and multi-key rotation strategies for LLM APIs are typically needed to ensure overall system elasticity.

Conclusion

The core value of this practical sharing lies in revealing the enormous engineering gap between an Agent that "can run a demo" and one that "can operate in production." Observability, log tracing, prompt governance, evaluation systems, and token accounting—these seemingly mundane engineering concerns are precisely the core interview topics for AI engineers and the foundation upon which enterprise agents can continuously iterate, reduce hallucinations, and improve accuracy. With its MIT open-source license, intranet self-hosting, and data-never-leaves-the-network characteristics, Langfuse has become virtually the only mainstream choice for domestic enterprises in this domain.