Langfuse in Practice: Building an Observable Evaluation Platform for Harness Agents

In the journey of AI agents from demo to enterprise deployment, finishing development is only the first step. How to trace every call, evaluate output quality, and manage prompt versions is what determines whether a project can run stably. Langfuse is an open-source LLMOps platform built for exactly this purpose. Based on a sharing by Bilibili instructor Xiaobin, combined with a financial analysis project built on the Harness multi-agent architecture, this article breaks down Langfuse's core concepts, capability boundaries, and practical value.

What Is Langfuse: More Than Log Tracing

The official definition describes Langfuse as an open-source LLMOps platform for large models. That description is a bit abstract, so the instructor translates it as: an open-source AI engineering platform for LLM RAG and agent applications, whose main purpose is to trace execution, analyze token cost and latency, manage prompts, collect feedback, perform automated and human annotation-based evaluation, and enable continuous improvement through datasets and experiments.

This statement has two layers of meaning. First, Langfuse targets already-built RAG or agent applications, collecting data, evaluating, and tracing them; second, it can also trace logs of the large models themselves. For example, coding agents like Codex commonly used in enterprises are essentially agents too, and once connected to Langfuse, their token cost and latency can likewise be analyzed.

The official English site summarizes its positioning as "Open Source Agent Evaluation & Observability"—Observability, while also covering analytics. Langfuse supports both Python and TypeScript, and integrates seamlessly with mainstream agent development frameworks such as LangChain, LangGraph, LlamaIndex, and the OpenAI Agent SDK.

LLMOps is an extension of MLOps (machine learning operations) into the large-model context, referring to a full set of engineering practices around the entire lifecycle of LLM applications, including prompt and model version management, call-chain tracing, token cost accounting, output quality evaluation, online monitoring, and feedback loops. Traditional MLOps focuses on training data versioning, model training, and deployment; in LLM applications, the model itself is usually provided externally, and the variable parts shift to prompts, retrieval, tool calls, and agent orchestration—so the focus of LLMOps tooling shifts accordingly. Understanding this background helps clarify Langfuse's positioning: it solves not "how to train a model" but "how to observe, evaluate, and iterate after the application goes live," which is also the fundamental difference between it and ordinary logging systems or APM monitoring tools.

Understanding Langfuse's Role Through the "Digital Employee" Analogy

If an agent is compared to a digital AI employee that can look up information, use tools, and perform calculations, then Langfuse is the combination of a flight recorder, a quality department, a laboratory, and a prompt configuration center.

The flight recorder traces every step this digital employee takes; the quality department evaluates the quality of generated answers; the laboratory is for experiments—when you modify a prompt and want to verify whether the new version performs better, you can run it in the lab first before deciding whether to go live.

The prompt configuration center is an easily underestimated capability. Many developers are used to writing prompts directly in project code, which is fine in itself, but prompts inevitably need modification during operation. After modification, performance may get worse—can you roll back? Can you complete version management and rollback without restarting the RAG or agent project? If prompts are hardcoded in code, these operations are nearly impossible.

The instructor emphasizes that the prompt configuration center is highly valuable in real enterprise development, but it does not replace the agent or RAG in making business decisions.

Prompt version management and rollback rely on platform-based prompt storage. The approach is to extract prompts from code and store them as independent objects with names and versions in Langfuse; at runtime, the application pulls a specified version by name (or the version tagged as production) via the SDK. This way, modification and release do not require rebuilding and redeploying code, and when problems occur, you can point the production tag back to an older version to roll back. This is usually accompanied by labels, environment separation (development/production), and A/B comparison, making prompt changes auditable and traceable like code releases. Note that hardcoding prompts is not wrong; it's just that in scenarios with frequent iteration and multi-person collaboration, the lack of version control makes "how to recover after breaking something" a real risk.

What Key Questions Langfuse Can Answer

From an operations perspective, Langfuse helps answer a series of specific questions: which steps and which sub-agents did what in this request, and which tools and models were called; how long tools and models took to execute; which versions of prompts and models were used; what are the input and output token statistics, costs, and latency.

On the quality level, it can answer: whether the answer is correct, whether there is a complete chain of evidence, whether the user is satisfied (collecting feedback through user annotations), and whether the new version is better than the baseline. Combined with laboratory capabilities, after developers modify prompts or switch models, they can first validate results in an experimental environment and replace the online version only if it outperforms the baseline.

Call-chain tracing is the foundation of observability. An agent request is often not a single model call but is split into multi-level nested steps such as planning, retrieval, tool calls, and sub-agent collaboration. The industry typically uses a Trace to represent a complete request, and Span or Generation to represent sub-steps and single model generations within it. Langfuse organizes them into a tree structure, recording the input and output, elapsed time, model used, and prompt version of each step, and carries session and user identifiers for correlation. Only with such structured chains can you answer questions like "which step is slow," "which sub-agent went wrong," and "how many tokens a request consumes in total"; plain text logs lack hierarchical relationships and unified fields, making cost allocation and problem localization difficult to support.

Capability Boundaries: What Langfuse Does Not Do

Clarifying boundaries is equally important. Langfuse can provide observability for large models, RAG, and agents, recording various call chains of models, tools, retrieval, and sub-agents; it can perform evaluation and feedback collection; it can manage prompts; it can analyze metrics.

But it does not provide an agent development framework itself, does not provide large models, does not provide a vector knowledge base, does not provide a business database, and is not responsible for server operations, health checks, and monitoring after agents or RAG go live. These still need to be handled by the project's own technology stack.

Integration Points with the Harness Architecture Project

Earlier in this course series, considerable time was spent explaining the development of Harness multi-agent projects, including a procurement assistant and an enterprise-level AI coding agent. The focus of this financial analysis agent project is not the development of the agent itself, but how to trace, evaluate, and build observability for it.

This also explains why the course starts with Langfuse: agent development capabilities are already in place, and the weak link in enterprise deployment lies in operations and evaluation systems. The Harness architecture handles multi-agent orchestration and business execution, while Langfuse handles collection of runtime data, evaluation, and prompt governance—together forming a closed loop of development and operations.

For teams pushing agents into production, this division of labor is worth learning from: first have a runnable agent, then use a specialized platform to fill in observability and evaluation capabilities, rather than cramming everything into business code.

Harness architecture here refers to an engineering framework for organizing multi-agent collaboration, whose core is to decompose a complex task into multiple sub-agents with clear responsibilities (such as retrieval, calculation, review, and summarization), with an orchestration layer responsible for scheduling, passing context, and aggregating results. This layered structure improves task decomposability and maintainability, but also makes the runtime harder to observe: a single user request triggers multiple model calls and tool calls, and cross-agent context flow is hidden inside the code. This is exactly where the value of an observability platform lies—the orchestration layer handles "how to execute," and the tracing platform handles "how well execution went." The two have separated responsibilities, avoiding the coupling caused by hardcoding instrumentation, evaluation, and logging logic into business code.

One-line Summary

Langfuse is an open-source LLMOps platform that provides tracing, evaluation, and prompt governance for agents.

Paragraph Summary

The article centers on the open-source LLMOps platform Langfuse, pointing out that when agents move from demo to enterprise deployment, finishing development is only the first step—tracing calls, evaluating quality, and managing prompt versions are the keys to stable operation. Using the "digital employee" analogy, the author explains Langfuse as a combination of a flight recorder, a quality department, a laboratory, and a prompt configuration center: it records every step along with token costs and latency, and also supports quality evaluation, experimental comparison, and prompt version rollback. The article also clarifies capability boundaries—Langfuse does not provide an agent development framework, large models, a knowledge base, or server operations. Finally, combined with the Harness multi-agent financial analysis project, it explains the division of labor where Harness handles orchestration and execution, and Langfuse handles observability and evaluation.