Getting Started with Langfuse: Building a Financial Analysis Agent and Evaluation Platform

Why You Need Langfuse: From Agent Development to Observability

Many developers can already build multi-agent applications based on the Harness architecture, but tracing, evaluation, and observability after an agent goes live are often overlooked. As the Bilibili creator pointed out at the start of the tutorial, previous videos covered projects like a procurement assistant agent and an AI coding agent, all focused on "how to develop." This time the core focus shifts to "how to trace, evaluate, and observe." Langfuse is the open-source AI Engineer platform built for exactly this purpose.

Langfuse's official definition is an "open-source LLM OPS platform," but that description is too vague. A more accurate understanding: it targets RAG and agent applications, providing runtime tracing, token cost and latency analysis, prompt management, feedback collection, automated and human-annotated evaluation, and continuous improvement through datasets and experiments. It does not develop the agents themselves; instead, it provides data collection, evaluation, and tracing capabilities for agents you have already built.

LLM Ops is an extension of traditional MLOps into the generative AI context, covering model deployment, prompt version management, cost monitoring, quality evaluation, and continuous iteration. Unlike traditional software operations, LLM application outputs are non-deterministic, and costs are directly tied to token consumption. This requires a dedicated toolchain to record the input/output, latency, and cost of every call, and to support regression testing based on real data. Langfuse is an open-source platform positioned at exactly this layer—it does not replace model services or application frameworks, but completes the observation-and-improvement loop from development through post-launch.

Langfuse's Core Capabilities and Boundaries

What It Can Do

Langfuse provides observability for LLMs, RAG, and agents, recording the various call chains of models, tools, retrieval, and sub-agents. It supports evaluation and feedback, prompt management, and metric analysis. A plain analogy: if an agent is a digital AI employee that looks up information, uses tools, and performs calculations, then Langfuse is the combination of a flight recorder, a quality department, a laboratory, and a prompt configuration center.

The flight recorder tracks every move of the digital employee; the quality department evaluates the quality of generated answers; the laboratory lets you run tests after changing a prompt or swapping a model to compare against the current stable version (Best9); and the prompt configuration center solves a key pain point—prompts modified at runtime may perform worse, and when stored in code they can't be rolled back, whereas Langfuse supports version management and rollback without restarting the RAG or agent project.

Tracing is the foundation of observability. In an agent application, a single user request may trigger multiple model calls, tool executions, and sub-agent collaborations, forming a tree-structured call hierarchy. Langfuse uses Traces and Spans to record the input/output, duration, model version, and token usage of each node, enabling developers to reconstruct the full execution path. This is conceptually similar to traditional distributed tracing (such as OpenTelemetry), but adds records of prompts, generated results, and evaluation scores for LLM scenarios, making it easier to pinpoint which step causes latency or quality degradation.

Prompt version management and Playground address reproducibility in prompt engineering. Prompts modified in code are hard to roll back, and effect comparisons lack a unified baseline. Langfuse treats prompts as independent resources under version control, generating a new version on each change while preserving history, with one-click rollback. The Playground feature allows comparing the effects of different prompts or model versions under the same input, using evaluation scores to decide whether to ship. Because prompts are fetched dynamically via API, changes take effect without restarting the application, lowering iteration costs.

What It Cannot Do

Langfuse does not provide the agent framework itself, nor an LLM, a vector knowledge base, or a business database. It is also not responsible for server operations, monitoring, or health checks after an agent or RAG goes live. Clarifying these boundaries is essential to correctly embedding Langfuse into an enterprise-grade tech stack.

How Langfuse Answers Key Questions

In real enterprise development, Langfuse can answer a range of critical questions: which steps and which sub-agents did what in this request, and which tools and models were called; how long tools and models took to execute; which versions of prompts and models were used; what the input/output token counts, costs, and latency are; whether the answer is correct and has a complete evidence list; whether the user is satisfied (via user annotations); and whether the new version is better than the current stable version. These questions cover the core observability needs of agents after launch.

Integration with a Harness-Architecture Financial Analysis Agent

The project planned in this tutorial has two layers: first, a multi-agent financial analysis agent based on the Harness architecture; second, the development of an enterprise-grade log tracing, evaluation, and observability platform. The Harness architecture was previously used for the procurement assistant agent and the AI coding agent, and this financial analysis scenario will reuse it, with Langfuse plugged in as the evaluation and tracing layer. The course does not focus on re-explaining agent development, but on showing how to integrate Langfuse with existing agents to build an enterprise-grade evaluation platform.

Harness architecture here refers to a multi-agent orchestration pattern, typically including a coordinator (Harness) responsible for task decomposition and routing, with multiple sub-agents handling retrieval, calculation, analysis, and other responsibilities. The advantage of this architecture is separation of concerns and scalability, but it also increases call-chain complexity. When integrating Langfuse with the Harness architecture, you need to instrument key nodes in the coordinator and each sub-agent to record task dispatch, tool calls, and result aggregation, so the tracing view clearly presents the full picture of multi-agent collaboration and provides a data foundation for evaluating each sub-agent's contribution and bottlenecks.

Supported Languages and Frameworks

Langfuse's top-level languages are Python and TypeScript/JavaScript, and it also supports secondary development on top of it. It integrates seamlessly with common RAG and agent development frameworks such as LangChain, LangGraph, LlamaIndex, and the OpenAI Agent SDK. This means that no matter which framework your agent is built on, you can integrate Langfuse's tracing and evaluation capabilities at low cost.

Getting Started Recommendations

For teams preparing to adopt Langfuse, it is recommended to first clarify the boundaries of the agent's observability needs, then deploy the self-hosted Langfuse service and connect it to your existing agent project. Prompt management should be brought into Langfuse as early as possible, leveraging its version management and Playground features for iterative comparison. For evaluation, start with a combination of human annotation and automated evaluation, and gradually build up datasets and experiment workflows. Langfuse is not a replacement for an agent framework, but an indispensable observability and evaluation infrastructure in the enterprise AI engineering loop.

One-line Summary

Langfuse is an open-source observability and evaluation platform for RAG and agents, completing the enterprise AI engineering loop.

Paragraph Summary

The article introduces Langfuse's positioning, core capabilities, and boundaries, noting that it is an open-source AI Engineer platform for RAG and agent applications, offering runtime tracing, token cost and latency analysis, prompt version management, feedback collection, and evaluation. Langfuse does not provide an agent framework, LLM, or knowledge base; instead, it integrates with existing projects as an evaluation and tracing layer. Using a Harness-architecture financial analysis agent as an example, the article explains how to connect Langfuse to a multi-agent system to achieve enterprise-grade log tracing, evaluation, and observability. It concludes with getting-started recommendations: clarify observability needs, self-host the deployment, adopt prompt management early, and combine human and automated evaluation to build datasets and experiment workflows.