Langfuse Open-Source LLM Observability: Stop Debugging RAG Apps Blind

A RAG app returns a confidently-worded answer in production that cites the wrong policy document. Without a trace, all you can do is guess in a Slack thread: "Is it the model?" "Is it prompt drift?" "Is it the embeddings?" "Try swapping in GPT?" That's not debugging — that's guessing with an API key.

Langfuse is an open-source LLM observability platform. It provides traces (a single user request), nested spans (each pipeline step), generations (LLM calls with tokens, cost, and latency), and scores attached by human or automated evaluation. The goal is simple: open that bad request and see exactly which step went wrong.

The Problem You're Actually Trying to Solve

A typical RAG path looks like this:

1. Retrieve top-k chunks
2. Build a prompt from those chunks
3. Call the model
4. Return the answer

Without instrumentation, production only shows you step 4. Langfuse makes steps 1–3 visible in the same call tree, along with the latency and cost of each LLM call.

The problem is never "the model sucks" — it's "the retrieval layer quietly fed the wrong documents to the model." The value of observability is exposing this kind of silent failure.

LLM Observability is the adaptation of software observability's three pillars — logs, metrics, and traces — to LLM-centric pipelines. Traditional APM tools typically only see the latency and status code of an HTTP request; they can't answer questions like "which text chunks actually got concatenated into the prompt," "how many tokens and dollars did this call cost," or "what did the similarity distribution of retrieved results look like." LLM observability therefore introduces the span as its basic unit: a span represents one step in the pipeline and can carry structured fields like input, output, latency, and cost; multiple nested spans form a trace, corresponding to one end-to-end user request. Under this model, a generation is just one span type among many, specifically recording the model name, token usage, and cost of an LLM call.

Installation and Credentials

pip install langfuse openai

Set the environment variables (Cloud and self-hosted use the same SDK):

export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_BASE_URL="https://cloud.langfuse.com"   # EU
# US: https://us.cloud.langfuse.com
# JP: https://jp.cloud.langfuse.com

Optional connectivity check (don't use this on the hot path):

from langfuse import get_client
get_client().auth_check()

Langfuse supports multi-region deployments (EU / US / JP / HIPAA / self-hosted). LANGFUSE_BASE_URL must match the region used when the project was created.

Drop-in OpenAI: Change One Import

If you're already using the OpenAI Python SDK, Langfuse's drop-in approach requires only an import change to automatically record prompts, completions, usage, cost, latency, and API errors:

# before: import openai
from langfuse.openai import openai
# or: from langfuse.openai import OpenAI, AsyncOpenAI

completion = openai.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Answer only from the context."},
        {"role": "user", "content": "What is the refund window?"},
    ],
    name="support-answer",  # shows up as the generation name in Langfuse
)
print(completion.choices[0].message.content)

You can also pass Langfuse attributes via metadata on the same call (for example langfuse_session_id, langfuse_user_id, langfuse_tags).

Nesting the Whole Pipeline with @observe

A single OpenAI call is just a generation, but a RAG pipeline needs a trace tree. Decorate each step:

from langfuse import observe, get_client, propagate_attributes
from langfuse.openai import openai

@observe(name="retrieve")
def retrieve(query: str, k: int = 4) -> list[dict]:
    return search_chunks(query, k=k)

@observe(name="build_prompt")
def build_prompt(query: str, chunks: list[dict]) -> list[dict]:
    context = "\n\n".join(c["text"] for c in chunks)
    return [
        {"role": "system", "content": "Answer only from the context. If missing, say you don't know."},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"},
    ]

@observe(name="rag_answer")
def rag_answer(query: str, user_id: str, session_id: str) -> str:
    with propagate_attributes(
        user_id=user_id,
        session_id=session_id,
        tags=["support-rag"],
        metadata={"pipeline": "support-rag-v2"},
    ):
        chunks = retrieve(query)
        messages = build_prompt(query, chunks)
        completion = openai.chat.completions.create(
            model="gpt-4o-mini",
            messages=messages,
            name="generate",
        )
        return completion.choices[0].message.content

Nested @observe calls automatically form parent-child spans, and OpenAI drop-in generations nest under the current active observation.

The Key Workflow: Debugging a Bad RAG Answer

This is the part that actually ships.

Step 1: Find the Failing Request

Locate the trace by user_id / session_id / time window. You should see a structure like this:

• rag_answer (root span)
  • retrieve — input query, output chunks (ID, score, text snippet)
  • build_prompt — the assembled messages
  • generate — model, tokens, cost (USD), latency, completion text

Step 2: Work Bottom-Up from the Answer

If generate's output is fluent but wrong, don't start by swapping the model. Open retrieve and check:

• Are the top chunks from the correct document / product / region?
• Are the similarity scores abnormally low, or clustered on boilerplate text?
• Did a filter drop the expected policy paragraph?

In the "wrong policy" case, the generate span is innocent. The retrieve span shows high-confidence chunks from an outdated refund FAQ. The root cause is a stale index or a missing metadata filter — not an "LLM hallucination."

Step 3: Attach a Score so the Failure Becomes Searchable

Within the current observation (or after the fact via create_score with a trace_id):

from langfuse import get_client
langfuse = get_client()

langfuse.score_current_span(
    name="retrieval_relevance",
    value=0.2,
    data_type="NUMERIC",
    comment="Top chunks from outdated refund FAQ; annual-plan policy missing",
)
langfuse.score_current_trace(
    name="answer_correctness",
    value=0,
    data_type="BOOLEAN",
    comment="Wrong refund window cited",
)

Score types support NUMERIC, CATEGORICAL, BOOLEAN, and TEXT.

Step 4: Fix the Broken Span and Re-run the Same Query

The new trace should show better retrieve output and a passing score. This pair of "before-fix / after-fix" traces plus scores is your regression evidence for the next prompt or index change.

"Working bottom-up" here means walking the call chain of the trace tree from the leaf nodes back to the root, rather than starting from the final answer and guessing at causes. In a RAG request, the root span is the whole pipeline, and the leaves are the specific LLM calls and retrieval functions. When debugging, first look at the step closest to the output that also depends most on upstream input (usually retrieve), and check whether the chunks it returned contain the correct documents, whether similarity scores are reasonable, and whether filtering conditions mistakenly removed a key paragraph. Only when the evidence shows the retrieval layer is fine, the prompt assembly is correct, and the model parameters are right should you escalate your suspicion to the model itself. The reason this works is that an LLM's bad output is often an amplification of an upstream error rather than an independent failure: a stale index, a broken metadata filter, or poor chunk splitting will cause the model to produce a wrong answer while staying "faithful to the given context," yet from the outside it looks like a hallucination.

Common Pitfalls

Flush in short-lived processes. The SDK uploads in background batches, so scripts and serverless handlers should call get_client().flush() (or shutdown()) before exiting — otherwise the last batch of events is lost.

@observe handles large I/O. Huge chunk dumps add overhead. You can disable it per decorator with capture_input=False / capture_output=False, or globally via LANGFUSE_OBSERVE_DECORATOR_IO_CAPTURE_ENABLED.

Regions must match. LANGFUSE_BASE_URL must match the region used when the project was created; a wrong base URL looks like "nothing is showing up."

Limited Assistants API coverage. The OpenAI Assistants path (server-side state) is not fully covered by the drop-in. Wrap it with @observe outside your own orchestration instead.

Sampling. High-traffic apps can set LANGFUSE_SAMPLE_RATE (or Langfuse(sample_rate=...)), but keep paths you're actively debugging at 100% sampling.

Getting Started Checklist

1. pip install langfuse openai, set LANGFUSE_PUBLIC_KEY / SECRET_KEY / BASE_URL
2. Switch to from langfuse.openai import openai to automatically record generations
3. Add @observe(name=...) to retrieve / prompt / answer (and any tools)
4. Propagate user_id, session_id, and tags early with propagate_attributes
5. When you hit a bad answer, open the trace → check the retrieve span first, blame the model later
6. Use score_current_span / score_current_trace to make failures filterable
7. Remember to flush() in scripts and serverless

When your production AI answer goes wrong, do you open a trace — or open a Slack guessing thread?