AI Gateway Observability: Tracing LLM Calls with OpenTelemetry

As teams move from experimental LLM playgrounds to production-grade applications, the AI gateway has become a standard component. It sits between application code and multiple model providers, handling routing, caching, rate limiting, and cost control. But without deep observability, teams are flying blind. When users complain that responses are "slow" or "weird," can you tell whether it's a network timeout, a model hallucination, or a cache miss triggering a costly repeated inference?

Traditional APM tools fall short under AI workloads because they lack semantic understanding of LLM interactions—they see only the HTTP request, not the semantic spans that actually matter: the prompt that was sent, the tokens consumed, the finish reason, and the semantic relationship between retrieval steps (RAG) and final generation. This article walks you through implementing a robust OpenTelemetry (OTel) tracing pipeline.

Why Standard Tracing Fails for AI Gateways

Traditional distributed tracing focuses on where latency lives—a database query or an API call. In AI systems, the question is often why the model behaves the way it does. A standard HTTP trace only tells you that /api/chat took 3.2 seconds—it doesn't tell you that 2.5 seconds of that was spent on the LLM inference engine generating tokens, or that the prompt was 4,000 tokens long (and therefore expensive).

AI workflows are typically multi-step. A typical RAG pipeline includes: embedding the user query, searching the vector database, reranking results, and feeding the context plus query into the LLM. Without linking these operations into a single trace, you can't diagnose problems like "the vector store returned irrelevant results, causing the LLM to hallucinate." OpenTelemetry's Span primitive lets us nest these operations into a hierarchical view of the request lifecycle.

OpenTelemetry (OTel) is an open-source observability framework under the CNCF that provides a unified API, SDK, and protocol (OTLP) for generating, collecting, and exporting three types of telemetry signals: Traces (distributed tracing), Metrics, and Logs. Its core data model is the Span—a structured record representing a single operation, containing a name, start/end times, status, attributes, and parent-child relationships to other spans. A trace is a directed acyclic graph formed by a set of spans sharing the same trace ID. OTel's value lies in vendor neutrality: the same instrumentation can be exported to Jaeger, Tempo, Datadog, or any other backend, avoiding the vendor lock-in of early APMs.

GenAI Semantic Conventions are the attribute naming conventions the OTel community has defined for generative AI scenarios (e.g., the gen_ai.* prefix), intended to make LLM instrumentation interoperable across languages and vendors. They specify standard keys for fields like model name, token usage, request parameters, and finish reason, so that tools like Jaeger and Grafana can understand LLM calls without custom parsing. These conventions are still evolving rapidly, but aligning with them can significantly reduce future migration costs.

AI Gateway Observability Pipeline Architecture

Define the architecture before writing code. An AI gateway typically acts as a reverse proxy or SDK wrapper. The data flow looks roughly like this: the gateway receives an inbound HTTP request and creates a root span representing the entire request; it then creates child spans for specific operations—retrieval.search (fetching context from the vector store), llm.inference (calling the model provider), llm.generate (streaming chunks). Each span is enriched with attributes defined by the OpenTelemetry GenAI semantic conventions, such as gen_ai.model.name and gen_ai.usage.prompt_tokens. Finally, spans are batch-exported via OTLP to a collector, which routes them to Jaeger or Tempo for storage and visualization.

Semantic Conventions: What to Record

To make traces readable by standard tools, follow the emerging OpenTelemetry GenAI semantic conventions. Although these conventions are still evolving, defining a consistent attribute set now can save you from rewriting your instrumentation later.

Key attributes include:

Attribute Key	Description	Example Value
gen_ai.system	Vendor or system name	openai, anthropic, local
gen_ai.request.model	Requested model identifier	gpt-4-turbo, claude-3-opus
gen_ai.request.temperature	Sampling temperature	0.7
gen_ai.response.model	Actual responding model	gpt-4-turbo-2024-04-09
gen_ai.usage.prompt_tokens	Prompt token count	150
gen_ai.usage.completion_tokens	Response token count	1024
gen_ai.prompt	Actual prompt content	"..." (mind PII)
gen_ai.completion	Actual completion content	"..." (mind PII)

Do not record raw prompt/completion content in spans unless you have a strict PII redaction pipeline. Even though spans are typically sampled at a lower rate than metrics, they are still recorded.

PII (Personally Identifiable Information) refers to data that can be used to identify a specific natural person, such as names, emails, phone numbers, ID numbers, and IP addresses. In LLM scenarios, prompts naturally may carry user-uploaded documents, conversation history, and account credentials, so the risk of recording raw text in spans is far higher than in traditional API tracing. Beyond compliance pressure (GDPR, CCPA, PIPL), span backends are often low-privilege systems shared across multiple teams, and a leak has an enormous blast radius.

Redaction and Pseudonymization are two different strategies: the former replaces sensitive fragments with placeholders (e.g., [EMAIL_REDACTED]), making the information irreversible; the latter replaces them with hashes or random IDs, preserving correlatability for debugging. In practice, character-level regex redaction of prompts while still recording the raw text is not recommended—regex struggles to cover all sensitive patterns and misses semantic-level leaks (e.g., "my social security number is..." followed by digits). A more robust approach is to not store the full text in spans at all, but rather store it in a secure log store keyed by trace ID.

Implementing Instrumentation for the Gateway in Python

Let's build a minimal AI gateway with FastAPI and httpx, and instrument it with opentelemetry-sdk and opentelemetry-exporter-otlp. Install the dependencies first:

pip install fastapi uvicorn httpx opentelemetry-sdk opentelemetry-exporter-otlp-proto-http opentelemetry-instrumentation-fastapi opentelemetry-instrumentation-httpx

Configuring the Tracer Provider

Initialize the OpenTelemetry SDK and set up the environment for span generation and export:

# app/telemetry.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased
import os

OTEL_ENDPOINT = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317")
OTEL_HEADERS = os.getenv("OTEL_EXPORTER_OTLP_HEADERS", "")

def get_tracer_provider() -> TracerProvider:
    resource = Resource.create({
        SERVICE_NAME: "ai-gateway",
        "service.version": "1.0.0",
    })
    provider = TracerProvider(resource=resource)
    # In production, consider a parent-based sampling strategy
    sampler = TraceIdRatioBased(0.5)
    exporter = OTLPSpanExporter(
        endpoint=OTEL_ENDPOINT,
        headers=OTEL_HEADERS,
        insecure=True  # For local development only; use HTTPS in production
    )
    processor = BatchSpanProcessor(exporter)
    provider.add_span_processor(processor)
    trace.set_tracer_provider(provider)
    return provider

Gateway Logic and Instrumentation

The core gateway service uses nested spans to distinguish the "retrieval" and "generation" phases. Note the use of tracer.start_as_current_span—this context manager handles parent-child relationships automatically, so vector_db.search and llm.generate are automatically nested inside gateway.handle_chat.

# app/gateway.py
import asyncio
from fastapi import FastAPI
from pydantic import BaseModel
from opentelemetry import trace
from app.telemetry import get_tracer_provider

tracer = trace.get_tracer("ai_gateway")
app = FastAPI()
get_tracer_provider()

class ChatRequest(BaseModel):
    message: str
    user_id: str

async def mock_vector_search(query: str):
    async with tracer.start_as_current_span("vector_db.search") as span:
        span.set_attribute("db.system", "pinecone")
        await asyncio.sleep(0.1)
        return ["Context chunk 1", "Context chunk 2"]

async def mock_llm_call(prompt: str, model: str = "gpt-4"):
    async with tracer.start_as_current_span("llm.generate") as span:
        span.set_attribute("gen_ai.system", "openai")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.request.temperature", 0.7)
        await asyncio.sleep(0.5)
        response_text = f"Response based on: {prompt[:20]}..."
        span.set_attribute("gen_ai.response.model", f"{model}-2024-01")
        span.set_attribute("gen_ai.usage.prompt_tokens", 150)
        span.set_attribute("gen_ai.usage.completion_tokens", 1024)
        span.set_attribute("gen_ai.finish_reason", "stop")
        return response_text

@app.post("/chat")
async def chat(request: ChatRequest):
    with tracer.start_as_current_span("gateway.handle_chat") as root_span:
        root_span.set_attribute("user.id", request.user_id)
        context = await mock_vector_search(request.message)
        final_prompt = f"Context: {str(context)}\nQuery: {request.message}"
        response_text = await mock_llm_call(final_prompt)
        return {"response": response_text, "usage": {"prompt_tokens": 150, "completion_tokens": 1024}}

Context Propagation in Async Python

A common pitfall when instrumenting Python AI gateways is context loss in async code. If you spawn a new asyncio.Task or thread, the OpenTelemetry context (which holds the current trace ID) does not automatically propagate. In the earlier example, we awaited all operations within the same task, so propagation works fine. But if your gateway uses separate worker processes, you must explicitly capture and inject context.

Here's how to propagate context to an httpx request—commonly used by gateways to call LLM providers. The opentelemetry-instrumentation-httpx package largely handles this, but you must ensure the traceparent header is sent:

import httpx
from opentelemetry.trace import SpanKind
from opentelemetry.propagate import inject

async def call_external_llm_with_trace(url: str, payload: dict):
    tracer = trace.get_tracer("http_client")
    with tracer.start_as_current_span("http.client.llm_call", kind=SpanKind.CLIENT) as span:
        span.set_attribute("http.method", "POST")
        span.set_attribute("http.url", url)
        headers = {}
        inject(headers)  # Adds the 'traceparent' header
        async with httpx.AsyncClient() as client:
            try:
                response = await client.post(url, json=payload, headers=headers)
                response.raise_for_status()
                span.set_attribute("http.status_code", response.status_code)
                return response.json()
            except httpx.HTTPError as exc:
                span.set_status(trace.StatusCode.ERROR, str(exc))
                span.record_exception(exc)
                raise

Once the header is injected, the trace stays continuous even if the LLM provider also supports OpenTelemetry (or if you're tracing across microservices). For local LLMs that don't support OTel, at least the client-side latency and status will be recorded.

Context Propagation is the cornerstone of distributed tracing. OpenTelemetry uses contextvars to implicitly pass the current span through async tasks, but this mechanism depends on Python's coroutine context inheritance rules: asyncio.create_task copies the current context, whereas run_in_executor, thread pools, and multi-process workers lose it.

When calling across processes or services, you need a Propagator to serialize context into the transport carrier. The default W3C Trace Context propagator writes a traceparent header in the format 00-{trace_id}-{span_id}-{flags}, which the receiver parses to attach the remote span to the same trace. inject(headers) writes, extract(headers) reads, and they are used in pairs. Note: if the gateway reuses connection pools or uses HTTP/2 multiplexing to downstream services, you still need to inject headers per logical request—you can't assume a single injection stays valid over the long term.

PII and Sensitive Data Redaction

This is the most critical security step in AI tracing. LLM prompts often contain user PII, API keys, or proprietary data. You must never record raw prompt/completion content into spans without a redaction layer.

You can use a utility function to sanitize attributes before attaching them to a span:

import re

def redact_pii(text: str) -> str:
    if not text:
        return text
    text = re.sub(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b', '[EMAIL_REDACTED]', text)
    text = re.sub(r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b', '[PHONE_REDACTED]', text)
    text = re.sub(r'sk-[A-Za-z0-9]{20,}', '[API_KEY_REDACTED]', text)
    return text

A better approach: store the full prompt/completion in a separate security log store with strict access control (such as CloudWatch or Datadog Logs), and only store a hash or reference ID in the OpenTelemetry span. This way developers can debug via the ID in the secure log store while keeping the trace backend clean and compliant.

Visualizing the Data: Jaeger and Grafana

You need a backend to see your work. The simplest way to get started locally is to run Jaeger's All-In-One mode:

docker run -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one

Visit http://localhost:16686 in your browser, search for the service ai-gateway, and you'll see a trace waterfall like this:

gateway.handle_chat (200ms)
├── vector_db.search (50ms)
└── llm.generate (150ms)

Reading the Waterfall

Latency breakdown: If llm.generate accounts for 80% of total time, the problem is inference, not retrieval; if vector_db.search is slow, you need to optimize your index. Error propagation: If llm.generate fails (e.g., 429 Too Many Requests), the root span gateway.handle_chat is also marked as an error thanks to context propagation, making it easy to filter for "all failed requests" in Jaeger. Cost correlation: By exporting gen_ai.usage.* attributes, you can build Grafana dashboards that correlate Trace IDs with cost, identifying expensive but low-quality queries.

Common Questions

Should I trace every LLM call, or only sampled ones? In high-throughput systems, tracing every call is expensive. OpenTelemetry's BatchSpanProcessor and sampling strategies (e.g., TraceIdRatioBased(0.1)) let you sample only 10% of traces. But you should apply a higher head-based sampling rate to error spans—this way, failed LLM calls are always captured even if successful calls are dropped.

How do I handle streaming responses? The challenge with streaming responses (SSE) is that completion_tokens isn't known until the stream ends. You can update span attributes after the stream completes—call span.set_attribute after the async for loop finishes. Don't end the span before the stream is fully processed, or the trace will be incomplete.

Can OpenTelemetry be used with local LLMs like Llama 3? Yes, but you'll need to instrument the client library (e.g., transformers, ollama, vllm). Local LLMs run on your own infrastructure, where network latency is replaced by CPU/GPU computation latency. You can use tools like opentelemetry-instrumentation-nvidia-smi to capture GPU utilization metrics alongside trace spans, giving you the full picture: traces tell you what happened, metrics tell you how the hardware performed.

With this observability layer in place, your AI gateway goes from a "black box" to a transparent, debuggable system, letting you answer complex questions about performance, cost, and quality—and deploy AI applications more safely and reliably in production.

BatchSpanProcessor and SimpleSpanProcessor are the two exporter processors provided by the OTel SDK. The former buffers spans in memory and pushes them in batches via OTLP once a batch size (default 512) or timeout (default 5 seconds) is reached—high throughput and low network overhead, but unflushed spans may be lost on process crash, making it suitable for production. The latter exports synchronously one by one—debug-friendly but poor performance, recommended only for local development.

Sampling strategies fall into two categories: head-based decides once at trace creation whether to sample, with low overhead but no ability to adjust based on later results; tail-based decides after the trace completes based on overall characteristics (e.g., whether it contains errors, whether total latency exceeds a threshold), precisely capturing anomalies but requiring caching all spans on the Collector side, which is resource-intensive. A common combination in practice: use TraceIdRatioBased for low-rate head sampling on the SDK side, and configure tail-based sampling on the Collector to force-keep error or slow traces, balancing cost and observability.

One-line Summary

Build a semantic tracing pipeline for AI gateways with OpenTelemetry, covering instrumentation, propagation, redaction, and sampling practices.

Paragraph Summary

The article argues that traditional APM tools cannot understand the semantics of LLM interactions—they only see HTTP requests and miss key information like prompts, token counts, and finish reasons, leaving teams unable to diagnose slow responses or model hallucinations. To address this, the author proposes a complete OpenTelemetry tracing pipeline for AI gateways: first define a hierarchical architecture with a root span plus child spans for retrieval.search, llm.inference, and llm.generate, then enrich attributes following the GenAI semantic conventions (gen_ai.system, gen_ai.usage.prompt_tokens, etc.). The article provides a full Python instrumentation example using FastAPI and httpx, and focuses on engineering details including async context loss, traceparent injection, PII redaction, sampling strategies, and Jaeger/Grafana visualization.