LangSmith Practical Guide: Building Full LLMOps Workflows with Python

LangSmith is the LLMOps-focused component within the LangChain ecosystem. If LangChain handles building agents and LangGraph orchestrates complex stateful workflows, then LangSmith takes care of the full observability stack — tracing, debugging, evaluation, and monitoring. This article is based on an approximately one-hour hands-on tutorial, walking through LangSmith's core capabilities and practical usage to help you operationalize the DevOps side of your agent applications.

What Is LangSmith: Positioning and Setup

At its core, LangSmith is an MLOps/AIOps platform for AI applications, covering monitoring, observability, evaluation, and tracing. It does not provide model capabilities — you still need an inference provider like OpenAI, Anthropic, Google, or a local Ollama instance. LangSmith's job is simply to record and analyze what happens during those calls.

Integration is lightweight: after registering and creating an API key in settings, you configure three types of variables in your project's .env file — the LangSmith API key, your project name, and the LANGSMITH_TRACING=true flag, plus your model provider's key (e.g., an OpenAI API key). The tutorial uses uv as the package manager; installing langsmith, openai, and python-dotenv is all you need to get started. It's worth emphasizing that everything covered in this article is available on the free tier — a paid hosted plan is not required for onboarding.

Tracing: Your Core Tool for Pinpointing Bottlenecks

Tracing is LangSmith's most important feature. The simplest approach is to wrap the OpenAI client with wrap_openai, which automatically logs every chat.completions.create call to LangSmith — including inputs, outputs, token counts, and cost.

But tracing a single Q&A call has limited value. The real power shows when call chains grow complex. The tutorial demonstrates a describe function that internally calls count_words to count words before passing the result to the model to judge text length. Simply decorating these functions with @traceable causes them to appear as nested nodes in the trace tree. You can clearly see that count_words took 0.00 seconds and isn't the bottleneck — the real time cost is the call to OpenAI. When code breaks (for example, mistakenly using choice instead of choices), the error is fully recorded for easy debugging.

Tracing gets even more interesting when you connect a real agent. The tutorial builds an agent using langchain.agents' create_agent with a get_favorite_food tool. Because LangChain and LangSmith share the same ecosystem, you don't even need to import langsmith — environment variables are auto-detected and tracing is enabled automatically. In a LangGraph-style trace record, you can drill down layer by layer to see which tool the model decided to call, what the tool returned, and the latency and cost at each step. The tutorial also shows that tools from MCP servers are traceable in the same way — to LangSmith, whether a tool is defined in code or comes from MCP, it all looks like "an agent called a tool."

The trace UI also supports filtering by conditions — for example, showing only traces with latency greater than 2 seconds to quickly isolate slow requests. Two terms are worth distinguishing here: a run refers to a single invocation, while a trace refers to the complete higher-level chain.

The @traceable decorator is the core mechanism provided by the LangSmith Python SDK. It's essentially a function wrapper that automatically reports metadata to the LangSmith backend before and after function execution: function name, input arguments, return values, execution time, exception info, and more. Decorated functions become independent nodes (called spans) in the trace tree, and multi-level nested calls naturally form a parent-child tree structure. This closely resembles the OpenTelemetry tracing model used in distributed systems — LangSmith actually supports ingestion via the OpenTelemetry protocol, which means non-Python stacks (like Node.js or Java) can also push trace data to LangSmith. wrap_openai is a higher-level convenience wrapper that monkey-patches the OpenAI SDK client object, enabling tracing without modifying existing call code — ideal for quickly integrating into existing projects.

Datasets and Experiments: Quantifying Model Performance

Datasets and Experiments are two related concepts. A dataset is a collection of input-output pairs with correct reference outputs; an experiment runs a model over a dataset, compares model outputs against reference outputs, and produces evaluation results.

The tutorial covers two evaluation paradigms:

Exact Match Evaluation

For tasks with deterministic answers — like sentiment classification — first build a dataset with client.create_dataset and client.create_examples, then write an inference function and an exact_match evaluator (directly comparing whether labels are identical, returning a boolean). Finally, call client.evaluate, passing in the inference function, dataset name, list of evaluators, and an experiment prefix. After running, you can see accuracy, latency, and other metrics updating in real time in the UI.

LLM-as-a-Judge

For open-ended questions like "What do mammals and birds have in common?", answers don't need to match word for word — semantic correctness is sufficient. Here, another model acts as the judge: it receives the question, the reference output, and the actual output, and determines whether the answer is "semantically correct," returning yes or no.

There's an illuminating detail here: by modifying the judge's system prompt — changing it from "is this semantically correct" to "does this precisely focus on a single answer with no extra information" — the tutorial causes evaluations that previously passed 100% of the time to start failing. This demonstrates that the wording of your evaluation criteria directly determines results. Adding a system prompt to the inference function ("answer concisely, give only one answer") subsequently raises the pass rate again.

LangSmith also supports experiment comparison: selecting two runs enables a side-by-side comparison of which is faster, uses fewer tokens, costs less, and achieves higher accuracy, with differences highlighted. This is the go-to tool for evaluating different prompts or models against each other.

You can also generate datasets retroactively from real trace records — run a batch of questions through the model, use feedback markers to filter out the desired runs with client.list_runs combined with filter query expressions (e.g., feedback_key equals a given marker, feedback_score equals 1), and bulk-export them as a dataset.

LLM-as-a-judge is one of the dominant paradigms in LLM evaluation today. The underlying rationale is that for semantic-level judgments, language models tend to align more closely with human evaluation standards than hard-coded rules or string matching. A common implementation involves inserting the "question, reference answer, and candidate answer" into a structured prompt and having the judge model output a yes/no or 1–5 score. The biggest weakness of this approach is exactly what the article reveals: the judge's verdict is highly sensitive to prompt wording, and different descriptions of the evaluation criteria can cause pass rates to swing from 100% to 0%. This makes the reproducibility of evaluations and the stability of criteria critically important. In practice, it's generally recommended to version-control the judge prompt itself and to periodically run human spot-checks to verify that the judge model's verdicts still align with business expectations.

Online Evaluation, Playground, and Prompt Management

Beyond offline dataset evaluation, LangSmith supports configuring online evaluators — always-on evaluators that automatically assess every new call as it arrives. The tutorial creates a "conciseness judge" and binds it to the tracing project. Note that this consumes your own API key quota. Since evaluation takes time, results won't appear instantly, but the requests are already logged and results will appear after a refresh.

The Playground is an experimentation environment for comparing different models, debugging system prompts, and testing output schemas. The tutorial finds that one model is naturally more concise while another is "verbose" — even adding prompts like "always be extremely concise" and "no code examples" fails to rein it in. Setting a JSON output schema ultimately yields more concise results. Satisfactory configurations can be saved as Prompts with one click, complete with version history and commit records. These can later be pulled into Python code with client.pull_prompt and used directly with LangChain's pipeline syntax (prompt | model).

Human Annotation, Monitoring, and Alerts

Annotation Queues are designed for scenarios requiring human evaluation. You can add specific runs to a queue so that humans can review model responses one by one, scoring them on dimensions like quality, conciseness, and whether tool calls were necessary. Note that the feedback feature is fundamentally intended for human assessment of run quality, not purely as a tagging mechanism (though in practice it's often used to filter high-quality samples as well).

Monitoring is split across two tabs: the Dashboard provides quantitative overviews of latency, LLM calls, tool usage, and more. More practically useful are Alerts. The tutorial demonstrates setting a rule that triggers a notification when "average latency over the past 5 minutes exceeds 1 second," with notification channels including Slack, PagerDuty, Datadog, or a custom webhook. After testing with webhook.site, a single request triggering 1.98 seconds of latency results in a POST notification containing the alert ID and rule details.

Hands-On Case Study: Document Extraction Pipeline

The tutorial closes with an example close to real-world usage: invoice PDF extraction. The workflow first uses Mistral OCR to recognize text from scanned documents (traceable), then uses Google Gemini (via Vertex AI) to generate structured output — a workflow spanning multiple providers.

One particularly instructive detail is custom cost accounting. LangSmith doesn't recognize Mistral OCR's pricing by default, and OCR is billed per page rather than per token. The workaround in the tutorial is to create a custom model in the pricing settings, entering "$4,000 per million pages" into the "price per million tokens" field to simulate per-page billing — this way, the ~$0.004 per-page cost displays correctly in the trace. You also need to set metadata in @traceable to indicate that mistral-ocr-latest is being used. After running, it's clear that the nearly 6-second latency for the entire chain comes primarily from Gemini Flash inference, not from OCR.

Mistral OCR is a document understanding service from Mistral AI that can process scanned PDFs, images, and other unstructured documents, outputting structured text or Markdown. Unlike traditional rule-based OCR engines, it uses a vision-language model as its backbone, giving it higher fault tolerance for complex layouts such as tables, multi-column text, and handwriting. It's billed per page rather than per token — which is exactly the context behind the custom cost accounting in the article. LangSmith's default cost model is token-based and can't directly map to per-page billing, requiring the workaround of expressing page counts as token fields to keep cost data readable in trace records. Vertex AI is Google Cloud's managed AI platform; Gemini series models can be called via Vertex AI endpoints, making it well-suited for enterprise scenarios that need to deploy on GCP infrastructure.

Summary

LangSmith advances AI applications from "it runs" to "it's observable, evaluable, and operable." Tracing helps you pinpoint bottlenecks and errors; datasets and experiments make model performance quantitatively comparable; online evaluators and alerts ensure production stability; Playground and Prompt management accelerate iteration; and annotation queues introduce human oversight. For developers building agent applications, this toolchain is the key capability for moving experimental prototypes into production. The free tier is more than sufficient to complete the full onboarding workflow — the recommendation is to work through it with your own real use cases rather than simply replicating the tutorial, since hands-on experience with real problems is what truly builds mastery.