LangChain and LangGraph in Practice: End-to-End Implementation of a Production-Grade RAG System

Building a RAG system that's actually ready to deploy has never been about the retrieval step itself. Many teams can get a demo running with LangChain in a day, but once deployed to production, a cascade of problems emerges: retrieval quality degrades, hallucinations become frequent, latency spikes, and costs spiral out of control. Based on a complete technical implementation guide, this article breaks down the engineering path from prototype to production-grade RAG, covering document ingestion, hybrid retrieval, guardrails, LangGraph orchestration, monitoring and evaluation, and Kubernetes deployment.

Where the Gap Between Prototype RAG and Production RAG Lies

The original article uses a comparison table to clearly outline the essential differences. In the prototype phase, ingestion is manual and one-time, retrieval relies solely on vector similarity, generation uses basic prompts, evaluation is done by eyeballing, and deployment lives in a local Notebook. Production, by contrast, demands automated, incremental, and monitorable ingestion; retrieval must be hybrid (dense vectors + sparse retrieval + reranking); generation requires structured output, citation annotation, and guardrails; evaluation needs automated metrics combined with LLM-as-Judge and A/B testing.

The essence of this gap: in production RAG, roughly 10% of the work is on retrieval strategy, and 90% is on engineering capabilities. The original article emphasizes this point at the end as well—observability, guardrails, and evaluation must be built in from day one; bolting them on later is 10 times harder.

System Architecture Overview

The entire pipeline is divided into four core stages, each with corresponding engineering enhancements:

Ingestion pipeline handles chunking, metadata management, deduplication, and incremental updates; embedding and indexing involves vector database selection (the original uses Qdrant), quantization, and replica strategies; retrieval pipeline implements dense + sparse hybrid search, reranking, and metadata filtering; generation stage handles structured output, citation generation, and guardrail validation.

The key design principle of this architecture is that each stage can be scaled and monitored independently.

Document Ingestion: Semantic Chunking and Incremental Processing

Why Fixed-Length Chunking Doesn't Work

The original implements a SemanticChunker class. The core logic first splits by Markdown heading hierarchy, then aggregates paragraphs within each section, keeping tokens under 512 while preserving a 50-token overlap window.

This chunking approach preserves a piece of information that is easily overlooked but crucial to retrieval quality—header_path. For example, if a chunk comes from the path "System Architecture > Retrieval Pipeline," this hierarchy information is stored in metadata, providing context for subsequent retrieval and citation generation.

Deduplication and Incremental Updates

Another engineering point of the ingestion pipeline is idempotency. The code computes a SHA256 hash of the document content (taking the first 16 characters) as the chunk_hash, checking whether it already exists before ingestion. Batch ingestion uses asyncio.Semaphore(10) to control concurrency, avoiding excessive pressure on the embedding API.

Key Design Decisions

• Each document generates an MD5 hash as its document_id
• Content hashes are used for exact deduplication; production environments should also add MinHash for approximate deduplication
• Failed documents enter a dead-letter queue, supporting resumable processing

Hybrid Retrieval: Dense + Sparse + RRF Fusion + Reranking

The Four-Stage Retrieval Pipeline

The original's HybridRetrievalPipeline implements a four-stage retrieval process—the most technically sophisticated part of the entire system:

Stage one runs dense and sparse retrieval in parallel. Dense retrieval uses an embedding model to convert the query into a vector and searches Qdrant; sparse retrieval uses a BM25-like approach to match keywords. Both are invoked in parallel via asyncio.gather.

Stage two merges the two result sets using RRF (Reciprocal Rank Fusion). The core formula is score += weight / (k_constant + rank + 1), where k_constant defaults to 60, with a dense weight of 0.7 and a sparse weight of 0.3. The advantage of RRF is that it doesn't rely on the absolute values of raw scores, only on rankings, avoiding the problem of incomparable scores across different retrievers.

Stage three reranks the fused results using a Cross-Encoder model (the original recommends ms-marco-MiniLM-L-6-v2). The Cross-Encoder encodes both query and document together, producing a more precise relevance score.

Stage four takes the Top-K results, with k=5 by default and an initial recall of fetch_k=20.

RRF (Reciprocal Rank Fusion) is a multi-source retrieval fusion algorithm that requires no training and no normalization. Its core idea: it doesn't care about the raw similarity scores from each retriever (since dense retrieval's cosine similarity and BM25 scores have completely incomparable distributions), only about each document's rank position in each result set. In the formula score(d) = Σ_r weight_r / (k + rank_r(d)), k is a smoothing constant (typically 60) that suppresses the excessive influence of top ranks; weight_r controls each retriever's contribution. RRF's advantages are strong robustness, simple implementation, and insensitivity to anomalous scores—so it's been adopted as the default fusion strategy by mainstream retrieval systems like Elasticsearch and Weaviate. Its downside is the loss of information from raw scores; when one retriever has extremely high confidence, that can't be reflected.

Cross-Encoder is fundamentally different from the common Bi-Encoder. A Bi-Encoder encodes the query and document independently into vectors, then computes cosine similarity—fast but with limited precision, suitable for large-scale first-stage recall. A Cross-Encoder concatenates [query, document] and feeds them together through a Transformer in one pass, letting the attention mechanism fully model their interaction and output a direct relevance score. Precision is significantly higher, but precomputation is impossible—each candidate document must be inferred in real time. Therefore it can only be used to rerank small candidate sets of Top-20 to Top-100. ms-marco-MiniLM-L-6-v2 is a small model distilled and trained on the MS MARCO passage ranking dataset, with 6 layers and roughly tens of milliseconds per inference—an extremely cost-effective reranking choice for production.

Why Hybrid Retrieval Is a Production Standard

Pure vector retrieval performs poorly on exact keyword matching (such as product model numbers, names, or abbreviations), while pure keyword retrieval can't capture semantic similarity. Hybrid retrieval leverages the strengths of both through RRF fusion and is currently recognized as the production-grade best practice.

Generation Stage: Structured Output, Citations, and Guardrails

Constraining LLM Output with Pydantic

The original defines a CitedAnswer model with four fields: answer (a response with inline citations), citations (a list of citations with doc_id, snippets, and metadata), confidence (a value from 0 to 1), and needs_clarification (whether clarification is needed).

Using LangChain's JsonOutputParser together with a Pydantic model, the LLM's output is constrained to structured JSON. The prompt explicitly requires "answer only from the provided context," and returns "I don't have enough information to answer this question" when context is insufficient.

Pydantic is the most mainstream runtime data validation library in the Python ecosystem. It defines data models through Python type annotations, automatically performing type conversion, range validation, and nested structure validation at instantiation time. LangChain's JsonOutputParser together with PydanticOutputParser can automatically convert a Pydantic model into JSON Schema, inject it into the prompt to guide the LLM toward structured output, and automatically validate it during parsing. Compared to simple string parsing, this approach provides detailed error messages when the LLM's output format is wrong (which field is missing, which type doesn't match), making it easier to trigger retries or fallbacks. The confidence field in CitedAnswer uses Field(ge=0, le=1) to constrain its range, while needs_clarification: bool lets the model proactively declare when information is insufficient, avoiding forced answers. This combination of "structured output + validation" is a universal means of suppressing hallucinations in production—not only constraining format but also, through field design, forcing the model to explicitly expose uncertainty.

Three-Layer Guardrail Mechanism

The RAGGuardrails class implements two layers of checks—input and output:

Input checks intercept two types of risk: prompt injection (detecting trigger phrases like "ignore previous instructions," "system prompt," "jailbreak") and PII leakage (using regex to match SSNs, emails, credit card numbers, and phone numbers).

Output checks include hallucination detection and PII leakage detection. The hallucination detection logic is quite practical: extract all numbers from the answer and check whether they can all be found in the context. Any number longer than 2 digits that doesn't appear in the context is flagged as an "unverified claim."

Output sanitization replaces sensitive information with [REDACTED] when PII is detected.

LangGraph Orchestration: A RAG Workflow with Retries

State Graph Design

The original uses LangGraph to build a three-node workflow: retrieve → generate → guardrail. RAGState defines the state structure, including the message list (annotated with add_messages), the question, retrieval results, the answer, a guardrail-passed flag, and a retry counter.

LangGraph is a stateful multi-step orchestration framework introduced by the LangChain team, specifically addressing the problem that traditional Chains can't express complex control flow like loops, branches, and retries. Its core abstractions include: StateGraph (declares the global state structure), Node (receives state and returns state updates), Edge, and ConditionalEdge (routes to different downstream nodes based on state). State is defined with TypedDict, and the Annotated[list, add_messages] annotation on a field indicates that the field uses an "append-merge" strategy rather than overwriting across node invocations. Checkpointer is a key feature of LangGraph: after each node executes, it automatically persists a state snapshot to a backend (memory, SQLite, Postgres, Redis), thereby supporting conversation resumption after interruption, human-in-the-loop approval, and fault recovery. Production environments typically use PostgresSaver or RedisSaver to ensure shared state across multiple instances. This model of "graph-based orchestration + persisted state" upgrades RAG from a single call into an observable, replayable, self-healing workflow.

Conditional Routing and Self-Healing

If the guardrail node detects an issue (such as hallucination) and the retry count is less than 2, the workflow returns to the generate node to regenerate. The conditional routing function route_after_guardrail decides whether to end or fall back based on the guardrail_passed flag.

This design gives the system self-healing capability—low-quality answers aren't returned directly to the user but trigger regeneration.

Checkpoint Persistence

SqliteSaver is passed in at compile time as the checkpoint store, supporting conversation state persistence. Production environments should switch to a Postgres or Redis backend.

FastAPI Service and Deployment Options

API Design Highlights

The /query endpoint receives the question, filter conditions, Top-K, and whether to enable reranking, and returns a response containing the answer, citations, confidence, retrieval count, and latency. Input guardrails intercept violating queries right at the request entry point, and metrics logging is written asynchronously via FastAPI's BackgroundTasks, without blocking the main request.

Additional endpoints include /ingest (async ingestion), /health (health check), and /metrics (system metrics).

Containerization and K8s Deployment

The Dockerfile is based on python:3.11-slim and includes health check instructions. The Kubernetes deployment is configured with 3 replicas, resource requests of 2Gi memory/1000m CPU, and limits of 4Gi/2000m. Environment variables inject API keys via Secrets, along with livenessProbe and readinessProbe configurations.

Dependency List

The core dependencies listed in the original include: langchain==0.2.0, langgraph==0.1.0, langchain-openai==0.1.0, qdrant-client==1.10.0, sentence-transformers==3.0.0, fastapi==0.110.0, and others.

Evaluation and Monitoring: The Lifeline of a Production System

Retrieval Evaluation Metrics

The original implements three retrieval metrics: Precision@K, Recall@K, and MRR. Precision@K measures the proportion of relevant documents in the Top-K results, Recall@K measures the proportion of all relevant documents that were retrieved, and MRR (Mean Reciprocal Rank) measures the average of the reciprocal ranks of the first relevant document.

Generation Evaluation

Generation quality is evaluated using LLM-as-Judge, covering four dimensions: accuracy, completeness, citation quality, and faithfulness. The original recommends preparing a golden test set of 100+ items and running regression tests in CI/CD.

Production Monitoring Dashboard

The ProductionMonitor class records the latency, retrieval count, citation count, confidence, and user feedback for each query, supporting statistical queries by time window. Monitoring dimensions include: latency percentiles (p50/p95/p99), error rate, confidence distribution, user feedback positive/negative ratio, and per-query cost tracking.

LLM-as-Judge is the current mainstream method for evaluating the quality of generative systems. Since open-ended answers can't be adequately measured by n-gram overlap metrics like BLEU or ROUGE, in practice a strong model (such as GPT-4) scores candidate answers according to a predefined rubric. The four dimensions in the original—Accuracy: whether the answer is factually correct; Completeness: whether it covers all key points of the question; Citation Quality: whether citations support the corresponding claims; Faithfulness: whether all assertions in the answer can be grounded in the context—essentially cover the main failure modes of RAG systems. In practice, note that LLM-as-Judge has position bias (preferring options that appear first), length bias (preferring longer answers), and self-preference (preferring outputs from models of the same family). Therefore the golden test set needs manual annotation for calibration, and CI/CD should use fixed seeds for regression comparisons to avoid false positives caused by judge drift.

Key Points from the Production Checklist

The original provides specific checklists across six dimensions: ingestion, retrieval, generation, evaluation, monitoring, and operations. A few key thresholds: retrieval latency p95 should be under 200ms, generation latency p95 should be under 2s, and rollback capability should complete within 5 minutes. On the operations side, it recommends blue-green deployment, monthly backup and recovery drills, vector index rebuild procedures, and embedding model version management.

Key Takeaways

The core challenge of a production-grade RAG system isn't the retrieval algorithm itself, but the engineering system built around it. Hybrid retrieval (dense + sparse + RRF fusion + Cross-Encoder reranking) is the key path to improving retrieval quality; structured output combined with guardrails is the baseline for controlling hallucinations and compliance risk; LangGraph's conditional routing gives the system self-healing capability; and the evaluation and monitoring system is the cornerstone of long-term stable operation.

One-line Summary

90% of production-grade RAG work is engineering: hybrid retrieval, guardrails, orchestration, monitoring, and deployment.

Paragraph Summary

The article systematically outlines the engineering path from prototype to production RAG, emphasizing that the real difficulty isn't the retrieval algorithm but the roughly 90% of work that is engineering. It covers four major stages: document ingestion (semantic chunking, header_path preservation, SHA256 deduplication, and incremental updates), hybrid retrieval (a four-stage pipeline with parallel dense + sparse retrieval, RRF fusion, and Cross-Encoder reranking), the generation stage (Pydantic structured output, two-layer input/output guardrails, number-level hallucination detection), and a LangGraph-orchestrated self-healing workflow with retries. Engineering implementation covers FastAPI service design, Docker/K8s deployment, dependency version management, and a complete monitoring system with retrieval metrics (Precision@K/Recall@K/MRR) and LLM-as-Judge generation evaluation. The article's core conclusion: observability, guardrails, and evaluation must be built in from day one—bolting them on later costs 10 times more.