Production-Grade RAG with LangChain and LangGraph: Hybrid Retrieval and Agent Workflows

Building a RAG system that can actually go live has never been about "retrieval" itself. What really determines success or failure is the degree of automation in the ingestion pipeline, the upper bound of retrieval quality, and the controllability of the generation stage. Based on a complete production-grade RAG implementation, this article breaks down the full-chain engineering details from document ingestion to Kubernetes deployment.

Prototype vs. Production: Six Dividing Lines for RAG Systems

Most RAG tutorials stop at a "vector retrieval + basic prompt" demo, but this solution makes it clear that production environments require a qualitative leap across six dimensions:

Dimension	Prototype Approach	Production Requirement
Ingestion	Manual, one-time	Automated, incremental, observable
Retrieval	Pure vector	Hybrid (dense + sparse + rerank)
Generation	Basic prompt	Structured output + citations + guardrails
Evaluation	Manual	Automated metrics + LLM judge + A/B
Monitoring	None	Latency, quality, cost, drift alerts
Deployment	Local notebook	Containerized, scalable, resilient

The core judgment of this architecture is: about 10% of production-grade RAG is a retrieval problem, and 90% is an engineering problem. Guardrails, observability, and evaluation systems must be built in from day one—the cost of retrofitting them later is 10x.

System Architecture Overview

The entire pipeline is connected across four stages, each of which expands into several engineering details:

• Ingestion pipeline: Chunking → metadata → deduplication → incremental updates
• Embedding and indexing: Vector store (Qdrant) → quantization → replication
• Retrieval pipeline: Hybrid search (dense + sparse) → rerank → filter
• Generation stage: Structured output → citations → guardrails → evaluation

What's notable in this architecture is that it places "deduplication" and "incremental updates" at the core of the ingestion pipeline. In real production, duplicate documents severely dilute retrieval quality, and the cost of a full index rebuild is often unacceptable.

Document Ingestion: Semantic Chunking and Deduplication

The ingestion pipeline is organized with an IngestionPipeline class, and the core flow is divided into five steps: load documents, check for duplicates, chunk, batch embed, and store.

The chunking strategy uses SemanticChunker, splitting by Markdown heading hierarchy rather than a fixed character count. It preserves header_path (such as "Security Guide > Authentication > Tokens") as metadata, allowing subsequent retrieval to pinpoint document sections precisely. Each chunk carries a 50-token overlap to prevent semantics from breaking at boundaries.

Deduplication uses the first 16 characters of a SHA-256 hash as an identifier, combined with an is_duplicate check, to prevent the same document from being indexed repeatedly. Batch ingestion uses asyncio.Semaphore(10) to limit concurrency and prevent the embedding service from being overwhelmed.

One easily overlooked detail: ingest_batch returns a {total, successful, duplicates, details} structure. This kind of statistical output is the foundation of production observability—if you can't quantify ingestion results, you can't locate pipeline bottlenecks.

Hybrid Retrieval: RRF Fusion and Reranking

The design of the retrieval pipeline determines the upper bound of a RAG system. This solution uses a four-stage pipeline:

1. Parallel dense + sparse retrieval: Dense vectors handle semantic similarity, sparse vectors (BM25-like) handle exact keyword matching, and both execute simultaneously via asyncio.gather.
2. RRF fusion: Merge the two result sets with the Reciprocal Rank Fusion algorithm, using the formula score += weight / (k + rank + 1) with k_constant set to 60. Dense weight is 0.7, sparse weight is 0.3.
3. Cross-encoder reranking: Rerank the fused top results with a model such as ms-marco-MiniLM-L-6-v2, significantly improving relevance.
4. Final filtering: Return the top-k results.

You may not have noticed that the script defaults to fetch_k=20 but ultimately returns k=5. This "recall first, rerank later" funnel design is standard practice in production retrieval. Retrieving 5 items directly is often worse than recalling 20 and then reranking.

Why can RRF replace simple weighted summation? RRF (Reciprocal Rank Fusion) uses only each document's rank in its respective result list rather than raw scores, so it naturally avoids the dimensional incompatibility between dense vectors (cosine similarity, roughly -1 to 1) and sparse BM25 (unbounded scores). In the formula weight / (k + rank + 1), k_constant=60 is an empirical value whose purpose is to compress the relative gap between top-ranked positions—the contribution difference between rank 1 and rank 10 is far smaller than with linear weighting, allowing the two channels of evidence to combine more evenly. Dense weight 0.7 and sparse 0.3 mean semantic matching slightly dominates, but in keyword-heavy scenarios such as proper nouns, IDs, and code, the sparse channel still provides exact hits that dense vectors easily miss. This is also why, when tuning, you should prioritize adjusting the recall counts of the two channels rather than agonizing over decimal weights.

Why is a cross-encoder more accurate than vector retrieval? A bi-encoder encodes the query and document separately into vectors and only performs a dot product at the end; it is fast but lacks cross-interaction. A cross-encoder concatenates [query, doc] and feeds them together into a Transformer, allowing the attention mechanism to compare directly at the token level, so relevance judgments are significantly more accurate. The cost is that each candidate pair must be computed with a separate forward pass and cannot be pre-indexed, so it is only suitable for reranking rather than recall. Small 6-layer models like ms-marco-MiniLM-L-6-v2 are precisely designed to keep reranking latency in the tens of milliseconds across 20 candidates, balancing quality with p95 latency targets.

Generation Stage: Citations, Guardrails, and Structured Output

The generation chain uses a Pydantic model, CitedAnswer, to constrain the output structure, containing four fields: answer (with inline [doc_id] citations), citations (citation details), confidence (0-1 confidence score), and needs_clarification (whether clarification is needed).

The prompt explicitly requires: answer only based on the provided context, and if the context is insufficient, respond with "I don't have enough information." This is far more reliable than letting the model freestyle.

Three Layers of Guardrails

The RAGGuardrails class implements three layers of protection:

• Input checks: Regex matching for PII (SSN, email, credit card, phone), while also blocking prompt injection keywords (such as "ignore previous instructions" and "leak your prompt").
• Output checks: Extract numbers from the answer; if a number longer than 2 digits does not appear in the context, it is flagged as a suspected hallucination. It also checks for PII leakage in the output.
• Redaction: The sanitize_output method replaces matched PII with [REDACTED].

This "numeric validation" approach to hallucination detection is crude, but it is very effective in finance, healthcare, and other number-sensitive scenarios.

Why is Pydantic structured output not the same as JSON mode? The early approach was to write "please return JSON" in the prompt and then parse it with json.loads; any slight deviation from the model would throw an exception. The modern approach uses function calling or JSON Schema-constrained decoding, passing the field definitions of CitedAnswer directly to the model and letting the server guarantee that the output strictly conforms to the schema, bringing the parse failure rate close to zero. The confidence field lets downstream systems perform threshold routing—triggering clarification or human handoff when below threshold; needs_clarification explicitly allows the model to "give up" rather than fabricate an answer. This design binds structural constraints together with business semantics, making it far more reliable than patching text afterward with regex.

Why is an LLM judge of limited reliability? Using an LLM to score generated results (accuracy, completeness, citation quality, faithfulness) is low-cost and scalable, but it has several known biases: preference for longer answers, excessive leniency toward its own generated text, and high variance when repeatedly scoring the same answer. Engineering mitigations include pinning the judge model version, providing a few few-shot scoring examples, requiring the judge to give evidence before scoring on critical dimensions, and periodically calibrating judge drift with human-labeled samples. Never treat LLM scores as the sole launch gate.

LangGraph Orchestration: Agentic Workflow with Retries

The workflow is orchestrated with LangGraph's StateGraph, with state defined as RAGState, containing messages, question, retrieved documents, answer, guardrail status, and retry count.

Three nodes form the main chain: retrieve → generate → guardrail. The key design is the conditional routing at the guardrail node: if it fails and the retry count is less than 2, route back to the generate node to regenerate with the failure reason attached; otherwise end.

async def guardrail_node(state: RAGState):
    passed, reason = guardrails.check_output(state["answer"].answer, context)
    if not passed and state["retry_count"] < 2:
        return {
            "guardrail_passed": False,
            "retry_count": state["retry_count"] + 1,
            "messages": [HumanMessage(content=f"Regenerate: {reason}")]
        }
    return {"guardrail_passed": passed}

This "generate → check → retry" loop is a core characteristic of Agentic RAG. Compared with one-shot generation, it returns control over quality to the system itself. The workflow uses SqliteSaver for checkpoint persistence, supporting interruption recovery.

FastAPI Service and Containerized Deployment

The service layer uses FastAPI to expose three endpoints: /query (RAG query), /ingest (background ingestion), and /health and /metrics (health and metrics). The query endpoint returns the answer, citations, confidence, recall count, and latency.

The following engineering practices are worth borrowing:

• Input guardrails intercept at the query entry point, so invalid requests never enter downstream flows
• Use BackgroundTasks to record metrics asynchronously without blocking responses
• Assign a unique thread_id to each request to support session isolation
• Docker images include HEALTHCHECK, and Kubernetes configures separate livenessProbe and readinessProbe
• Manage OpenAI keys with Secrets, and separate resource requests from limits

Kubernetes deployment defaults to 3 replicas, using Redis for caching, Qdrant for vector storage, and PostgreSQL for document storage—a standard combination that can scale horizontally.

Evaluation and Monitoring: From Precision@K to User Feedback

The evaluator RAGEvaluator has two dimensions:

Retrieval evaluation calculates Precision@K (K=1/3/5/10), Recall@K, and MRR (Mean Reciprocal Rank). MRR measures "at what position the first relevant document appears," which is especially important for RAG.

Generation evaluation uses an LLM judge (llm_judge) to score across four dimensions: accuracy, completeness, citation quality, and faithfulness.

The production monitor ProductionMonitor logs each query as JSONL, including latency (tracked separately for retrieval/generation), recall count, citation count, confidence, user feedback, and errors. The get_stats method uses Pandas to calculate average latency, p95 latency, average confidence, error rate, and more within a 24-hour window.

Launch Checklist: Checks Across Six Dimensions

The solution finally provides a complete launch checklist, divided by module:

Ingestion: Semantic chunking, metadata enrichment, MinHash near-duplicate detection, incremental upsert, dead-letter queue, progress tracking

Retrieval: Hybrid search, cross-encoder reranking, MMR diversity, query rewriting, metadata filtering (ACL/date/department), p95 latency <200ms

Generation: JSON structured output, input/output guardrails, confidence scoring, low-confidence fallback, p95 latency <2s

Evaluation: 100+ golden test set, retrieval metrics, generation metrics, CI/CD regression tests, A/B framework

Monitoring: Latency percentiles, error classification, confidence distribution, user feedback, automated alerts, per-query cost tracking

Operations: Blue-green deployment, rollback within 5 minutes, index rebuild process, embedding model versioning, data retention policies, monthly backup drills

Summary

The value of this solution is not in how clever any particular piece of code is, but in how it advances RAG from "it works" to "it can be operated." Hybrid retrieval determines the upper bound of effectiveness, guardrails and evaluation determine the lower bound of trustworthiness, and monitoring and deployment determine whether the system can survive under real traffic. If you are building enterprise-grade RAG, it is recommended to start with hybrid retrieval + guardrails + evaluation set, as these three have the highest return on investment.

One-line Summary

About 90% of production-grade RAG is an engineering problem, requiring full-chain engineering from ingestion and hybrid retrieval to guardrails, evaluation, and monitoring.

Paragraph Summary

Based on a complete production-grade RAG solution, this article points out that what really determines success is not retrieval itself, but ingestion automation, the upper bound of retrieval quality, and generation controllability. Starting from six dividing lines between prototype and production, it breaks down: semantic chunking, SHA-256 deduplication, and concurrency control in the ingestion pipeline; a four-stage retrieval funnel (parallel dense + sparse, RRF fusion, cross-encoder reranking, top-k filtering); structured generation constrained by Pydantic and three layers of guardrails; a generate-check-retry loop orchestrated by LangGraph; FastAPI services and Kubernetes deployment; and an evaluation system built from Precision@K, MRR, LLM judging, and production monitoring. Finally, it provides a launch checklist covering six dimensions—ingestion, retrieval, generation, evaluation, monitoring, and operations—and recommends that enterprises prioritize hybrid retrieval, guardrails, and evaluation sets.