LangChain in Practice: Building the ChatDoc Document Chat Application with RAG

Why LLMs Need RAG

When using large language models like ChatGPT, we often run into a thorny problem: the model handles general knowledge with ease, but when it comes to internal corporate documents, specialized domain materials, or the latest data, it tends to "confidently spout nonsense." This is precisely the core pain point of current LLMs—the inherent limitations in knowledge timeliness and domain expertise.

This kind of "hallucination" is not an occasional program bug, but a structural byproduct of how LLMs are trained. A large language model is essentially a probabilistic prediction system—it learns "what the next word is most likely to be" from massive amounts of text, rather than genuinely understanding whether facts are true or false. When the model encounters a question that wasn't adequately covered in its training data, it still generates "plausible-sounding" output based on probability, without being able to perceive that it's fabricating content. In addition, models have a strict Knowledge Cutoff—for example, GPT-4's training data ends in early 2023, meaning everything that happened afterward is a blind spot for it.

RAG (Retrieval-Augmented Generation) is the technical solution born to address this problem. It decouples "what the model knows" from "what can be looked up externally." Instead of relying on knowledge memorized during training, it first retrieves relevant content from a reliable knowledge base, then has the model generate answers based on the retrieval results—an "evidence-based" answer is naturally more accurate and trustworthy.

RAG's three core advantages are clear at a glance:

• Reduced hallucination probability: Answers come from real documents, not fabricated out of thin air by the model
• Breaking through knowledge boundaries: Easily handle vertical-domain questions beyond the training data
• No model retraining required: No dependence on fine-tuning, low deployment cost, and short implementation cycle

For these reasons, RAG has become the most mainstream and cost-effective technical path for enterprises deploying LLM applications.

Three Core Topics in This Chapter

This chapter centers on the principles and practice of RAG, unfolding across three progressively deeper modules.

Part One: Understanding the Principles and Value of RAG

This part focuses on the theoretical foundations: what RAG is, what problems it solves, and why the industry proposed this solution. Only by truly understanding the working mechanism of retrieval-augmented generation can you make sound architectural decisions in real-world development. The theoretical content is relatively dense, but it's the bedrock of the entire knowledge system, so be sure to master it thoroughly.

Part Two: Loading and Processing Documents with LangChain

The second part moves into hands-on practice, focusing on how to use LangChain to "feed" documents of various formats to large models. LangChain is currently the most mainstream LLM application development framework. Its core design philosophy is "chained composition"—encapsulating document loading, text processing, vectorization, retrieval, prompt construction, model invocation, and other stages into standardized, composable modules (Chain/LCEL).

In real-world scenarios, document formats are wildly diverse—PDF, Word, Markdown, web pages, plain text… LangChain provides a unified Document Loader that supports dozens of data sources including Notion, Confluence, and CSV, helping us elegantly handle these heterogeneous data sources. This highly modular design allows developers to build a fully functional RAG application prototype in just a few dozen lines of code, and to flexibly swap out individual components in production (such as switching to a different vector database or Embedding model) without rewriting the core logic.

Mastering document loading means you can let large models "learn" private materials from any source—this is the crucial first step in building vertical-domain AI applications. This part will guide you through hands-on practice with plenty of code demonstrations.

Part Three: Building the ChatDoc Document Chat Application

The third part is the core deliverable of this chapter: comprehensively applying techniques such as document vectorization, vector databases, and semantic retrieval to build a complete document Q&A application from scratch—ChatDoc. Users can ask questions directly about their own documents, and the model provides precise answers based on document content, completely doing away with the inefficiency of traditional information retrieval.

A Detailed Look at the RAG Pipeline: From Document to Answer

To understand how ChatDoc is implemented, you first need to fully grasp RAG's complete technical pipeline. The entire process is divided into two stages: offline indexing and online retrieval.

Offline Stage: Document Vectorization and Ingestion

The offline workflow is as follows:

1. Document loading: Read raw documents with LangChain
2. Text chunking: Split long documents into semantically complete text chunks (Chunk)
3. Embedding vectorization: Use an Embedding model to convert each text chunk into a high-dimensional vector, where semantically similar content is closer together in vector space
4. Writing to the vector database: Build a knowledge index for fast retrieval

Text chunking strategy directly determines the retrieval quality of a RAG system and deserves special attention. If chunks are too large, a single chunk contains too much irrelevant information, diluting retrieval precision; if chunks are too small, a single chunk is semantically incomplete, and the generated answers lack context. Common strategies include fixed-character-count chunking, chunking by paragraph/sentence, and the recursive character splitting that LangChain adopts by default (balancing length and semantic boundaries). In addition, the "sliding window" strategy retains a certain overlap between adjacent chunks, effectively preventing key information from being cut off at chunk boundaries. In real-world engineering, chunk size (typically 200-1000 tokens) and overlap ratio (typically 10%-20%) are key hyperparameters that need to be tuned based on document type and business scenario.

Embedding vectorization is the technical core of this stage. Embedding maps natural language text to a point in a high-dimensional numerical space (usually 768 or 1536 dimensions), performed by specially trained Embedding models such as OpenAI's text-embedding-ada-002 or open-source models like the BGE and E5 series. Its key property is that semantically similar sentences have higher cosine similarity in vector space. For example, the vector distance between "Apple phone" and "iPhone" would be much smaller than the distance between "Apple phone" and "Beijing weather."

Vector databases are database systems optimized specifically for storing and performing similarity retrieval on high-dimensional vectors—the core infrastructure of a RAG architecture. They need to efficiently execute ANN (Approximate Nearest Neighbor) search algorithms. Current mainstream options include: Pinecone (fully managed cloud service, ready out of the box), Weaviate and Qdrant (open source with support for hybrid retrieval), Chroma (lightweight, ideal for local development prototypes), and Milvus (high performance, suitable for billion-scale production deployments). LangChain provides a unified abstraction interface for all of the above databases, so developers can write and retrieve vectors without needing to dive into the underlying implementation details.

Online Stage: Semantic Retrieval and Answer Generation

After a user asks a question, the system's processing flow is concise and efficient:

1. Question vectorization: Convert the user's question into a vector
2. Semantic retrieval: Find the most semantically relevant text chunks in the vector database (based on mathematical measures such as cosine similarity, rather than simple keyword matching)
3. Building the prompt: Assemble the retrieved content and the question into a structured prompt
4. LLM generation: The model outputs precise answers based on real reference materials

This "retrieve + generate" combination is the essence of RAG and the driving engine behind ChatDoc.

Learning Advice: Ground Yourself in Theory, Implement with Code

This chapter is organized with a "theory first, then practice" logic: the first part requires you to settle down and understand RAG's design motivation and operating principles, while the latter two parts focus on code demonstrations and emphasize hands-on ability.

For engineers aspiring to master AI Agent application development, RAG is an unavoidable core skill. It's not only the foundational paradigm for document Q&A, but also the general-purpose architecture for nearly all enterprise-grade LLM applications.

While learning, aim to do two things: first, follow along with the code line by line and practice hands-on; second, think as you learn about how to transfer this technology to your own real-world business. When you can independently build a working ChatDoc application, it means you've truly crossed an important threshold in LLM application development.

Key Takeaways