Agentic RAG in Practice: Architecture Deep Dive and LangChain Implementation

The limitations of traditional RAG systems are becoming increasingly apparent to developers: retrieve once, generate once, end of pipeline—completely helpless when retrieval fails or information is incomplete. Agentic RAG transforms RAG from a rigid linear pipeline into an intelligent agent loop with planning, tool invocation, reflection, and iteration capabilities.

This article starts from the fundamentals of traditional RAG, dives deep into the core mechanisms of Agentic RAG, and demonstrates a complete implementation using LangChain + LangGraph.

Traditional RAG: Implementation and Limitations

Offline Pipeline: From Document Chunking to Vector Storage

Traditional RAG implementation can be clearly divided into two pipelines: offline and online.

The offline pipeline's core task is transforming raw documents into searchable vector data. The specific steps are: first load documents (PDF, Word, TXT, etc.), then chunk them—since complete documents may contain tens of thousands of characters and can't be fed into an LLM all at once. Chunking produces fixed-length segments (e.g., 256 characters), with overlap between segments to maintain contextual coherence.

It's worth noting that text chunking, while seemingly simple, is actually one of the most critical factors affecting RAG system performance. Beyond fixed-length chunking, the industry has developed several advanced strategies: Semantic Chunking splits text at semantic boundaries to ensure each segment is semantically complete; Recursive Chunking progressively subdivides by paragraph, sentence, and character hierarchies; and structure-based chunking leverages headings, sections, and other structural information for intelligent splitting. The choice of chunk_size requires balancing retrieval precision against context completeness—segments that are too small may lose critical context, while segments that are too large introduce noise and consume more tokens. The overlap setting mitigates information fragmentation at chunk boundaries and is typically set to 10%-20% of chunk_size.

Next, an Embedding model converts each segment into a fixed-dimensional vector, which is then stored in a vector database. Embedding models (such as OpenAI's text-embedding-ada-002, the open-source BGE series, etc.) work by mapping text into a high-dimensional vector space where semantically similar texts are closer together. This process is based on Transformer-architecture pretrained language models that use training paradigms like Contrastive Learning to capture deep semantic features. Vector dimensions typically range from 768 to 1536—higher dimensions theoretically offer greater expressiveness but increase computational cost. Vector databases (such as ChromaDB, Milvus, Pinecone, etc.) are specifically optimized for Approximate Nearest Neighbor (ANN) search on high-dimensional vectors, using index algorithms like HNSW and IVF to achieve millisecond-level retrieval across millions or even billions of vectors.

Online Pipeline: Retrieval, Assembly, and Generation

The online pipeline is the actual user interaction process. When a user asks a question, the system first rewrites the query (since the user's original phrasing may not be optimal for retrieval), then performs retrieval in two ways: BM25 keyword search and semantic similarity matching using the vectorized query.

BM25 (Best Matching 25) is a classic information retrieval algorithm based on Term Frequency (TF) and Inverse Document Frequency (IDF) for keyword matching—a sparse retrieval method. Its strength lies in high sensitivity to exact keyword matches, such as proper nouns and model codes. Semantic retrieval (Dense Retrieval), on the other hand, excels at understanding synonyms, paraphrases, and contextual meaning. Hybrid Search combining both has become standard in production-grade RAG systems.

The combined results from both retrieval rounds undergo Reranking, which typically uses a Cross-Encoder that jointly encodes the query-document pair. Compared to bi-encoder models, this provides more precise relevance judgments but at higher computational cost—hence it's only used for re-ranking the initial candidate set. Finally, the Top-K document segments are injected into a Prompt template and passed to the LLM for answer generation.

This pipeline is straightforward, but has a fundamental flaw: the entire process is unidirectional, fixed, and one-shot. If the first retrieval round misses relevant content, the system won't try a different approach. If contextual information is incomplete, the model can't proactively seek supplementary information. When you ask "what documents are in the knowledge base," traditional RAG simply crashes—because it can only retrieve segments, not reason.

Core Philosophy of Agentic RAG

From Fixed Pipelines to Intelligent Decision Loops

Agentic RAG is a fundamental upgrade to traditional RAG. Its core idea is: encapsulate every component of RAG (query rewriting, vector retrieval, keyword search, file reading, etc.) as callable Tools, and grant the LLM autonomous decision-making capability.

The model no longer passively executes a fixed pipeline but enters a "Think → Act → Observe → Think Again" loop. This loop originates from the ReAct (Reasoning + Acting) paradigm jointly proposed by Google Research and Princeton University in 2022. ReAct's core insight is that pure Chain-of-Thought reasoning is prone to hallucination and error accumulation, while pure action execution lacks reasoning guidance. ReAct interleaves both—the model first reasons in natural language (Thought), then executes a concrete action based on that reasoning (Action), and observes the action's result (Observation), forming a closed loop. This paradigm enables LLMs to continuously adjust their strategy when solving complex problems, rather than going down a single path to the end.

Specifically: after a user asks a question, it goes directly to the LLM, which decides whether to call a tool and which one. After the tool returns results, the model observes the result quality and decides whether to call additional tools for supplementary information or generate a final answer based on what it already has.

Three Core Capabilities of Agentic RAG

Agentic RAG implementation relies on three core model capabilities:

1. Planning: Manifested in Chain-of-Thought reasoning, where the model plans the steps to solve a problem
2. Tool Calling: The model can identify and invoke appropriate tools, and even orchestrate multi-Agent collaboration
3. Multi-step Iteration: The model can perform multiple rounds of tool calls before generating the final answer, progressively refining information

ChatPDF Boss: An Agentic RAG Implementation Analysis

Architecture Design: Trading Time for Intelligence

Using the open-source project ChatPDF Boss as an example, its Agentic RAG implementation logic is highly instructive. The system first checks whether the model supports Function Calling:

Without Function Calling support: A prompt determines whether the user's question requires retrieval. If not, it responds directly; if so, it performs semantic search then generates an answer. This "judge first, then retrieve" approach yields better results than instructing the model to ignore irrelevant context in the prompt.

With Function Calling support: All tools are registered with the model, letting it autonomously decide which tools to invoke. This is true Agentic RAG—from the moment the user inputs a question, the LLM participates in decision-making, rather than only intervening at the final generation stage.

Function Calling was first introduced by OpenAI in June 2023 and has since been widely adopted by major model providers. Technically, it works by including extensive function definitions and invocation examples in the model's training data, teaching the model to output structured JSON function call requests at appropriate times instead of natural language text. The model doesn't actually execute functions—it generates structured output containing function names and parameters, while external systems handle actual execution and return results to the model. This mechanism is the foundation of Agentic systems—without reliable Function Calling capability, models cannot accurately interact with external tools. Models currently supporting Function Calling include the GPT-4 series, Claude 3 series, Qwen series, GLM-4, and others, though significant differences remain in call accuracy and complex parameter handling.

Four Core Tool Designs

ChatPDF Boss features four core tools, each solving a pain point of traditional RAG:

1. Search Query (Basic Retrieval): The most fundamental semantic retrieval tool, corresponding to traditional RAG's retrieval function
2. List Files: Lists all files in the knowledge base. Traditional RAG cannot answer questions like "what documents are in the knowledge base" because it can only retrieve segments, not global information. With this tool, the model can access file counts, names, and other metadata
3. Read File (Precise Reading): Reads specific segments by document ID. When the model detects incomplete information, it can proactively read adjacent segments for additional context, no longer relying solely on semantic similarity retrieval
4. Gather File Meta (Metadata Reading): Retrieves file metadata to provide richer context

Traditional RAG vs. Agentic RAG: A Practical Comparison

Here's a concrete example illustrating the difference:

Traditional RAG: User query → vector retrieval → retrieval results → generate answer. If the first retrieval has low hit rate, it can only produce a forced answer based on low-quality context.

Agentic RAG: First search round observes low hit rate → rewrites query → second search round hits relevant results → third search round retrieves supplementary documents → generates high-quality answer based on consolidated complete information. The entire process is multi-step iterative, with "observe-think-act" decisions at every step.

Agentic RAG Code Implementation

Traditional RAG with LangChain

Using the LangChain framework, traditional RAG implementation has two parts:

Offline part: Load documents → text splitting (setting chunk_size and overlap) → vectorize with Embedding model → store in ChromaDB vector database. The core code simply calls LangChain's text splitter and ChromaDB's storage interface.

Online part: Load vector database → receive user query → call similarity_search to retrieve Top-K segments → assemble into Prompt template → LLM generates answer. In practice, if retrieved segments aren't sufficiently relevant to the question, the model can only answer "I don't know."

Agentic RAG with LangGraph

The core implementation uses LangGraph's create_react_agent, an Agent framework with built-in "observe and think" patterns.

LangGraph is an Agent orchestration framework released by the LangChain team in 2024, with a core design philosophy of modeling Agent execution flow as a Directed Graph. Unlike LangChain's earlier chain-based invocations (Chain), LangGraph supports loops, conditional branches, and state persistence, making it possible to build Agents with complex decision logic. In LangGraph, each Node represents a processing step (e.g., calling an LLM, executing a tool), and Edges define transition conditions between nodes. create_react_agent is a high-level abstraction provided by LangGraph that internally implements a standard ReAct loop graph: the LLM node decides whether to call a tool; if so, it transitions to the tool execution node; tool results flow back to the LLM node for the next judgment round, until the LLM decides to directly output the final answer. The advantages of this graph structure include strong observability, ease of debugging, and support for checkpoint recovery and Human-in-the-Loop interaction.

Implementation steps:

1. Define the tool set: Encapsulate Search Query, List Files, Read File, and Gather File Meta as Tools
2. Write the System Prompt: Guide the model on how to use these tools, with format constraints to prevent tool call failures
3. Create the React Agent: Pass in the LLM instance, tool list, and system prompt
4. Run the Agent: Call the invoke method with the user question; the Agent automatically enters the think-act-observe loop

# Core code structure (simplified)
agent = create_react_agent(
    llm=model,           # LLM instance
    tools=tool_list,     # Tool list
    system_prompt=prompt  # System prompt
)
result = agent.invoke({"input": user_query})

The code appears simple, but it grants the model autonomous decision-making and dynamic adjustment capabilities. This is also why many "wrapper" applications are essentially built on similar core logic.

Summary and Reflections

Traditional RAG is a linear "retrieve and answer" pipeline—simple and direct but inflexible. Agentic RAG is a closed-loop system where "tools provide capability, intelligence lies in choice." Its core value includes:

• Toolifying retrieval capabilities: No longer a fixed retrieval pipeline, but capabilities the model can invoke on demand
• Granting the model decision authority: The model can plan, invoke, observe, and iterate until it has sufficient information
• Trading time for quality: Improving answer accuracy and completeness through multi-round interactions

True Agentic RAG begins with retrieval and succeeds through decision-making. For LLM engineers, understanding and mastering this paradigm has become an essential skill in 2025.