Complete Guide to Agentic RAG: Principles, Architecture, and Hands-On Implementation

The limitations of traditional RAG systems are increasingly apparent to developers: irrelevant retrieval results, inability to handle incomplete information, and lack of autonomous decision-making. In 2025, Agentic RAG—a fundamental upgrade to RAG technology—is becoming the core paradigm for LLM application development. This article starts from traditional RAG, dives deep into the principles of Agentic RAG, and provides a complete code implementation.

Traditional RAG: Workflow and Limitations

Offline Pipeline: Document Processing and Vectorization

Traditional RAG can be broken down into two pipelines. The first is the offline pipeline, responsible for converting raw documents into retrievable vector data.

The specific steps are as follows: first, load the raw documents (PDF, Word, TXT, etc.), then chunk the documents. Since a complete document may contain tens of thousands of words and cannot be fed into an LLM all at once, it needs to be split into fixed-length segments (e.g., 256 characters), with overlapping regions between segments to preserve semantic continuity.

After chunking, an Embedding model converts each segment into a fixed-dimensional vector representation, and these vectors are stored in a vector database (such as ChromaDB).

About Embedding Models and Vectorization: An Embedding model is a neural network that maps text into a high-dimensional vector space. Its core idea stems from the distributional semantics hypothesis—semantically similar texts are closer together in vector space. Common Embedding models include OpenAI's text-embedding-3-small (1536 dimensions), the BGE series (optimized for Chinese), and Sentence-BERT. Vectorized text can be semantically matched using cosine similarity or Euclidean distance, which captures synonyms, near-synonyms, and semantically equivalent expressions far better than traditional keyword matching. The choice of vector dimensions directly affects the trade-off between retrieval accuracy and storage cost.

About ChromaDB: ChromaDB is an open-source, lightweight vector database designed specifically for AI applications. It supports both in-memory and persistent storage modes, includes multiple built-in distance metrics (cosine similarity, L2 distance, inner product), and offers metadata filtering. Compared to production-grade vector databases like Pinecone, Milvus, and Weaviate, ChromaDB is better suited for prototyping and small-to-medium scale applications. In production environments, developers typically need to consider vector index algorithm selection (e.g., HNSW, IVF), sharding strategies, and hybrid query capabilities with traditional databases.

Online Pipeline: Retrieval, Assembly, and Generation

The second is the online pipeline, which handles real-time user queries. When a user asks a question, the system first performs Query Rewriting to make it more suitable for retrieval.

The rewritten query goes through two retrieval paths: one performs keyword retrieval via BM25, while the other converts the query into a vector using an Embedding model for semantic similarity matching. The results from both paths are merged and then reranked to select the most relevant document segments, which are finally injected into the Context section of a Prompt template and sent to the LLM for answer generation.

About BM25 and Hybrid Retrieval: BM25 (Best Matching 25) is a classic probabilistic ranking algorithm in information retrieval. It calculates relevance scores between queries and documents based on term frequency (TF), inverse document frequency (IDF), and document length normalization. It excels at exact keyword matching but cannot understand semantic equivalence. Hybrid Search combines BM25 keyword retrieval with vector semantic retrieval, merging results through weighted fusion or RRF (Reciprocal Rank Fusion) algorithms to leverage both exact matching and semantic understanding. This strategy has been proven more effective than single retrieval methods in production environments.

Core Pain Points of Traditional RAG

The entire process is unidirectional, fixed, and one-shot. The model can only generate answers based on the first round of retrieval results and cannot:

• Automatically retry with a different approach when retrieval fails
• Proactively determine whether additional context is needed
• Answer metadata questions like "What documents are in the knowledge base?"
• Dynamically adjust retrieval strategies to obtain more complete information

When no effective content is retrieved, the model can only reply "I don't know" instead of attempting to rewrite the query or call other tools to find the answer. This is exactly the core problem that Agentic RAG aims to solve.

Agentic RAG: From Pipeline to Intelligent Closed Loop

Core Concept: Turning Retrieval into Tools

Agentic RAG is a fundamental upgrade to traditional RAG. Its core idea is: encapsulate every component of RAG (query rewriting, vector retrieval, keyword search, etc.) as callable tools, and grant the LLM autonomous decision-making authority.

Instead of following a fixed pipeline, the model enters a "Think → Act → Observe → Think Again" loop. Before generating the final answer, the model can perform multi-step tool calls and dynamically adjust strategies until it has gathered sufficient information.

About the ReAct Paradigm: ReAct (Reasoning + Acting) is an Agent reasoning paradigm proposed in 2022 by Google Research and Princeton University. It interleaves the LLM's reasoning capability (Chain-of-Thought) with action capability (tool calling), forming a Thought → Action → Observation loop. Compared to pure reasoning (CoT), ReAct can correct the reasoning process by obtaining real-time information through external tools; compared to pure action, it plans the next step through explicit reasoning. LangGraph's create_react_agent is built on this paradigm, allowing the model to think at each step and decide whether to call a tool or output the final answer.

Three Core Capabilities of Agentic RAG

Implementing Agentic RAG requires the model to possess three types of capabilities:

1. Planning: Manifested in the Chain-of-Thought reasoning process, where the model plans execution steps and evaluates intermediate results
2. Tool Calling: The model selects and invokes appropriate tools as needed, potentially coordinating across multiple agents
3. Multi-step Iteration: Before delivering the final answer, the model can perform multiple rounds of tool calls and result evaluation

About the Function Calling Mechanism: Function Calling is a structured tool invocation capability provided by LLM vendors (such as OpenAI, Anthropic, Google, etc.). Developers define tool names, descriptions, and parameter structures in JSON Schema format. During reasoning, the model determines whether a tool call is needed and outputs a structured invocation request (containing the tool name and parameter values). After the system executes the tool, the result is returned to the model, which continues reasoning based on the output. The key to this mechanism is that models undergo specialized Instruction Tuning, enabling them to understand tool descriptions and generate properly formatted invocation instructions rather than simple text completion.

The Essential Difference Between Agentic RAG and Traditional RAG

In traditional RAG, the LLM is only used at the final generation stage. In Agentic RAG, the LLM begins participating in decision-making from the moment the user inputs a question—determining whether retrieval is needed, choosing which retrieval method to use, evaluating whether retrieval results are sufficient, and deciding whether supplementary information is required.

Here's a concrete example: when traditional RAG faces a query, it performs a single vector retrieval and finds a low hit rate, then has no choice but to pass the low-quality results to the model. Agentic RAG, on the other hand, observes the low hit rate, automatically rewrites the query for a second round of search, retrieves more relevant segments, performs a third supplementary search, and finally generates an answer based on the consolidated, high-quality context.

ChatPDF Boss: An Open-Source Implementation Analysis

Architecture Design: Trading Time for Intelligence

The open-source project ChatPDF Boss provides a highly instructive implementation of Agentic RAG. The system's core design philosophy is to trade time for model intelligence.

When a user sends a question, the system first checks whether the model supports Function Calling:

If Function Calling is not supported: A Prompt-based approach determines whether the question requires retrieval. If no retrieval is needed, it replies directly; if retrieval is needed, it performs semantic search, injects relevant segments into the Context, and generates an answer. This approach is superior to instructing the model via Prompt to ignore irrelevant context, because it uses two separate models for decision-making.

If Function Calling is supported: All tools are registered with the model, which autonomously decides which tools to call. This is the true Agentic mode.

Design of Four Core Tools

ChatPDF Boss features four core tools covering the main scenarios for knowledge base interaction:

Tool Name	Description
Search Query	Basic semantic retrieval tool that performs vector similarity search
List Files	Lists files in the knowledge base, addressing traditional RAG's inability to answer metadata questions
Read File	Precisely reads specific segments by document ID, supports proactively reading adjacent segments to supplement context
Gather File Meta	Retrieves file metadata information

List Files and Read File are the most distinctive designs. Traditional RAG cannot answer questions like "What documents are in the knowledge base?" because it can only retrieve content segments. The List Files tool enables the model to access file listings and count information. Read File allows the model to proactively read surrounding segments to supplement context when information is incomplete, rather than relying entirely on semantic similarity retrieval.

Code Implementation: From Traditional RAG to Agentic RAG

Traditional RAG with LangChain

Using the LangChain framework, traditional RAG implementation consists of offline and online components:

Offline component: Load documents → Text splitting (set chunk_size and overlap) → Vectorize with Embedding model → Store in ChromaDB. The core code uses LangChain's text splitter and ChromaDB's from_documents method.

Online component: Load vector database → Construct user query → Call similarity_search to retrieve Top-K relevant segments → Concatenate segments into Prompt template → Send to LLM for answer generation.

In practice, the offline pipeline is where the real work lies. How to chunk documents, which Embedding model to choose, whether fine-tuning is needed, and how to design caching strategies—all of these directly impact the quality of retrieval results.

Agentic RAG with LangGraph

The core implementation leverages LangGraph's create_react_agent, resulting in a surprisingly concise code structure:

1. Define the toolset: Wrap Search Query, List Files, Read File, etc. as standard tool functions
2. Write the System Prompt: Guide the model on how to use these tools; be sure to constrain the response format to avoid tool call failures
3. Create the React Agent: Pass in the LLM instance, tool list, and system prompt
4. Run the Agent: Call the invoke method with the user question

About the LangGraph Framework: LangGraph is an Agent orchestration framework developed by the LangChain team, based on the concept of directed graphs to organize Agent execution flows. Unlike LangChain's chain-based invocation, LangGraph supports loops, conditional branching, and state persistence, making it better suited for building complex Agents that require multi-round iteration. Its core concepts include: State (global state), Node (execution nodes such as LLM calls or tool execution), and Edge (transition conditions between nodes). create_react_agent is a high-level abstraction provided by LangGraph that internally implements the ReAct loop's state transition logic—developers only need to define tools and prompts to get full Agent capabilities.

# Agentic RAG core code structure (simplified)
agent = create_react_agent(
    llm=model,           # LLM instance
    tools=tool_list,     # Tool list
    system_prompt=prompt  # System prompt
)
result = agent.invoke({"messages": user_query})

The code looks simple, but it unleashes the model's autonomous decision-making capability, enabling dynamic, flexible retrieval-generation interaction. The core logic of many LLM application products is essentially this kind of implementation.

Summary and Reflections

Traditional RAG is a fixed pipeline: chunk → vectorize → retrieve → assemble → generate. It's simple and straightforward but lacks flexibility, unable to handle retrieval failures or incomplete information scenarios.

Agentic RAG encapsulates retrieval, file reading, and other capabilities as tools, granting the LLM autonomous decision-making authority. The model can dynamically plan, call tools, observe results, and iterate through multiple rounds until it has gathered sufficient information to generate the final answer. This truly embodies intelligent agent behavior: Think → Act → Observe → Think Again.

For developers, the key to migrating from traditional RAG to Agentic RAG comes down to two things: first, decompose the retrieval pipeline into independent tool functions; second, choose an LLM that supports Function Calling and use frameworks like LangGraph to build a React Agent.

In one sentence: Tools provide capability; intelligence lies in the choice. True Agentic RAG begins with retrieval and succeeds through decision-making.