Agentic RAG Practical Guide: The Complete Evolution from Traditional RAG to Agent-Based Retrieval

Why Traditional RAG Falls Short

Have you ever encountered situations like these: you spent considerable time building a RAG system, only to have the model give irrelevant answers? Retrieved a bunch of seemingly related but utterly useless content? When users ask "what documents do you have," the system crashes — because it can only search, not think? When retrieval fails, it simply gives up rather than trying a different approach?

The root cause of these problems is: Traditional RAG is a fixed pipeline that lacks flexibility and autonomous decision-making capability. 2025 has been called the Year of Agents, and Agentic RAG is becoming an essential skill for LLM engineers. This article starts from traditional RAG, progressively reveals how it evolves into Agentic RAG, and walks you through its core implementation with hands-on code.

How Traditional RAG Works

Traditional RAG (Retrieval-Augmented Generation) can be divided into two pipelines: the offline pipeline and the online pipeline. Understanding these two pipelines is the prerequisite for grasping the upgrade logic of Agentic RAG.

RAG (Retrieval-Augmented Generation) was first proposed by Meta AI in 2020. Its core motivation was to solve the knowledge cutoff problem and hallucination problem of large language models. The parametric knowledge of LLMs is static and cannot be automatically updated after training. RAG addresses this by dynamically retrieving from external knowledge bases during inference, injecting the most recent and relevant information into the generation process, making model responses more accurate and reliable. This paradigm quickly became the standard architecture for enterprise AI applications — virtually every scenario that requires answering questions based on private data uses RAG.

Offline Pipeline: Document Chunking → Vectorization → Storage

The offline pipeline is user-independent and belongs to the data preprocessing stage. The specific workflow is as follows:

1. Document Loading: Load PDFs, Word documents, TXT files, etc. into memory
2. Text Chunking: Since documents can contain tens of thousands of characters and cannot be fed to the LLM all at once, they need to be split into fixed-length paragraphs (e.g., 256 characters), with optional overlap between chunks
3. Vectorization: Use an Embedding model to convert each chunk into a fixed-dimensional vector representation
4. Storage: Store vectors in a vector database (e.g., Chroma, Milvus, etc.)

Embedding models (such as OpenAI's text-embedding-ada-002, the BGE series, or Qwen Embedding) map text into a high-dimensional vector space where semantically similar texts are closer together. Vector retrieval typically uses cosine similarity or inner product as the distance metric, achieving millisecond-level retrieval through ANN (Approximate Nearest Neighbor) algorithms. Regarding vector database selection, Chroma is suitable for lightweight prototyping and local experiments, while Milvus and Pinecone target production-grade large-scale scenarios, supporting distributed deployment and efficient retrieval over billions of vectors.

Online Pipeline: Retrieval → Prompt Assembly → Generation

When a user asks a question, the online pipeline kicks in:

1. Query Rewriting: The user's original question may not be suitable for direct retrieval and needs to be rewritten to improve recall
2. Dual-Path Retrieval: First use BM25 for keyword retrieval, then use the Embedding model for semantic vector retrieval
3. Merging and Reranking: Merge results from both paths and perform reranking (Rerank) to select the most relevant chunks
4. Prompt Assembly: Inject retrieved document chunks into the Context section of the prompt template
5. LLM Generation: The model generates the final answer based on the prompt

Regarding the design logic of dual-path retrieval: BM25 is a classic information retrieval algorithm based on improvements to TF-IDF (Term Frequency-Inverse Document Frequency), excelling at exact matching of keywords and proper nouns. Semantic vector retrieval excels at understanding synonyms and semantic associations. Hybrid Search combining both can significantly improve recall — for example, when a user searches for "heart disease," BM25 can exactly match that term, while semantic retrieval can also recall chunks containing "coronary heart disease" or "myocardial infarction." Rerank models (such as Cohere Reranker or BGE-Reranker) perform fine-grained sorting after merging results, ensuring the final returned chunks are highly relevant to the user's question.

The prompt template for traditional RAG is very simple. The core is: "You are a professional assistant. Please answer based on the following question and retrieved documents." Plus the Context (Top-K retrieved document chunks) and the user's question.

The core problem with traditional RAG is: the entire process is unidirectional, fixed, and one-shot. If the first retrieval doesn't find what's needed, the model cannot re-retrieve or try a different approach, much less call other tools to obtain supplementary information.

Agentic RAG: A Fundamental Upgrade from Pipeline to Agent

The Core Idea of Agentic RAG

Agentic RAG is a fundamental upgrade to the traditional RAG workflow. Its core idea is: Encapsulate every component of RAG (query rewriting, vector retrieval, keyword search, etc.) as callable tools, and grant the LLM autonomous decision-making, multi-turn invocation, and dynamic adjustment capabilities.

The Tool Calling/Function Calling capability was a key technical breakthrough introduced by OpenAI in 2023, subsequently adopted by other model providers. The principle is that during model inference, in addition to generating natural language text, the model can output structured function call instructions (JSON containing function names and parameters). Through specialized training, models learn when to call tools, which tool to call, and how to construct parameters. This capability is the technical foundation enabling Agentic RAG's autonomous decision-making — without Function Calling, models cannot interact with external systems in a structured way.

In other words, Agentic RAG is no longer a straight line from start to finish, but an intelligent agent behavioral loop that can iterate: Think → Call Tool → Observe Results → Think Again → Act Again, until a satisfactory final answer is generated.

This behavioral pattern is essentially an implementation of the ReAct (Reasoning + Acting) framework. ReAct was proposed by Yao et al. in 2022, alternating Chain-of-Thought reasoning with external tool interactions, allowing the model to adjust its strategy based on real feedback at each step. Compared to pure reasoning (prone to hallucinations) or pure acting (lacking planning), ReAct performs more reliably on complex tasks because each tool call result provides a real "anchor" for the next reasoning step, effectively reducing error accumulation.

Traditional RAG vs Agentic RAG Workflow

Traditional RAG workflow: User question → Retrieval → Prompt assembly → Answer generation (one-shot, no going back)

Agentic RAG workflow:

1. User asks a question
2. The question is passed directly to the LLM (GPT, Claude, DeepSeek, etc.)
3. The model autonomously decides whether to call tools
4. After calling a tool and receiving results, the model decides the next step — call another tool or answer directly
5. Iterative loop until sufficient information is gathered to generate the final answer

Three Core Capabilities of Agentic RAG

The implementation of Agentic RAG relies on the model possessing three core capabilities:

• Planning: Manifested in Chain-of-Thought reasoning, the model can plan execution steps and evaluate the quality of intermediate results
• Tool Calling: The model can invoke various external tools and collaborate with multiple agents to complete complex tasks
• Multi-Step Iteration: The model can make multiple tool calls before generating the final answer, progressively completing the information puzzle

ChatPDF's Agentic RAG Implementation Analysis

Overall Architecture Design

The open-source project ChatPDF (ChatBoss) offers a highly instructive implementation approach. Its core design philosophy is: Trading time for model intelligence.

When a user question arrives, the system first determines whether the model supports tool calling:

When tool calling is not supported: Use a prompt to determine whether the question requires retrieval. If retrieval isn't needed, respond directly; if it is, perform semantic search and inject the Context before generating an answer. This approach is superior to telling the model to ignore irrelevant Context in the prompt, because it uses two independent models for decision-making.

When tool calling is supported: Register the entire tool collection with the model, letting it autonomously decide which tools to call — semantic retrieval tool, file listing tool, file reading tool, etc. All decision-making authority is delegated to the model rather than following a fixed process.

Four Core Tools Explained

ChatBoss designed four core tools, each solving a pain point of traditional RAG:

1. Search Query (Semantic Retrieval Tool): The most basic semantic retrieval function, searching for relevant chunks in the vector database based on the user's query
2. List Files (File Listing Tool): Lists the files in the knowledge base. Traditional RAG cannot answer questions like "what documents are in the knowledge base" because it can only retrieve chunk contents. With this tool, the model can obtain file counts and complete listings
3. Read File (Precise Reading Tool): Reads specific chunks precisely by document ID. When information is found to be incomplete, the model can proactively read surrounding chunks to supplement context, independent of semantic similarity
4. Gather File Meta (Metadata Reading Tool): Retrieves file metadata such as filename, creation time, size, etc.

Practical Comparison: Different Behaviors on the Same Question

Here's a concrete example illustrating the difference between traditional RAG and Agentic RAG:

Traditional RAG approach: User query → Vector retrieval → Retrieval results → Generate answer. If the first retrieval has low hit rate, it simply returns "no relevant information found," resulting in poor user experience.

Agentic RAG approach:

• First search: Retrieves using the original query, finds low hit rate
• Observation and rewriting: The model observes unsatisfactory results and proactively rewrites the query keywords
• Second search: Retrieves again using the rewritten query, successfully hitting relevant chunks
• Third round supplementation: Further calls the Read File tool to get context from related documents
• Final generation: Generates a high-quality answer based on the consolidated complete information

This is the core advantage of Agentic RAG — it doesn't give up when facing problems, but thinks, adjusts, and retries like a human would.

Hands-On Code: Building Traditional RAG and Agentic RAG with LangChain

Traditional RAG Code Implementation

The entire implementation is based on the LangChain framework, with the following core steps:

Offline part: Use TextLoader to load documents, split text with RecursiveCharacterTextSplitter (setting chunk_size and chunk_overlap), vectorize with an Embedding model (e.g., Qwen Embedding 0.6B), and store in a Chroma vector database. In practice, a TXT file was split into 37 text chunks and all stored in the database.

Online part: Load the Embedding model and Chroma database, construct the user query, call the similarity_search method to retrieve the K most relevant chunks, concatenate the chunks' page_content into the prompt template, and pass it to the LLM for answer generation.

In actual execution, when asking about a character in a novel, since none of the three retrieved chunks contained relevant content, the model could only answer "I don't know." This precisely demonstrates the core bottleneck of traditional RAG lies in Context quality — if useful information isn't retrieved, even the strongest model is powerless.

Agentic RAG Code Implementation

The core implementation uses create_react_agent from LangGraph, requiring only three key parameters:

1. LLM instance: An initialized LLM (e.g., GPT-4, DeepSeek, etc.)
2. Tool list: Register four core tools (Search Query, List Files, Read File, Reason File)
3. System Prompt: A system prompt guiding the model on how to use tools

LangGraph is an Agent orchestration framework from the LangChain team, building Agent workflows based on the concept of Directed Graphs. Unlike traditional chain-based calling (Chain), LangGraph supports loops, conditional branching, and persistent state management, making it ideal for implementing multi-turn iterative Agentic RAG. create_react_agent is a high-level API provided by LangGraph that internally implements a complete ReAct loop state machine — including a "call model" node, an "execute tool" node, and conditional edges connecting them. Developers only need to define tools and prompts; the framework automatically handles message passing, tool execution, and loop termination decisions.

agent = create_react_agent(
    llm, 
    tools=[search_query, list_files, read_file, reason_file], 
    system_prompt=system_prompt
)
result = agent.invoke({"messages": [user_query]})

The code looks remarkably concise, but it grants the model autonomous decision-making and dynamic adjustment capabilities. There's a critical practical detail: The System Prompt must strictly constrain the model's response format, otherwise the model may strip quotes or brackets, causing tool call parameter parsing failures.

Summary: Tools Grant Capability, Intelligence Lies in Choice

Traditional RAG is a combination of two fixed workflows — document chunking → vectorization → storage, and vector retrieval → prompt assembly → answer generation. Simple and direct, but lacking flexibility, unable to handle retrieval failures or incomplete information.

Three core upgrades of Agentic RAG:

• Toolification: Encapsulate retrieval, file reading, and other capabilities as callable tools
• Delegated Decision-Making: Grant the LLM autonomous decision-making authority, letting the model determine when to call which tool
• Iterative Loops: The model can dynamically plan, call tools, observe results, and perform multiple iterations based on task requirements

Tools grant capability, while intelligence lies in choice. True Agentic RAG begins with retrieval and succeeds through decision-making. Many so-called "wrapper applications" in China are fundamentally this simple at their core — extract the key components and leverage frameworks like LangChain and LangGraph to implement a RAG system with agent-like behavior.

If you're building LLM applications, consider starting with traditional RAG and progressively introducing tool calling and Agent mechanisms to evolve your system from "can only search" to "can think."