RAG Knowledge Base: A Complete Guide from Fundamentals to Optimization

What Problem Does RAG Actually Solve

RAG stands for Retrieval-Augmented Generation. Its core purpose is to supply large language models with external knowledge they don't inherently possess. An LLM's parameters contain no enterprise-internal data — RAG bridges that gap by introducing an external knowledge base, allowing the model to reference that data when answering questions or generating content. This improves accuracy and reduces the likelihood of hallucinations.

Put simply, RAG is like plugging an external knowledge base into a large model. That external data can be structured (e.g., records from a relational database) or unstructured (e.g., PDFs, Word documents, plain text). RAG's value is especially pronounced in knowledge-intensive tasks — when a user's question depends on a large amount of specialized or proprietary information, the model's internal memory alone often can't produce a reliable answer.

It's worth noting that vector databases support multimodal storage. Images can also be converted into vectors and stored — though image embedding requires a model specifically designed for visual content, and image vectors should be stored in a separate collection from text vectors to ensure clean retrieval later.

Vector databases are the storage backbone of any RAG system, and they work in a fundamentally different way from traditional relational databases. Traditional databases match records by exact field values. Vector databases, by contrast, store high-dimensional floating-point vectors — each chunk of text, after passing through an embedding model, becomes a numerical array of hundreds to thousands of dimensions, where semantically similar content ends up closer together in that high-dimensional space. At query time, the user's question is also converted into a vector, and the database uses an ANN (Approximate Nearest Neighbor) algorithm to quickly find the most similar vectors and return the corresponding original text chunks. This capability for "semantic retrieval" rather than "keyword matching" is the technical foundation that allows RAG to understand natural language queries and surface relevant knowledge. Milvus, Chroma, Weaviate, and Pinecone all fall into this category, each with different trade-offs in performance, deployment model, and ecosystem.

The Basic RAG Pipeline

A minimal RAG system can be broken down into two major phases: building the knowledge base and retrieval and generation.

The build phase starts with data ingestion. Data may come from local files or from an enterprise's relational databases. Once you have the data, text needs to be chunked — split into individual pieces; images, on the other hand, are converted directly into vectors as a whole. After chunking or conversion, an embedding model transforms each piece into a vector, which is then stored in the vector database.

The retrieval phase begins when a user asks a question. The system queries the vector database based on that question, processes the results as needed, passes the processed content to the LLM as context, and the model generates the final answer.

The pipeline sounds straightforward, but the implementation is full of nuance. Developers who are used to traditional CRUD projects tend to underestimate RAG — in a traditional project, you basically ship once the features work and tests pass. AI projects, however, require continuous attention to whether the accuracy and recall of the output actually meet business requirements. If they don't, optimization is mandatory.

Key Areas for RAG Optimization

The underlying principles of RAG are the same everywhere — what truly differentiates systems is the quality of optimization. Optimizing RAG means improving each stage of the pipeline to push the probability of hallucinations as low as possible. Here are the core optimization areas:

Choosing a Vector Database

Different data scales call for different vector databases. Small-scale and large-scale deployments often make different choices. Milvus was selected for the author's project based on feedback from peers in AI roles, who noted that Milvus is widely used in large-scale production deployments. That said, it's not the only option — just one of the more mainstream representatives.

Data Loading and Chunking

Local files come in many formats — PDF, Word, Markdown, TXT — and each requires a different loading approach. Chunking is one of the areas where real craft shows. Some tools (like Dify) default to fixed-length chunking: set the chunk size to 200 characters, cut every 200 characters. But fixed-length chunking isn't always optimal. Other strategies include splitting on punctuation, splitting by semantics, splitting by paragraph, splitting PDFs by page, or splitting Markdown by heading.

Bad chunking has direct consequences: if a complete sentence gets sliced in half, its semantic features are broken, increasing the risk that the LLM will hallucinate in its response.

Choosing an Embedding Model

You can use commercial embedding models (OpenAI, Zhipu, Qwen, etc.) or open-source models from HuggingFace. One important distinction: text embedding and image embedding are handled by different models — they are not interchangeable.

The core function of an embedding model is to map text into a semantic space so that sentences with similar meanings have vectors that are close together. Different models vary significantly in vector dimensions, maximum input length (context length), and Chinese language support. For example, OpenAI's text-embedding-3-small outputs 1536-dimensional vectors, while HuggingFace's BGE family of models tends to handle Chinese better and can be deployed locally — avoiding data-residency compliance concerns. One critical rule: you must use the same embedding model to build the knowledge base and to run queries. Different models produce incompatible vector spaces, and mixing them will cause retrieval to fail entirely. Embedding models also have token limits, so long texts must be chunked before encoding — which is exactly why chunking strategy and embedding model selection need to be considered together.

Retrieval: Combining Multiple Strategies

The simplest RAG systems use only similarity search — returning results ranked by cosine distance. But mature retrieval pipelines go much further, incorporating range search, grouping, hybrid search, full-text search, and more. In practice, multiple retrieval strategies are often combined to maximize the accuracy of the final result set.

Re-ranking

After retrieving the top 10 or top 5 results, those results are ranked by similarity — but high similarity doesn't necessarily mean high relevance. That's why a separate reranking model is introduced to re-score the retrieved results by relevance before they're packed into the prompt and handed to the LLM.

The position of re-ranking in the pipeline is clear: retrieve → take top results → re-rank → assemble prompt → pass to LLM. Skipping re-ranking will degrade the quality of the final answer.

Reranker models operate differently from embedding models. Embedding models encode the query and each document independently and then compare distances — fast, but limited in precision. Rerankers typically use a Cross-Encoder architecture, where the query and candidate document are concatenated and fed through the model together, producing a direct relevance score. This is more accurate but also more computationally expensive. That's why real-world systems typically use a two-stage approach: use vector similarity to quickly recall a top-100 or top-50 candidate set from a large corpus, then use a reranker to score that candidate set precisely, passing only the top 5 or top 10 to the LLM. Widely used open-source rerankers include BGE-Reranker and Cohere Rerank; commercial API options are also available.

Integrating Agents and Workflows

More advanced RAG systems incorporate Agents and graph-based workflows, making the overall processing logic more flexible and controllable.

In a RAG context, an Agent plays the role of "decision-maker and orchestrator." Rather than routing every question through a fixed retrieval pipeline, it dynamically determines whether retrieval is needed at all, which knowledge base to query, and whether external tools (such as a search engine, calculator, or database query) should be invoked. Graph Workflow systems allow each step in the RAG pipeline to be defined as a node, with conditional branching between nodes — for example, "if vector retrieval confidence is too low, route to the full-text search node." This kind of flexibility is especially valuable in complex business scenarios: when a user's question contains multiple sub-questions, an Agent can decompose the question, run parallel retrievals, and merge the results — rather than dumping a compound question into a single retrieval call, which typically leads to poor recall. Tools like Dify, LangGraph, and LlamaIndex Workflows all provide implementation frameworks for this kind of capability.

Why Start with the Vector Database

The author makes a point that in AI projects, the "requirements" are usually extremely simple — user asks, system answers, one sentence covers it. The real substance of the project isn't in the requirements; it's in how well you engineer each stage of the RAG pipeline.

And at the center of the entire pipeline sits the vector database. Chunked text gets stored there; retrieval happens against it; re-ranking operates on what it returns. Without the vector database, there's no foundation for chunking, retrieval, or re-ranking. That's why the tutorial starts with the vector database (Milvus) — establish the storage foundation first, then work outward through chunking, embedding, retrieval, and re-ranking.

Summary

The principles behind RAG aren't complicated. But between those principles and a production-grade system lies a mountain of engineering details. Vector database selection, chunking strategy, embedding model choice, multi-strategy retrieval, re-ranking, Agent and workflow integration — every one of these areas has room for optimization, and every one of them influences the final hallucination rate. For developers building private knowledge bases, understanding what each stage does and how they relate to one another matters far more than memorizing the flow.