RAG from Beginner to Practitioner: A Complete Tutorial for Building a Private Knowledge Base

Large language models are powerful, but they have one fundamental limitation: they don't know your company's internal data. Whether it's customer service conversation logs, product documentation, or business information stored in databases, general-purpose LLMs can't access any of it directly. To make an LLM answer questions based on private data, there are currently two mainstream approaches: connecting to internal enterprise data sources through tool calling, or using RAG (Retrieval-Augmented Generation).

According to the instructor sharing this tutorial on Bilibili, the core idea behind RAG isn't complicated—before the LLM answers a user's question, it first retrieves relevant content from the enterprise knowledge base, then generates an answer based on that content. This article provides a complete breakdown of how RAG works, the knowledge base construction process, and every key step from document processing to final answer generation.

What Is RAG and Why Do We Need It

RAG stands for Retrieval-Augmented Generation. The problem it solves is straightforward: LLMs don't know your company's internal data, and RAG enables LLMs to dynamically retrieve and leverage this private knowledge when answering questions.

Here's a concrete scenario. Suppose you have a batch of historical conversation records between human customer service agents and users, and you want to build an intelligent customer service assistant. A user asks, "My computer is broken, what aspects should I consider when repairing it?" You want the Agent's response to draw on the customer service knowledge your company has accumulated, rather than giving generic advice. This is where RAG becomes the core technical path to achieving that goal.

Compared to having the LLM connect to internal enterprise data through tools, RAG's advantage is that it doesn't require the model to directly access databases. Instead, it "feeds" relevant knowledge to the model through a retrieval mechanism, offering better flexibility and security.

The Core Workflow of RAG

The entire RAG pipeline can be broken down into two phases: the knowledge base construction phase and the user Q&A phase.

Phase One: Building the Knowledge Base

Enterprise raw data (Docs) comes in all sorts of formats—PDF, Markdown, Excel, images, tables, and more. The first step in building a knowledge base is data conversion. For example, if a document contains embedded images, the knowledge base can't directly store the image content, so you need to use OCR or similar methods to recognize the text in the images and convert it into searchable text.

After conversion, the next step is data chunking (Chunks). A large document contains too much content and needs to be split into smaller segments. The granularity of chunking and the overlap strategy both affect the accuracy of subsequent retrieval—there's a lot of nuance here.

After chunking comes the Embedding step. Each text segment is converted into a vector (a mathematical numerical representation) through an embedding model, then stored in a vector database. Currently, the most commonly used vector database is Milvus, though Redis, Pinecone, Elasticsearch, and others can also serve as storage solutions.

Embedding is the semantic foundation of the entire RAG system. Embedding refers to using a neural network model to map a piece of text into a high-dimensional vector (typically a floating-point array between 768 and 1536 dimensions), such that semantically similar texts are closer together in vector space. For example, "computer won't turn on" and "the machine can't start" use different words but their embedded vectors will be very close. Common embedding models include OpenAI's text-embedding series, BGE, M3E, and others. In Chinese-language scenarios, BGE and M3E perform well. The choice of embedding model directly determines whether retrieval can "understand" user intent rather than merely matching keywords—this is also the core difference between vector retrieval and traditional full-text search (such as Elasticsearch's BM25 algorithm).

Phase Two: User Queries and Answer Generation

When a user asks a question, the system also embeds the user's question, converting it into a vector. Then it performs similarity computation in the vector database—cosine similarity is the most widely used—to find the text segments most semantically relevant to the user's question.

Once relevant segments are found, the system passes these segments along with the user's original question to the LLM. The prompt explicitly tells the model: the user's question is X, the relevant internal enterprise knowledge found is Y, please answer based on this knowledge.

This process is the most general principle of RAG. User asks → vectorization → similarity retrieval → context assembly → LLM generates answer, forming a complete closed loop.

Cosine similarity measures how close two vectors are in direction, with values ranging from -1 to 1. The closer to 1, the more semantically similar. It only considers vector direction and is unaffected by magnitude, making it particularly suitable for comparing text embedding vectors. Related metrics include Euclidean distance (measuring straight-line distance between vectors) and dot product (a variant combining both), but cosine similarity has become the most mainstream choice in RAG scenarios due to its insensitivity to vector magnitude and stable computation. Most vector databases (such as Milvus) support multiple distance metrics by default; in practice, you can choose the metric that corresponds to how the embedding model was trained to achieve the best retrieval results.

Key Components and Technology Selection

After understanding the workflow, the selection of several key components deserves attention.

Embedding models determine the quality of text vectorization and directly affect retrieval precision. Different models show notable differences in Chinese semantic understanding, so selection must be based on actual scenarios.

For vector databases, Milvus has become the community's top choice due to its open-source nature and good scalability. If you already have Redis or Elasticsearch infrastructure, you can reuse it directly to reduce deployment costs.

Chunking strategy is an easily overlooked but hugely impactful step. If segments are too large, retrieved information contains too much noise; if too small, contextual semantics may be lost. Real-world projects require repeated tuning.

Prompt design is equally critical. How to effectively organize retrieval results into the Prompt so the LLM accurately uses this knowledge rather than ignoring it is a detail that needs refinement in engineering practice.

The tutorial sharer also mentioned that the problem with self-learning RAG is that the knowledge system isn't systematic. When encountering specific problems (such as unclear project execution workflows), there's no one to provide timely answers, and a thin layer of confusion can hold you up for a long time.

From RAG to Knowledge Graphs: Advanced Directions

RAG is not the endpoint. The sharer revealed they are writing the knowledge graph section of a new project, which also points to an important advanced direction for RAG technology—when simple vector retrieval can't satisfy complex reasoning and relationship query needs, knowledge graphs can serve as a supplement or even replacement for RAG.

Knowledge graphs excel at handling relationships between entities, such as multi-hop reasoning like "product A's failure cause leads to customer complaint type B." Combining knowledge graphs with RAG is one of the frontier directions the industry is currently exploring.

For developers just getting started, running through the basic RAG pipeline and building a working private knowledge base Q&A system is the most pragmatic starting point. Later, you can gradually introduce more complex retrieval strategies and knowledge organization forms based on business needs.

Knowledge graphs are a way of organizing knowledge using graph structures, where nodes represent entities (such as products, customers, failure types) and edges represent relationships between entities (such as "causes," "belongs to," "complains about"). Unlike vector databases that store text segments, knowledge graphs explicitly model semantic associations between entities, making them naturally suited for answering questions like "what is the relationship between A and B" or "what are the causes of C" that require multi-hop reasoning. A typical approach to combining knowledge graphs with RAG is GraphRAG: first extract entities and relationships from documents to build a graph, then during queries perform both vector retrieval and graph traversal, passing structured relationship paths to the LLM as well. This approach often outperforms pure vector retrieval in scenarios requiring cross-document relational reasoning.

One-line Summary

RAG retrieves enterprise private knowledge and feeds it to LLMs for answer generation, solving the limitation of LLMs not knowing internal data.

Paragraph Summary

The article systematically breaks down the working principles and implementation workflow of RAG (Retrieval-Augmented Generation). The core idea of RAG is to have the LLM first retrieve relevant content from the enterprise knowledge base before answering, then generate a response based on that content, enabling general-purpose LLMs to use private data. The workflow has two phases: the knowledge base construction phase includes data conversion (such as OCR for recognizing text in images), data chunking, and Embedding vectorization stored in a vector database (commonly Milvus); the Q&A phase vectorizes the user's question in the same way, retrieves the most relevant segments through cosine similarity, assembles them into a prompt, and passes it to the LLM for answer generation. The article also discusses key selections such as embedding models, vector databases, chunking strategies, and prompt design, and points out that knowledge graphs are an advanced direction for handling complex reasoning.