RAG Retrieval-Augmented Generation in Practice: Building an Enterprise Private Knowledge Base from Scratch

What Is RAG: Making LLMs Understand Your Enterprise Data

RAG (Retrieval-Augmented Generation) is an indispensable core technology for building enterprise-grade AI applications today. The problem it solves is straightforward: existing large language models simply don't know anything about your company's internal data. Whether it's audio, video, text, or business information stored in databases — none of this private content was seen by general-purpose LLMs during training.

There are typically two approaches to enabling an LLM to answer questions about internal enterprise knowledge. The first is Tool/Function Calling, where the model actively connects to internal data sources when generating a response. The second — and the focus of this article — is RAG: when a user asks a question, relevant content is first retrieved from a knowledge base and then passed to the LLM to generate an answer.

Here's a practical example: imagine a company has accumulated a large archive of conversations between human customer service agents and users, and wants to build an intelligent customer service assistant from this data. These historical conversations become the raw material for the knowledge base. When a user asks "My computer is broken — what should I check first for repairs?", we want the Agent to give a professional answer grounded in the company's internal knowledge, not a generic response.

Building the Knowledge Base: From Document Processing to Data Chunking

The first step in RAG is transforming enterprise documents (Docs) into a searchable knowledge base. This isn't as simple as dumping files into a system — it requires a series of processing steps.

Data Conversion: Normalizing Heterogeneous Document Formats

Enterprise documents come in all formats — PDF, Markdown, Excel — and may contain complex structures like embedded images, tables, and headers. The goal of data conversion is to transform this heterogeneous content into a form the knowledge base can handle.

Take images as an example: a knowledge base typically can't store or retrieve images directly, so image content must first be extracted via OCR or multimodal recognition and stored as searchable text. Structured elements like tables and headers also need targeted processing at this stage.

Data Chunking: Breaking It Down

Once converted, documents are split into many small segments called Chunks. The reason is simple: a large document contains too much content to process as a whole — doing so would hinder precise retrieval and exceed model context limits. Splitting large documents into smaller pieces makes subsequent retrieval more focused and efficient.

It's worth noting that chunking strategy involves many considerations — fixed-length splits, semantic splits, overlapping windows, etc. Different approaches directly affect retrieval quality, making this one of the key tuning areas in real-world RAG implementations.

The three main chunking strategies are: Fixed-length chunking, which splits by character count or token count (e.g., 512 tokens per chunk) — simple to implement but may cut mid-sentence and break semantic coherence; Semantic-aware chunking, which splits at natural boundaries like paragraphs, headings, or sentence endings — better semantic coherence but uneven chunk sizes; and Sliding window chunking, which introduces an overlap region between adjacent chunks to prevent key information from being split across boundaries.

Chunk granularity is a core tuning parameter for RAG performance: chunks that are too long introduce noise and dilute relevance; chunks that are too short may lose context, leaving the LLM with insufficient information. In practice, it's common to start experimenting with 256–512 token granularity and iterate based on actual retrieval quality.

Vectorization and Retrieval: Embedding Is the Core Mechanism

After chunking, each Chunk must go through Embedding processing. Embedding is the process of converting text into a mathematical vector representation using an embedding model.

Why Vectorization Is Needed

When a user asks a question, how does the system know which parts of the knowledge base are relevant? The answer lies in vector computation. The user's question is also embedded into a vector, which is then compared against the vectors of all chunks in the knowledge base using similarity calculations. The most commonly used method is cosine similarity, which identifies the chunks most semantically related to the user's question.

The elegance of this mechanism is that it matches on semantics rather than literal keywords. Even if the user's phrasing differs entirely from the original document text, semantically similar content can still be retrieved.

From a technical perspective, embedding models are neural networks specialized in mapping text to high-dimensional vector spaces. Common open-source options include OpenAI's text-embedding-ada-002, Alibaba's text-embedding-v3, and the BGE series optimized for Chinese (e.g., BAAI/bge-large-zh). Output vector dimensions range from 768 to 3072 across different models — higher dimensions generally offer stronger semantic representation, but at the cost of increased storage and computation.

The logic of cosine similarity: the smaller the angle between two vectors, the closer the value is to 1, indicating higher semantic similarity; the larger the angle, the closer to 0 or even negative, indicating greater semantic distance. This is fundamentally different from traditional keyword search (BM25), which requires literal term matching. In practice, many systems combine both approaches — known as hybrid retrieval — to benefit from both precise matching and semantic generalization.

Choosing a Vector Database

Embedded vectors need to be stored in a dedicated vector database. The most widely used is Milvus, with alternatives including PostgreSQL (pgvector), Redis, and even Elasticsearch (ES). The right choice depends on data scale, performance requirements, and your team's existing tech stack.

The key difference between vector databases and traditional relational databases lies in the index structure: traditional databases use B-Trees for fast exact lookups, while vector databases use Approximate Nearest Neighbor (ANN) algorithms to accelerate similarity search over high-dimensional vectors. Common ANN algorithms include HNSW (Hierarchical Navigable Small World) and IVF (Inverted File Index), which balance search speed and recall rate — enabling millisecond-level results even across tens of millions of vectors.

Milvus supports multiple index types and hybrid retrieval, making it well-suited for large-scale production deployments. pgvector, as a PostgreSQL extension, offers minimal migration overhead for teams already on a PG stack. Chroma and Qdrant are popular for prototyping due to their lightweight and easy-to-use nature. Beyond performance, also consider whether the database supports metadata filtering — narrowing retrieval results by document source, timestamp, department, etc. alongside vector search — which is often a hard requirement in enterprise scenarios.

Generating Answers: Combining Retrieval Results with an LLM

The final step in the RAG pipeline is passing the retrieved relevant chunks to an LLM to generate the final response.

Concretely: after a user asks a question, the system uses the question's embedding to query the vector database and retrieves the most relevant Chunks. These chunks are organized into a prompt along with the user's original question and passed to the LLM. The prompt typically tells the model explicitly: what question the user asked, what relevant knowledge was retrieved from internal sources, and instructs the model to answer based on that knowledge.

This way, the LLM can give accurate answers grounded in private enterprise knowledge, rather than improvising based on generic training data. The result is an AI application with enterprise knowledge capabilities — what's commonly called an intelligent Agent.

Full RAG Pipeline Recap and Practical Recommendations

Putting all the steps together, a complete RAG pipeline can be summarized as:

1. Document Processing: Convert enterprise Docs, handling complex content like images and tables;
2. Data Chunking: Split large documents into appropriately sized Chunks;
3. Vectorization and Indexing: Embed each chunk and store it in a vector database (e.g., Milvus);
4. Retrieval Matching: Vectorize the user's question and retrieve the most relevant chunks via cosine similarity;
5. Answer Generation: Organize retrieved results and the question into a prompt, and pass it to the LLM to generate the final answer.

The pipeline is conceptually straightforward, but real-world implementation hides a wealth of details at every step — how to determine chunk granularity, which embedding model to choose, how to improve retrieval recall, how to design effective prompts, and even how to incorporate knowledge graphs to enhance retrieval. These details are where self-learners often get stuck: the theory makes sense, but a single small issue in practice can eat up hours.

For developers who want to master RAG systematically, the recommendation is to learn with a real project — start by building a minimal viable knowledge base and getting the full pipeline running end-to-end, then gradually optimize each component. Only through repeated hands-on iteration and debugging can you truly internalize this technology.