Deep Dive into RAG Systems: Why Enterprise Private Knowledge Bases Are Harder Than You Think

Many people's first impression of RAG (Retrieval-Augmented Generation) is that it seems "almost too simple." A student once said bluntly during a live session: doing RAG development at an enterprise, he could pick it up in a week. But developers who've shipped multiple RAG projects will tell you — beneath that seemingly three-step process lies an entire layer of engineering complexity. This article draws on the hands-on teaching of a seasoned RAG developer to break down the real gap between introductory understanding and production deployment.

What Problem Does RAG Actually Solve

In plain terms, RAG is about giving an existing open-source LLM access to an additional batch of external data. Models like DeepSeek^Ad or other open-source options available out of the box often lack sufficient data coverage and domain-specific knowledge. When you need a model to understand a vertical industry or a company's proprietary internal knowledge, RAG is the go-to solution.

The core workflow breaks down into two phases. The first is data preparation: ingest the company's text, images, audio, video, and other content through structured or unstructured data loaders, convert it into vectors using an embedding model, and store everything in a vector database. The second is querying: when a user asks a question, the system doesn't send it directly to the LLM. Instead, it first retrieves relevant results from the vector database using various search strategies, injects those results into the prompt, and then passes everything to the LLM to generate a complete answer.

This workflow has been explained in countless blog posts and technical articles, and many people know it by heart. But as the author repeatedly emphasizes: do you really think a RAG project is that simple?

Data Loading and Chunking: The First Engineering Hurdle

The heart of a knowledge base is the vector database. As data volume grows, you need to consider distributed retrieval and distributed data management — which databases support distributed architectures, which are commonly chosen in enterprise environments — all of this requires upfront planning. Fortunately, the database layer is relatively manageable: hand off the installation and deployment to your cloud infrastructure colleagues and get it running in Kubernetes or a Docker container.

What's truly tricky is data loading. Take PDFs as an example: a single file often contains text, tables, and images all at once. During a live course demo, the author opened a PDF: the first few pages appeared to be pure text, a table showed up on page five, and page eleven contained an image with embedded text.

The questions follow immediately: Can you extract table contents in a structured format? Can you perform OCR on text embedded in images? Handling complex, mixed-format data is itself a specialized technical skill that requires dedicated study.

After loading comes chunking. If you ingest an entire PDF in one shot, the excessive text length will seriously degrade downstream retrieval quality. Chunking strategy matters a great deal: should you split by heading, by paragraph, by semantics, or some combination? Take a research paper PDF — a single paragraph can easily run one to two thousand characters. Splitting purely by paragraph creates chunks that are too large; splitting by period breaks semantic coherence. The right approach is often semantic chunking — and "how to do semantic chunking" is itself a technical topic that deserves its own deep dive.

Semantic chunking typically uses an embedding model or a dedicated text-splitting model to map text into a vector space and then identify semantic boundaries — when the vector similarity between adjacent sentences drops significantly, that's treated as a semantic breakpoint. LangChain's SemanticChunker and LlamaIndex's SemanticSplitterNodeParser both provide ready-to-use implementations. Beyond semantic chunking, a "parent-child chunk" strategy is also common in production: large chunks store the full context, while small chunks serve as retrieval units. When a small chunk is matched, the system traces back to retrieve the parent chunk to pass to the LLM, balancing retrieval precision with contextual completeness. Chunk overlap is equally important — too little loses cross-chunk semantics, too much introduces noise and inflates storage costs. In practice, overlap is typically set to 10–20% of chunk size.

Embedding Model Selection: Open-Source Options Are More Than Competitive

After chunking comes vectorization. The choice of embedding model directly affects retrieval quality. The author specifically highlights BGE-M3, released by the Beijing Academy of Artificial Intelligence (BAAI), describing it as being in the global top tier.

What makes it especially valuable is that BGE-M3 is released under the MIT license, meaning you can deploy it directly on your own infrastructure, use it to vectorize images, text, audio, and other modalities — completely free of charge. By contrast, OpenAI's embeddings come with usage fees, and DeepSeek's embedding model isn't open-source and requires paying for tokens, making it more expensive. Alibaba has also open-sourced the GTE series of embedding models, which are another solid option. For organizations looking to control costs, open-source solutions are more than capable.

Retrieval and Re-ranking: The Real Hard Part of Enterprise RAG

The author is explicit: the hardest components in the entire pipeline are the retriever and re-ranking.

To make retrieval results as accurate as possible, enterprises often run multiple retrieval strategies simultaneously: similarity search (ANN / approximate nearest neighbor), relevance-based retrieval, full-text search, keyword matching, filtered retrieval, and even grouped retrieval. The richer the retrieval strategies, the more precise the context fed to the LLM, and the higher the answer quality.

But multi-path retrieval creates a new problem: too many results. Suppose similarity search returns 5 results, full-text search returns 5, and filtered search returns another 5 — that's 15 total, when you only want to pass 5 to the LLM. You need to consolidate and rank all 15 results to select the best 5. That's what re-ranking does. Re-ranking is itself a model and can employ various strategies including weighted scoring.

The author also demonstrated code using a self-query retriever — a more intelligent retrieval approach that can automatically correct and optimize retrieval logic based on the query.

Re-ranking models (Cross-Encoders) are fundamentally different from the Bi-Encoders used in vector retrieval: Bi-Encoders encode the query and document into separate vectors and compute similarity — fast, but limited in precision. Cross-Encoders concatenate the query and candidate document and compute a relevance score jointly — more accurate, but computationally expensive, making them suitable only for precision re-ranking of a small set of candidates after the initial recall stage. Widely used open-source re-ranking models include BAAI's BGE-Reranker series and Cohere's Rerank API. In practice, multi-path retrieval results typically need to go through deduplication and score normalization before entering the re-ranker — otherwise, the scores from different retrieval paths are on incompatible scales, and direct weighted aggregation introduces bias.

From RAG to Agent: Dynamic Routing and Evaluation

Is it done once you pass the re-ranked results to the LLM? The author argues that a system designed this way isn't intelligent enough. In enterprise practice, the RAG vector store is typically connected to an Agent.

The reason: not every question needs to go through RAG. Some questions the LLM can answer directly without retrieval; some require querying the vector store; others should be answered by querying a relational database via tool calling. That's why you need an Agent to perform dynamic routing — deciding which path each user question should take to get the best answer. For more complex scenarios, a Workflow combined with RAG is also common.

The final layer is RAG evaluation, handled by the RAGAS (Retrieval-Augmented Generation Assessment) framework. Whether the RAG system you've built actually works needs to be measured quantitatively — and the evaluation itself is typically automated using an LLM.

RAGAS quantifies RAG system quality through four core metrics: Faithfulness measures whether the answer is grounded solely in the retrieved context, avoiding LLM hallucinations; Answer Relevancy measures how well the response addresses the question; Context Precision measures the proportion of useful information in the retrieved results; and Context Recall measures whether the retrieval covers all key information needed to answer the question. These four metrics are typically scored automatically by a powerful model like GPT-4, so RAGAS is essentially a paradigm of "using LLMs to evaluate LLMs." In enterprise deployments, building a high-quality test set of "question — ground-truth answer — reference document" triplets is a prerequisite for making RAGAS evaluation results genuinely trustworthy.

Closing Thoughts: Simple Principles, Complex Engineering

The fundamental principles of RAG can indeed be explained in a few sentences — which is precisely why it tends to be underestimated. But from data loading, chunking strategy, and embedding model selection, to multi-path retrieval, re-ranking, and Agent-based dynamic routing, all the way to final RAGAS evaluation, every single step hides a substantial amount of engineering detail that needs careful attention. Getting a complete, production-ready RAG system running inside an enterprise is far from something you can "learn in a week." Understanding the depth of that complexity is the very first step from introduction to real-world deployment.