Compiled RAG in Practice: How to Choose Among Three RAG Approaches

Why Your Knowledge Base Always Gets Answers Wrong

Many people encounter the same frustration when building knowledge bases: the model is fine, the prompts are long enough, but the answers just aren't accurate. The root cause often isn't the model — it's that the knowledge itself has never been truly "compiled."

The traditional approach is to chunk documents and toss them into a vector database, but after this processing, fragments are isolated from each other with no connections. When a user asks a cross-document question, the system can only fish out a similar-looking text snippet and attempt an answer — it can't explain the source clearly, and users can't build trust. Worse still, after documents are updated, the index remains stuck on old versions, likely outputting outdated content.

In other words, what you're missing isn't a more expensive vector database, but the ability to compile scattered documents into a navigable knowledge network. This is exactly the core problem that Compiled RAG — the so-called LLM Wiki approach — aims to solve.

What Is Compiled RAG (LLM Wiki)

From "Chunk Retrieval" to "Knowledge Compilation"

The vector RAG approach breaks documents into chunks, vectorizes them, and matches by similarity during retrieval. Specifically, it relies on Text Embedding technology, using pre-trained language models to map text fragments into points in a high-dimensional vector space, where semantically similar texts are closer together. During retrieval, the user's query is also encoded as a vector, and the closest document fragments are found using metrics like cosine similarity or Euclidean distance. This approach is simple and direct, but inherently suffers from a "fragmentation" flaw — it preserves surface-level textual similarity while losing structural relationships and logical threads between documents. It cannot capture logical reasoning chains across documents, nor handle composite questions that require synthesizing multiple pieces of information.

The key difference in Compiled RAG lies in the "compilation" step. The term "compilation" borrows from software engineering: just as a compiler transforms human-readable source code through lexical analysis, syntax analysis, and semantic analysis into machine-executable binary code, Compiled RAG transforms human-written unstructured documents through content parsing, knowledge point extraction, relationship modeling, and conflict resolution into a structured knowledge network that machines can efficiently navigate and retrieve. Rather than simply storing text fragments, it first performs structured processing on documents — extracting knowledge points, establishing references and associations between them, and forming a navigable knowledge network. It's like organizing a pile of scattered notes into a Wiki with a table of contents, index, and cross-references, where both users and models can follow links to find related content.

Three-Layer Architecture Design of Compiled RAG

Compiled RAG typically adopts a three-layer architecture to achieve this goal:

• Raw Layer: Preserves the original document content as the ultimate basis for traceability;
• Compilation Layer: Performs structured extraction and relationship modeling, transforming scattered documents into knowledge nodes and relationship networks;
• Application Layer: Serves Q&A retrieval, supporting answers with provenance so every conclusion can be traced back to a specific source.

The value of this architecture is that it preserves the traceability of original information while solving cross-document association problems through the intermediate compilation layer.

Q&A Retrieval with Provenance: Making Knowledge Base Output Trustworthy

Whether a knowledge base can be trusted hinges on its ability to "cite its sources clearly." A major advantage of Compiled RAG is provenance-based Q&A: when the model provides an answer, it can simultaneously annotate which document and which section the conclusion comes from.

Provenance is a mature research direction in data management, referring to tracking the complete path of data from generation to its current state. In RAG systems, provenance is typically implemented by maintaining reference chains from generated answers to source document fragments. Specific technical approaches include: preserving metadata (document ID, section position, version number) for each knowledge node during the compilation phase, returning reference information alongside answers during the retrieval phase, and presenting them as footnotes or hyperlinks in the display layer.

This is especially important for enterprise scenarios. Whether it's compliance review, technical support, or internal knowledge management, users need to verify the authority of information sources. In financial compliance (e.g., EU MiFID II regulations requiring investment advice to be traceable), healthcare (diagnostic recommendations needing evidence-based medical citations), and legal services (legal opinions needing specific statute references), provenance capability has evolved from "nice to have" to "hard requirement." An answer without sources, even if correct, is difficult to rely on for critical decisions. A retrieval mechanism with provenance essentially builds a credibility endorsement for every output from the knowledge base.

Automated Knowledge Update Mechanism

Solving the Old Problem of "Stale Indexes"

Documents get updated but indexes don't — this is the biggest headache for many knowledge base projects. Compiled RAG addresses this through automated cascading knowledge association updates: when a source document changes, the system can identify affected knowledge nodes and update related associations accordingly, rather than waiting for someone to manually re-ingest the entire corpus.

This cascading update mechanism draws from Reactive Programming and Incremental Computation concepts. When a source document changes, the system uses change detection (diff) to identify the specific modified content, then follows association edges in the knowledge network to find all downstream nodes that depend on that content, triggering recompilation. This is similar to dependency graphs in build tools (like Make or Bazel): only the affected parts are rebuilt, not the entire system. Key technologies for implementing this mechanism include: fine-grained content hashing for change detection, topological sorting to determine update order, and conflict detection mechanisms to handle contradictory descriptions of the same knowledge point across multiple documents.

This automation capability truly brings the knowledge base to life, ensuring users always get the latest version rather than outdated content from months ago. For teams with frequently iterating documents, this is practically a must-have.

Comparison of Three RAG Approaches and Selection Guide

Vector RAG: Lightweight and Fast, Suitable for Simple Retrieval

Vector RAG is suitable for scenarios with moderate data volumes and weak inter-document associations. Its advantages are simple setup, low cost, and fast retrieval speed. Common vector databases include Pinecone, Milvus, Weaviate, Qdrant, etc., which achieve millisecond-level retrieval across millions of vectors through Approximate Nearest Neighbor (ANN) search algorithms (such as HNSW, IVF). If your primary need is "finding the most relevant passage from a collection of documents," vector RAG is often sufficient. Its weakness lies in struggling with cross-document questions that require reasoning about associations.

Graph RAG: Excels at Entity Relationships and Multi-Hop Reasoning

Graph RAG constructs knowledge graphs with explicit modeling of entities and relationships, making it particularly suitable for complex queries requiring multi-hop reasoning and emphasizing entity associations. Knowledge graphs use triples (entity-relationship-entity) as their basic unit, first proposed by Google in 2012 to enhance search engine semantic understanding. Graph RAG builds on this by using large language models to automatically extract entities and relationships from unstructured text, then performing multi-hop reasoning based on graph traversal algorithms after constructing the graph. Microsoft Research's GraphRAG project released in 2024 is a representative work in this direction, using community detection algorithms to create hierarchical summaries of the graph, supporting answers to global questions.

When your questions take the form of "What's the connection between A and B," graph structures can provide answers that vector retrieval cannot. The tradeoff is higher cost to build and maintain the graph, with main challenges including entity disambiguation (how to unify different expressions of the same entity), accuracy of relationship extraction, and storage and query performance issues as the graph scales.

Compiled RAG: Best Practice for Structured Knowledge Networks

Compiled RAG (LLM Wiki) sits between and goes beyond both approaches — it compiles documents into a navigable knowledge network that balances provenance, associations, and automatic updates. Compared to Graph RAG, Compiled RAG doesn't require strict triple modeling but adopts more flexible knowledge node and association representations, lowering the construction barrier; compared to vector RAG, it explicitly preserves structural relationships between documents through the compilation layer, compensating for the shortcomings of pure semantic matching. For personal or enterprise knowledge bases that require long-term maintenance, frequent updates, and emphasis on trustworthy provenance, this approach offers the highest overall value.

Core Criteria for RAG Selection

The key to selection isn't which approach is "more advanced," but rather your data characteristics and use cases:

• Simple data, infrequent updates, only need similarity retrieval → Vector RAG
• Emphasis on entity relationships, multi-hop reasoning → Graph RAG
• Need a navigable, traceable, auto-updating knowledge network → Compiled RAG

It's worth noting that these three approaches are not mutually exclusive. In practice, more and more systems adopt hybrid architectures: using vector retrieval for initial recall, graph structures for relationship reasoning, and the compilation layer for knowledge management and provenance. When making selections, you should think more about: which approach serves as the backbone, and how the others complement it.

Final Thoughts

For developers building or planning to build knowledge bases, and decision-makers who need to make technical choices, understanding the applicable boundaries of these three RAG approaches is more important than blindly pursuing more expensive vector databases.

Compiled RAG offers not just a technical solution, but a philosophy for upgrading "scattered documents" into "knowledge assets" — only by first compiling knowledge into a navigable, traceable, auto-updating network can models deliver truly accurate and trustworthy answers.

Key Takeaways