Building an Offline RAG Application from Scratch: A Complete Guide to a Private PDF Knowledge Base

A Fully Offline PDF Q&A Application

Recently, a developer shared their hand-built local RAG (Retrieval-Augmented Generation) application on Reddit. The project had a clear goal: upload a PDF document and let a large language model answer questions based solely on that document's content, rather than "fabricating" answers from its training data. The entire pipeline runs completely locally and works even without an internet connection.

This case is worth attention not because it uses cutting-edge technology—quite the opposite. It uses the most straightforward tech stack to clearly demonstrate the complete working principles of a RAG system. For developers who want to understand RAG but have been discouraged by complex frameworks, this is a rare entry-level example.

Tech Stack: The Triumph of Minimalism

The author's technology combination is anything but fancy:

Ollama: Handles running the chat model and embedding model locally
ChromaDB: Serves as the vector database for storing document vectors
Flask: A lightweight web framework that ties all the components together

Ollama is an open-source framework for running large language models locally that rose rapidly in the second half of 2023. Its core value lies in dramatically simplifying the once-complex model deployment process—developers only need a single command (such as ollama run llama3) to pull and run mainstream open-source models locally, without manually managing CUDA environments, model weight format conversions, or inference engine configurations. Ollama is built on top of llama.cpp, which uses quantization techniques (compressing model weights from 32-bit floating point to 4-bit or 8-bit integers) to significantly lower hardware requirements, enabling consumer-grade GPUs or even pure CPUs to run models with billions of parameters. In this RAG application, Ollama serves dual roles as both the chat model (responsible for understanding questions and generating answers) and the embedding model (responsible for converting text into vectors), called through a unified API interface—one of the key reasons this tech stack can remain so simple.

ChromaDB is an embedded vector database designed specifically for AI applications, open-sourced in 2022 by Chroma. Unlike cloud-native vector databases such as Pinecone and Weaviate, ChromaDB supports fully local operation and can be embedded directly into applications as a Python library, with no need to deploy a separate database service. Its architecture prioritizes developer experience: data can be stored in memory (suitable for prototyping) or persisted to local disk (suitable for production use), and its API design is extremely concise—adding documents or querying vectors both require only a few lines of code. In terms of performance, ChromaDB uses the HNSW (Hierarchical Navigable Small World) algorithm under the hood to build its vector index, allowing it to maintain millisecond-level query response even at scales of hundreds of thousands of vectors. For personal projects and small-to-medium knowledge bases, ChromaDB is currently one of the most beginner-friendly vector database options available.

In the author's own words: "Nothing exotic." The value of this minimalist choice is that it makes the entire data flow process crystal clear, avoiding the comprehension barriers that come with the black boxes of large frameworks.

Breaking Down the Core Mechanics of RAG

RAG (Retrieval-Augmented Generation) was formally introduced by Meta AI's research team in 2020 in the paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." Its core motivation was to address two inherent flaws of large language models: the knowledge cutoff date (training data is time-bound) and the hallucination problem (models tend to confidently fabricate nonexistent information). The traditional solution is fine-tuning the model, but fine-tuning is costly, time-consuming, and difficult to keep knowledge updated in real time. RAG offers a lighter-weight path: without changing the model weights, it dynamically injects external knowledge during the inference phase, allowing the model to "read" content it was never trained on. This approach closely mirrors how humans consult reference books before answering, which is why it has rapidly gained widespread adoption in knowledge-intensive scenarios such as enterprise knowledge bases, legal, and medical fields.

The essence of RAG is to have the large model retrieve relevant information from an external knowledge source before generating an answer, then answer based on that information. This application fully reproduces this process, which can be broken down into four key steps.

Step One: Chunking

The PDF is split into text chunks with overlapping portions. The importance of chunking in RAG systems is often underestimated, but it is widely regarded in the industry as one of the most critical factors affecting the final outcome. Chunking strategy involves balancing three core parameters: chunk size, overlap length, and the choice of split boundaries. Chunks that are too small lack sufficient context, making it hard for the model to make accurate judgments based on fragmented information; chunks that are too large introduce excessive noise, diluting truly relevant content while also being constrained by the model's context window length.

The overlap design here is crucial—if you split rigidly at fixed lengths, you may very well cut a complete sentence in half, causing semantic loss. The core purpose of the overlap design is to prevent key information from falling exactly at a split point and being separated into the edge regions of two chunks. By preserving a certain amount of overlapping content between adjacent chunks, the system can maximally maintain contextual coherence. More advanced chunking strategies include: splitting based on sentence boundaries, splitting by semantic units such as paragraphs or sections, and the recently popular "parent-child chunk" structure (retrieving small chunks but providing the larger parent chunk to the model as context).

This detail may seem insignificant, but it is actually one of the key factors determining RAG retrieval quality. The quality of the chunking strategy directly affects whether subsequent retrieval can hit truly relevant content.

Step Two: Vectorization and Persistent Storage

Each text chunk is converted into a vector (embedding) via the embedding model, then stored in ChromaDB.

Embedding is the technique of mapping discrete text symbols into a continuous high-dimensional vector space. Its core idea derives from the distributional semantics hypothesis: semantically similar words or sentences are also closer together in the vector space. Modern embedding models (such as the sentence-transformers series) typically output 768- or 1536-dimensional vectors, where each dimension implicitly encodes some semantic feature of the text. Vector similarity is usually computed using cosine similarity, which measures the angle between the directions of two vectors rather than their absolute distance, making it insensitive to vector length and better suited for semantic matching scenarios. Intuitively: no matter how long or short a passage is, as long as it expresses the same meaning, the "direction" its vector points to should be similar—and cosine similarity is precisely the mathematical tool for capturing this directional consistency.

The author specifically used PersistentClient, so vector data is written to disk rather than memory. This choice is very practical: with in-memory storage, all processed documents would disappear every time the application restarts, forcing users to re-upload and re-process everything. Persistent storage allows the knowledge base to accumulate over the long term—a key step in moving from a "toy project" to a "usable tool."

Step Three: Semantic Retrieval

When a user asks a question, the question itself is also converted into a vector. ChromaDB then performs an Approximate Nearest Neighbor (ANN) search, computing the similarity between this vector and all text chunk vectors in the database, and returns the closest chunks.

The ANN algorithm is the core technology of vector databases. Take HNSW (Hierarchical Navigable Small World), which ChromaDB adopts, as an example: this algorithm was proposed by Yury Malkov and colleagues in 2016. Its core idea borrows from "six degrees of separation" theory, constructing a multi-layered graph structure—upper-layer nodes are sparse and responsible for long-distance jumps, while lower-layer nodes are dense and responsible for fine-grained localization—reducing the search time complexity from linear O(n) to the order of O(log n). In a million-scale vector database, HNSW typically completes retrieval in just a few milliseconds while maintaining a recall rate above 95%. This is precisely what distinguishes RAG from traditional keyword search—it matches semantic similarity rather than literal matches. Even if the user's wording differs from the original document text, the system can accurately locate the corresponding content as long as the meaning is close. For example, when a user asks "Tell me the payment terms of the contract," it can match a passage in the original text stating "Party B shall complete settlement within 30 days after delivery"—different expressions but aligned semantics.

Step Four: Constrained Generation

The retrieved relevant text chunks, along with the question, are passed to the chat model as context. There is a crucial design here: the prompt explicitly requires the model to answer using only the provided context, and if the answer is not within it, to directly reply "I don't know."

The root of large language models' hallucination problem lies in the fact that the model's training objective is to predict the next most likely token, not to verify the truthfulness of statements. This means that when facing questions beyond its training knowledge, the model often "fills in" answers with plausible-sounding content rather than honestly expressing uncertainty. Prompt engineering is currently the most direct mitigation method: by explicitly setting behavioral rules in the system prompt, one can constrain the model's output tendencies to some extent. This technique of anchoring the model's generation process to verifiable information sources is called "grounding"—literally providing a verifiable "foundation" for the model's output to prevent it from "drifting away" from known facts during reasoning.

Without this constraint, the large model would likely "take matters into its own hands" and fabricate answers using knowledge from its training data—a typical source of the "hallucination" problem. By forcing the model to adhere to the principle of "knowing what you know and knowing what you don't," the application's reliability is greatly improved. It's worth noting that prompt constraints are not foolproof; in scenarios with high reliability requirements, mechanisms such as output verification and citation tracing are needed as supplementary safeguards.

Insights from Two Validation Tests

The author conducted two tests on the application, which happened to hit the two areas most prone to problems in a RAG system.

Test One: Refusing to Fabricate

The author deliberately asked a question whose answer was not contained in the PDF. As a result, the model correctly answered "I don't know" rather than guessing wildly. This validated the effectiveness of the prompt constraint—the system genuinely achieved "faithfulness to the document."

For enterprise-level applications or scenarios requiring factual accuracy (such as contract analysis, legal document queries, technical manual Q&A), this behavior of "rather saying I don't know than making things up" is often more important than answering fluently.

Test Two: Fully Offline Operation

After the author turned off WiFi, the application still worked normally. The chat model, embedding model, and vector database all run locally, and the entire process involves no external API calls whatsoever.

This point is highly significant, manifesting specifically in:

Data privacy protection: Sensitive documents don't need to be uploaded to third-party servers; everything is processed locally
Zero call cost: No need to pay API fees for each inference
No network dependency: Works normally on airplanes, in intranet environments, or in any offline scenario

For individuals or teams handling confidential materials, the appeal of this localized RAG deployment approach is obvious. Against the backdrop of increasingly strict data compliance requirements (such as GDPR and domestic data security laws), localized deployment is gradually shifting from a "nice-to-have" to a rigid necessity in certain scenarios.

Project Value and Directions for Advanced Optimization

From an engineering perspective, this application does not solve any highly difficult technical problems. But its value lies in being complete, clear, and reproducible—turning RAG from an abstract concept into an actual system that runs in just a few hundred lines of code.

For developers hoping to get started with RAG, this project offers the shortest path: Ollama handles local models, ChromaDB handles vector storage, and Flask handles the interactive interface—each with its own role, with transparent logic.

To push it toward a production environment, the following directions are worth further optimization:

Intelligent chunking strategy: Try semantic-based dynamic splitting rather than fixed-length truncation
Improved retrieval precision: Introduce a reranking mechanism to filter noise and improve the accuracy of relevant chunks. Reranking typically uses a CrossEncoder architecture, concatenating the query with each candidate chunk before performing fine-grained relevance scoring—unlike the Bi-Encoder that generates embedding vectors, the CrossEncoder can directly model the relevance of "question-passage" pairs end-to-end, offering higher precision but slower speed, and is therefore usually only used for a second refinement of the coarse ranking results. In industrial-grade RAG systems, this two-stage "coarse ranking + fine ranking" architecture has become standard configuration
Multi-document joint retrieval: Expand from a single PDF to unified retrieval across an entire private document library
Hybrid retrieval mode: Combine keyword retrieval with vector retrieval to compensate for the coverage blind spots of pure semantic retrieval. The keyword retrieval here typically uses the BM25 algorithm—a classic algorithm proposed by Stephen Robertson and colleagues in 1994 that scores keyword matches through an elegant combination of term frequency, inverse document frequency, and document length normalization, and remains Elasticsearch's default scoring mechanism to this day. For content lacking "semantic neighbors" such as proper nouns, product model numbers, and code variable names, BM25's literal exact matching is often more reliable than semantic retrieval. The two retrieval results are usually merged via the RRF (Reciprocal Rank Fusion) algorithm, and this "hybrid retrieval" combination has become standard practice in industrial-grade RAG systems

The message this case ultimately conveys is clear: many seemingly sophisticated AI applications have surprisingly intuitive underlying logic. As long as you understand how data flows, any developer can build their own offline intelligent Q&A system.