Building RAG from Scratch: Creating a Domain Expert System with Local Small Models

What if you could make a large language model an expert in a domain it has never encountered — not by retraining, but by letting it retrieve relevant documents in real time? This is the core value of RAG (Retrieval-Augmented Generation).

RAG didn't appear out of thin air. It is the culmination of decades of NLP research in "Open-Domain QA." As early as 2017, Facebook AI Research proposed the DrQA system, combining TF-IDF retrieval with reading comprehension models for Wikipedia Q&A, establishing the prototype of the "retrieve first, then read" two-stage paradigm. RAG was formally introduced by the Meta AI research team in 2020 in the paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, rooted in the longstanding debate between "parametric memory vs. non-parametric memory" in NLP. The fundamental difference lies in how knowledge is stored: traditional LLMs encode knowledge in a distributed fashion across neural network weights (parametric memory), making updates extremely costly; the external document storage introduced by RAG (non-parametric memory) exists in an editable form that can be added, deleted, or modified at any time — a design philosophy highly aligned with the classic computer science principle of "separating compute from storage." This approach was inspired by classical information retrieval algorithms like TF-IDF and BM25, combined with the powerful language understanding capabilities of neural networks, forming the "retrieval-generation" hybrid paradigm. Its core motivation is to address two inherent limitations of LLMs: Training Cutoff and Hallucination. Traditional LLMs have their knowledge "frozen" at the training data's time point and cannot perceive the latest information; RAG dynamically injects external documents during inference, enabling the model to access proprietary knowledge from any point in time without incurring the massive computational cost of retraining tens of billions of parameters. This article uses a fun hands-on project to walk you through building a complete RAG pipeline from scratch, transforming a local small model with only 1.5 billion parameters into an AI detective capable of "cracking cases."

Project Overview: An "Investigation" About Elrond

The tutorial's scenario is quite entertaining: prepare 10 custom "evidence documents" for the model (covering witness testimonies, forensic reports, etc.) and have it investigate whether "the Elf Lord Elrond's secret identity is Agent Smith." Beneath the absurd premise lies a serious tech stack:

• Ollama: Run LLMs locally for free with complete data sovereignty
• LangChain: A unified interface for Python to call Ollama. LangChain is essentially a "glue layer" framework whose core value lies in providing unified abstract interfaces for numerous heterogeneous components (models, vector stores, document loaders, memory modules, etc.), enabling developers to assemble complex AI pipelines declaratively. It's worth mentioning that LlamaIndex, as another important alternative, has a design philosophy more focused on data indexing and retrieval layers, each with its own strengths when handling complex document structures
• FAISS: Efficiently store and retrieve vector embeddings
• Qwen 2.5 (1.5B): Responsible for final conversation and answer generation
• BGE-M3: Responsible for text embedding representation

The entire pipeline runs on CPU without requiring an expensive GPU, making it perfect for beginners to get hands-on experience.

Environment Setup

After installing Ollama via the Linux command provided at ollama.com, pull two lightweight models: ollama pull qwen2.5:1.5b (conversation) and ollama pull bge-m3 (embedding). Then create a Python 3.12 environment with conda (named ragenv), install dependencies like langchain, faiss, pypdf, and jupyter, and start coding in Jupyter Lab.

Document Loading and Chunking: The Starting Point of the RAG Pipeline

The first step of RAG is loading documents into memory. The tutorial uses os.listdir to list all PDFs in the data folder, sorts them with sorted to ensure consistent ordering, then loads each one via PyPDFLoader, extracts each page's content with loader.load(), and finally aggregates everything into the all_pages list — yielding 49 pages in this case.

Chunking: Why 49 Pages Became 147 Chunks

After loading comes the Chunking stage. This step splits pages into smaller text units while preserving some overlap between adjacent chunks to prevent semantic breaks at boundaries.

Text chunking is a critically underestimated step in the RAG pipeline — chunking strategy directly determines the upper bound of retrieval quality. Chunks too large cause embedding vectors to become "diluted," reducing retrieval precision; chunks too small may lose complete semantic units. Beyond fixed character count splitting (RecursiveCharacterTextSplitter), several advanced strategies have emerged in engineering practice: Semantic Chunking (dynamically determining split points based on sentence embedding similarity, better preserving paragraph semantic boundaries), structure-based splitting (e.g., by Markdown heading hierarchy or HTML tags, suitable for technical documents with clear hierarchical structure), and Parent Document Retriever (storing small chunks for precise retrieval while returning the corresponding large parent document to preserve complete context). The choice between strategies depends on document type, domain characteristics, and downstream task requirements, and typically needs to be validated quantitatively through evaluation frameworks.

The author uses a vivid analogy to explain why overlap is necessary: split "this is not my first rodeo" into two chunks, getting "this is not my" and "not my first rodeo" — the overlapping "not my" acts like a safety net, allowing adjacent chunks to share partial context. The chunk_overlap parameter design borrows from the sliding window concept, ensuring that key information spanning chunk boundaries isn't lost due to boundary cuts.

A noteworthy practical detail: after the initial split, the chunk count remained at 49, equal to the page count. The reason was that these PDFs had large fonts and plenty of whitespace, so the character count per page was already below the default 2000-character limit — the splitter had "nothing to work with." The solution was to explicitly adjust parameters: set chunk_size to 500 and chunk_overlap to 150, re-run and obtain 147 chunks. This demonstrates that chunking parameters must be tuned based on actual data characteristics rather than blindly using defaults.

Embeddings and Vector Database: Making Semantic Retrieval Possible

Next comes the Embeddings stage. Text embedding technology has evolved through three generations: the first was sparse representations based on word frequency statistics (TF-IDF, BM25), unable to capture synonym relationships; the second was static word vectors like Word2Vec and GloVe, which first encoded semantic relationships as geometric distances but couldn't distinguish polysemous words across different contexts; the third is contextual dynamic embeddings represented by BERT, where the same word receives different vector representations in different sentences, greatly improving semantic understanding precision. Embedding is the process of transforming discrete text into continuous high-dimensional vectors — its essence is encoding semantic relationships as geometric distance relationships: semantically similar texts are closer in high-dimensional vector space. Notably, the choice of embedding model can impact RAG system performance even more than the generation model — the "garbage in, garbage out" principle applies equally to vector retrieval.

The BGE-M3 (BAAI General Embedding - Multi-Functionality, Multi-Linguality, Multi-Granularity) used in this project is a multi-functional embedding model released by the Beijing Academy of Artificial Intelligence, supporting over 100 languages and representing the cutting edge of current embedding technology. Its architectural uniqueness lies in unifying three retrieval paradigms within a single model: Dense Retrieval (matching semantics via cosine similarity), Sparse Retrieval (term frequency weight matching similar to BM25, excelling at precise keyword alignment), and Multi-Vector Retrieval (ColBERT-style token-level fine-grained interaction). This multi-mode fusion maintains high performance in both semantically ambiguous queries and precise terminology queries. Unlike closed-source solutions such as OpenAI's text-embedding-ada-002, BGE-M3 is fully open-source and locally deployable, with a vector dimension of 1024, performing on par with commercial models on standard benchmarks like MTEB — making it an ideal choice for local RAG solutions.

The embedding model acts like a "LEGO organizer": first sorting bricks by color so that actual assembly becomes much more efficient. Loading the embedding model requires just one line: OllamaEmbeddings(model='bge-m3'). Storing the processed text chunks relies on the vector database FAISS.

As RAG technology has gained popularity, the vector database market has formed a layered ecosystem: among local lightweight solutions, FAISS excels with its serverless, zero-network-dependency design; ChromaDB offers a more user-friendly Python API and is a popular choice for prototyping. Among cloud-native solutions, Pinecone pioneered commercialization, while Weaviate and Qdrant stand out for being open-source, self-hostable, and supporting hybrid search. Traditional databases are catching up too — PostgreSQL supports vector indexing through the pgvector extension. Choosing the right solution requires considering data scale, update frequency, latency requirements, and team operational capabilities.

vectordb = FAISS.from_documents(chunks, embeddings)
vectordb.save_local("faiss_index")

FAISS (Facebook AI Similarity Search) is Meta's open-source high-efficiency similar vector search library, designed for billion-scale vector retrieval scenarios. Its core idea is replacing exact nearest neighbor search (Exact NN) with approximate nearest neighbor search (ANN), offering multiple index types for different scales: small-scale scenarios use Flat Index (brute-force exact search, guaranteeing 100% recall); medium-to-large-scale scenarios leverage IVF (Inverted File Index) to partition the vector space into multiple Voronoi cells, searching only the nearest cells during queries rather than the entire database, drastically reducing computation; PQ (Product Quantization) reduces memory usage by segmenting and compressing high-dimensional vectors, at the cost of slight precision loss. Engineers must balance recall rate, query latency, and memory cost. In this tutorial's small-scale scenario, FAISS with a Flat index is more than sufficient for retrieving a few hundred vector chunks. As a local library, FAISS requires no network requests and keeps all data on your machine, perfectly aligned with Ollama's local-first philosophy.

from_documents completes "chunking → embedding → storage" in one step, while save_local persists the results to disk, eliminating the need for repeated preprocessing — simply load and use.

Retriever: Giving the Model the Ability to "Search Through Evidence"

With the vector database in place, you need a Retriever to find relevant text chunks based on questions. After creating a retriever with vectordb.as_retriever(), calling retriever.invoke("why is Elrond under investigation") returns relevant content.

By default, 4 chunks are returned, controllable via search_kwargs={'k': 5} — once again reflecting the RAG philosophy that "every parameter is tunable." The retrieval count k is an important hyperparameter affecting system performance: too small a k may miss critical evidence, while too large a k injects noise into the model, increasing context window pressure. Additionally, LangChain supports MMR (Maximal Marginal Relevance) retrieval strategy — which actively penalizes highly redundant candidate chunks while maintaining relevance, improving diversity in retrieval results and avoiding injecting homogenized information repeatedly into the context. A more cutting-edge direction includes Query Rewriting techniques, where an LLM expands or rewrites the user's vague question into multiple more precise retrieval queries before searching, then merges multi-path recalls to improve overall recall rate.

The retrieval produces a list of chunks, but the conversational model needs a continuous string. Therefore, iterate through all chunks, concatenating each chunk's page_content with \n\n into a single context string, to be fed to the conversational model as "case materials."

Comparison Experiment With and Without RAG: The Difference Is Crystal Clear

To visually demonstrate RAG's value, the author first conducted a control test: directly calling llm.invoke with a question and providing no context. The model either refused to answer citing "political sensitivity" or fabricated content like "blockchain" and "money laundering" — classic Hallucination behavior.

The root cause of hallucination lies in LLMs' autoregressive generation mechanism — the model is essentially predicting "the next most probable token," not "the most truthful information." Hallucinations can be subdivided into two types: Factual Hallucination (fabricating specific facts beyond the training data, such as fake paper citations or fictional biographical details) and Faithfulness Hallucination (generated content deviates from or contradicts the given context, i.e., the model "fails to faithfully follow" the input material). RAG primarily addresses the first type: by injecting real document fragments into the prompt and requiring the model to "answer based only on the following materials," it constrains open-ended generation into an evidence-based information extraction task, significantly reducing the probability of the model "making things up." However, RAG cannot completely eliminate hallucination — the model may still reason incorrectly about retrieved evidence or "fill in" information from parametric memory when context is insufficient. Production systems typically require additional layers of defense, including answer confidence assessment, source citation mechanisms, and post-generation grounding checks.

With RAG introduced, the author encapsulated an ask(question) function that passes the retrieved context along with the user's question to the model via an f-string. When asked again "why is Elrond being investigated," the model provided an accurate answer: because he may be connected to a cross-dimensional entity known as "Agent Smith."

At this point, the complete RAG pipeline is built, and Qwen, the small model, has "officially become an expert on this case." It's worth noting that Qwen 2.5 1.5B is a lightweight model released by Alibaba Cloud's Tongyi Qianwen team in 2024, supporting 32K context length and specifically optimized for instruction following and structured output. In RAG scenarios, the model doesn't need to "memorize" knowledge — it only needs good reading comprehension and information extraction capabilities, which is precisely what instruction-tuned small models excel at. This explains why a 1.5B model can produce professional-grade output that exceeds what its parameter count might suggest.

From Demo to Production: Key Best Practices

To elevate a RAG system from "toy-level" to "production-ready," several areas deserve attention:

• Conversation History Management: Real systems involve multi-turn conversations, but models have no inherent memory — developers must store and maintain history themselves. LangChain provides various memory management solutions like ConversationBufferMemory, which can retain key historical information through strategies such as summary compression within limited context windows.
• System Prompts and Role Definition: Assigning the model a clear identity (detective, lawyer, consultant, etc.) through a System Prompt can significantly improve output consistency and professionalism. Well-designed system prompts can also constrain the model to proactively state "unable to find relevant information from the given materials" when context is insufficient, rather than filling in gaps — further suppressing hallucination.
• Vector Store Persistence and Loading: Using indexes saved with save_local avoids rebuilding the database on every restart, dramatically reducing startup overhead.
• GPU Acceleration: With a CUDA-capable GPU, simply replace faiss-cpu with faiss-gpu — no other code changes needed; Ollama automatically detects and enables GPU.
• Systematic Hyperparameter Tuning: Parameters like chunk size, overlap, and k significantly impact retrieval quality. Quantitative evaluation can be conducted using frameworks like RAGAS (RAG Assessment). RAGAS solves the challenge of quantitatively evaluating open-domain Q&A through the LLM-as-Judge paradigm — using another powerful language model as a judge, it constructs four core evaluation dimensions: Answer Relevancy (how well the generated answer matches the question), Context Precision (the proportion of retrieved document chunks that are actually useful, measuring retrieval "accuracy"), Context Recall (whether all needed information was retrieved, measuring retrieval "completeness"), and Faithfulness (whether the answer is entirely based on retrieved content rather than model parameters). These four dimensions form a mutually balancing evaluation matrix, enabling engineers to replace intuition-based tuning with data-driven optimization, finding the optimal balance between retrieval efficiency and answer quality.
• Moving Toward Agentic RAG: A more cutting-edge direction is giving the system autonomous reasoning capabilities to decide "whether retrieval is needed," "which knowledge base to search," and "how to decompose complex problems" — evolving from a static pipeline to a dynamic reasoning system. LangGraph, launched by LangChain, is designed precisely for such Agent systems requiring iterative reasoning and conditional branching.

Conclusion

This seemingly absurd "Elf Lord investigation case" clearly breaks down every core component of RAG: Load → Chunk → Embed → Store → Retrieve → Generate. It demonstrates that even a local small model with just 1.5 billion parameters, when paired with the right retrieval pipeline, can deliver reliable, evidence-backed answers in specific domains.

For developers looking to build domain knowledge bases locally at low cost and with full control, Ollama + LangChain + FAISS + Qwen is a solid starting point. Once you've mastered RAG's core logic, the natural next step is exploring AI Agents capable of autonomous planning and task execution.