Getting Started with RAG Knowledge Bases: From Vector Retrieval to Embedding Models Explained

What Problem Does RAG Actually Solve

The core idea behind Retrieval-Augmented Generation (RAG) is straightforward: insert a "find relevant material" step between the user's question and the LLM's generated response. The system retrieves content related to the user's query from a vector database and provides it as context to the model, giving it a factual basis for its answer.

This design addresses a fundamental limitation of LLMs. The models we use every day have been pretrained on vast amounts of general internet knowledge, which makes them quite capable of answering common, general-purpose questions. But pretraining data rarely includes domain-specific expertise — such as a company's internal business definitions, technical documentation, or proprietary terminology. The model has never "seen" this content, so it naturally struggles when asked about it.

That's precisely where RAG adds value. By feeding a model enterprise-private documents as reference material, it gains targeted, relevant context when generating answers — leading to noticeably more accurate responses, especially in specialized vertical domains.

The Retrieval Pipeline: From User Query to Context Assembly

A complete RAG pipeline can be broken down into a few clear steps. When a user submits a question (user query), it's first passed to an Embedding model, which converts it into a vector. Whether the user's question is a few words or a detailed paragraph, the Embedding model "understands" the text and outputs a corresponding vector representation.

Next comes the critical retrieval step. The vector database already stores a large collection of vectors generated by chunking and vectorizing documents in advance. The system takes the new vector of the user's question and performs a similarity search against the database, retrieving the top-K most relevant documents (top-k similar documents). This step is essentially a query — it surfaces the text fragments that are semantically closest to the user's question.

The retrieved content is then used as context and combined with the user's original question — think of it as string concatenation, merging the question and reference material into a larger input that gets sent to the LLM to generate a final answer. At this point, the model not only knows what was asked, but also has access to relevant background knowledge.

Building the vector database (the "preparation phase") is a prerequisite for RAG to function. Before the system goes live, all private documents must be split into smaller chunks, each converted into a vector by an Embedding model, and stored in the vector database. The chunking granularity directly affects retrieval quality: chunks that are too long introduce too much unrelated semantic content, reducing precision; chunks that are too short don't carry enough information to answer a question. Common chunking strategies include fixed character count splits, paragraph or heading-based splits, and sliding window splits with overlap — the last of which prevents key information from being cut off at chunk boundaries. Vector databases themselves (such as Pinecone, Weaviate, Chroma, Milvus, etc.) build indexes over these vectors so that approximate nearest neighbor (ANN) search can be completed in milliseconds across millions of vectors, rather than brute-force comparing every entry.

Why Context "Relevance" Is Critical

Retrieval quality directly determines RAG's effectiveness. If the reference material returned to the model has little to do with the user's question, it won't just fail to help — it will actively interfere with the model's ability to generate a correct answer. Organizations often have enormous volumes of articles and technical documentation; the key is whether the system can precisely recall the portion most relevant to the current question.

This is exactly why the quality of the Embedding model matters so much. It needs to ensure that the vector for a user's question accurately matches the truly relevant document vectors in the vector space, rather than pulling in unrelated content. The goal of similarity search is always to find the one or few fragments that are the best match.

An Easily Overlooked Detail: The Embedding Model Must Be Consistent

One critically important point that beginners often miss: the preparation phase (pre-production, when building the vector database) and the serving phase (in-production, when processing new user queries) must use the same Embedding model.

A relatable analogy helps explain why. Take the exact same sentence and ask two different people to interpret it — the results may diverge or even contradict each other. An Embedding model's core job is to "understand" a piece of text — what it's about, what topic it covers. If you use one model to build the knowledge base and then switch to a different model at query time, you're effectively having two different "interpreters" process the text. Their semantic spaces won't align, and similarity calculations will produce meaningless results.

As one viewer succinctly put it during a livestream: using the same Embedding model ensures that historical documents and incoming user queries are mapped into the same semantic space — only then can similarity calculations be consistent and reliable.

Understanding Embeddings: Mapping Text into High-Dimensional Space

The term "Embedding" refers to the process of converting a piece of text or a sentence into a vector — essentially a sequence of numbers separated by commas, where each number represents one dimension of the vector. If a vector has 7 components, it exists in 7-dimensional space.

Why call it "embedding"? Think of it this way: placing that 7-component vector into a seven-dimensional space turns it into a point in that high-dimensional space. In other words, the embedding process "embeds" a piece of text into high-dimensional space, giving it a specific coordinate. Text with similar meaning will cluster close together in this space — which is the geometric foundation that makes similarity search possible.

In practice, when calling an Embedding model, the input might be a previously chunked document fragment, or it might be a new user query. Sometimes an instruction (system prompt) is prepended — for example, "Translate the following content" — to guide how the model processes the input.

The choice of similarity metric determines what "closeness" means mathematically. The most commonly used is cosine similarity: it measures the cosine of the angle between two vectors, with values ranging from −1 to 1 — the closer to 1, the more semantically similar. Cosine similarity's advantage is that it focuses on direction rather than vector magnitude, making it insensitive to text length. Euclidean distance (L2 distance) is another option, but cosine similarity is the dominant approach for text semantic retrieval. It's worth noting that modern Embedding models output far more than 7 dimensions — mainstream models typically produce 768, 1536, or even more dimensions. Higher-dimensional spaces can encode finer semantic distinctions, but they also place greater demands on storage and computation.

Conclusion: RAG Is an Essential Skill for LLM Application Development

The pipeline described above outlines the most fundamental RAG approach. In real-world LLM application development, RAG is nearly indispensable — and it's already extending into multimodal territory, with multimodal RAG incorporating images and audio becoming increasingly common. For developers looking to work in LLM application development, mastering RAG and multimodal RAG is a core foundational skill.

Understanding the two places where an Embedding model appears in the pipeline (building the database and processing queries) and the consistency requirement between them lays the groundwork for deeper topics ahead — such as fine-tuning Embedding models.

Beyond the basic "single-round retrieval" paradigm, several advanced techniques have emerged in engineering practice. HyDE (Hypothetical Document Embeddings) has the model first generate a "hypothetical answer" from scratch, then uses that answer to perform vector retrieval — compensating for cases where a user's query is too brief for an adequate vector representation. Re-ranking adds a precision-ranking step after top-k retrieval, using a stronger cross-encoder model to rescore candidate fragments and further improve the quality of context sent to the LLM. Multi-hop RAG addresses complex questions that require reasoning across multiple documents, using iterative retrieval rounds to progressively piece together a complete answer path. All of these techniques build on the foundational pipeline introduced in this article and represent the key engineering methods for taking a RAG system from "functional" to "genuinely good."