Don't Start with a Vector Database: Build a Keyword Search Baseline First

When building RAG assistants, many teams' first instinct is to deploy a vector database. But Italian developer Omar Baruzzo offers a different take in a widely discussed blog post: don't rush to vectors — build a baseline with the search capabilities you already have, and let your evaluation data tell you when you actually need embeddings.

First, look at what users are actually asking

Baruzzo's suggested first step isn't technology selection — it's collecting real questions. Get questions from users who actually use the assistant, not from your imagination. After collecting them, do a categorical breakdown, and you'll notice a pattern: a sizable share of questions are actually pointing queries — contract numbers, customer names, date ranges, material codes, specific people's names.

These kinds of queries happen to be a weak spot for vector search. Vector search compares semantic similarity and is inherently bad at exact matching. Conversely, keyword search and SQL filters are extremely precise at matching identifiers, but they fall short when the user's wording differs from the document's wording.

The key insight is this: both of these capabilities usually already exist in your stack — you don't need to introduce new infrastructure.

Pointing queries (pointer queries) are questions where the user has a clear, unique matching target — for example, locating a particular record or group of records by contract number, customer name, date range, material code, or a specific person's name. The core requirement for these queries is "exact hit," not "semantic similarity." In traditional search, they're typically handled directly by inverted indexes, B-tree indexes, or SQL WHERE conditions. In vector retrieval, since embeddings map strings into high-dimensional semantic vectors, strings like numbers, dates, and codes that have no semantic structure can produce unpredictable similarities, causing exact targets to be drowned out or mis-recalled. So identifying what proportion of real questions are pointing queries is one of the key criteria for deciding whether vector search is needed.

First, establish a comparable baseline

Once you've settled on a set of real questions, the second step is to write standard correct answers for about fifty of them. Then run full-text search plus structured filters and record how many questions return correct results.

That number is your baseline. Its significance is this: any subsequent new technical approach must be able to beat this baseline, otherwise it doesn't count as an improvement.

Baruzzo puts it bluntly: without a baseline, adding vector search can't be called "improvement" — only "change." This is a subtle but important distinction — teams can easily mistake "the system changed" for "the system got better."

Baseline here refers to a reproducible evaluation reference: a fixed set of real questions, fixed standard answers, a fixed scoring method, and then running the simplest existing approach to get a score. Its value isn't in how high or low the score is, but in giving every subsequent technical change a comparable anchor. Without a baseline, teams can only judge by gut feeling that "it seems to have gotten better"; with a baseline, they can answer verifiable questions like "vector search raised accuracy from 62% to 71%." For RAG systems, a baseline usually also includes metrics like retrieval hit rate, answer correctness rate, and permission-filter correctness rate — and it must be re-run repeatedly on the same question set, otherwise comparisons lose their meaning.

Where vectors should come in

Embeddings truly show their value in vocabulary mismatch scenarios. A user asks "how do I end a lease early," while the document says "early termination clause for lease agreements" — same meaning, different words — and pure keyword search will miss the relevant document.

When the evaluation set shows this type of error dominates, hybrid search has a solid justification: keywords plus vectors, merging the results. At that point, the benefit can be quantified, and the team can judge whether that benefit is worth the extra cost.

The core of the methodology here is: let error types drive technical decisions, not the other way around.

Vocabulary mismatch is a classic problem in information retrieval: the user's wording doesn't match the terminology actually used in the document, though the meaning is the same. For example, the user says "end the lease early," while the document says "early termination clause for lease agreements"; the user says "the computer won't turn on," while the document says "device fails to start properly." Keyword search relies on literal matching and easily misses results when faced with synonyms, near-synonyms, or different phrasings. Vector search maps different wordings into a similar vector space through semantic embeddings, thereby closing this gap. But vector search isn't a silver bullet: it may recall content that's semantically similar but factually irrelevant, and it can be unstable when exact matching is needed. So hybrid search (keywords + vectors) is usually more robust than either approach alone — provided the evaluation data proves that vocabulary mismatch really is the primary source of errors.

One index is one copy

The cost of introducing a vector index is often underestimated. Baruzzo points out an easily overlooked problem: a vector index is essentially a copy of the documents.

This means three things: the index needs to stay in sync with the data source; switching embedding models requires rebuilding the entire index; and the index must enforce the same access controls as the original system.

The last point is especially critical. A user who has no permission to open a document in the original application shouldn't be able to retrieve it through the assistant. Permission gaps in copies are silent and hard to detect, and if personal data is involved, they can also constitute a compliance issue.

Access control in RAG assistants is an often overlooked but high-risk area. When documents are loaded into a vector index, an independent data copy is effectively created, and this copy often bypasses the original system's permission checks. If a user has no right to open a contract or work order in the original application, but that document has been embedded into the index, the assistant may "leak" it to unauthorized users through semantic retrieval. What's worse, this leak is silent: the user won't get a "permission denied" message, and the system won't throw an error — it only surfaces during an audit or incident review. If personal data, financial records, or compliance-sensitive information is involved, this can also violate data protection regulations. So a vector index must inherit the same permission model as the original system and enforce filtering at the retrieval stage, not just check at the display stage.

This is an argument about order of operations

Baruzzo makes clear that the above discussion is not an argument against vector search itself, but about order of operations.

Teams with a measurement baseline can clearly explain what incremental value embeddings actually bring; teams without a baseline can only say vaguely that "the assistant seems to have gotten better." When the next generation of models or index versions is released, teams with a baseline can re-run the question set and determine whether switching is worthwhile; teams without a baseline have no basis for comparison.

Start with boring search, measure it, and let the errors tell you what to add. In the midst of the vector database hype, this line of thinking deserves serious consideration from every team building a RAG system.

One-line Summary

When building a RAG assistant, start with a keyword search baseline and let error types decide whether to introduce vector search.

Paragraph Summary

Italian developer Omar Baruzzo argues that building a RAG assistant shouldn't begin with deploying a vector database. Instead, you should first collect real user questions, establish a keyword search baseline, and then let evaluation data decide whether embeddings are needed. Many user queries are actually pointing queries — contract numbers, customer names, date ranges — where keyword search and SQL filters are more precise than vector search. Embeddings truly solve the vocabulary mismatch problem, and hybrid search is only well justified when the evaluation set shows this type of error dominates. Moreover, a vector index is essentially a document copy, requiring sync maintenance, rebuild costs, and permission consistency; permission gaps are silent and may involve compliance risk. The core methodology is to let error types drive technical decisions — only teams with a baseline can quantify the real incremental value embeddings bring.