Don't Rush to a Vector Database: First, Nail Down Your Retrieval Baseline

When building a RAG assistant, many teams reach for a vector database as step one. But Omar Baruzzo, in a blog post, proposes the opposite: don't touch vectors yet—use your existing full-text search and structured filters to establish a credible baseline, and let the evaluation data tell you exactly where vectors are needed.

First, Understand What Users Are Actually Asking

Baruzzo's first suggestion isn't about technology choices—it's about gathering real questions from the people who will actually use the assistant, not making them up yourself. Once you have those questions, categorize them. You'll often find that a large portion are essentially lookup queries: contract numbers, customer names, date ranges, material codes, a specific person's name.

The common thread here is the need for exact matches. Vector retrieval compares semantic similarity, and it performs poorly precisely on the identifiers that need to be "exactly the same." Keyword search and SQL filtering are the opposite: extremely precise on identifiers, but prone to missing results when the wording in the query doesn't match the wording in the documents.

The key insight: these capabilities are usually already sitting in your existing tech stack. No need to introduce a new storage system.

Full-Text Search vs. Structured Filtering: Full-text search builds an inverted index over document text, matching through keywords, stemming, and boolean combinations—implemented by Elasticsearch, PostgreSQL's tsvector, SQLite's FTS5, and others. Structured filtering, on the other hand, performs equality or range queries against existing columns in a relational database (such as contract numbers, customer IDs, dates, status fields)—essentially WHERE clauses. The two are often combined: first narrow the candidate set with SQL, then rank with full-text search.

Why Vector Retrieval Performs Poorly on Identifiers: Embedding models compress text into dense vectors. Semantically similar strings end up close together in vector space, but identifiers like "C-1024" and "C-1025" that differ by a single digit may be mapped to distant regions because they're semantically unrelated, or to arbitrary positions because the model never saw that combination during training. Similarity ranking has no constraint for exact equality—it lacks the database index's guarantee of "hit means equal"—making it unsuitable for primary-key-style lookups.

Building a Baseline with Fifty Questions

With real questions in hand, the next step is to write down, for about fifty of them, what the "correct answer should look like." Then run those questions through full-text search plus structured filtering and record how many return accurately.

That recall number is your baseline. Baruzzo emphasizes that anything you add afterward must beat this number—otherwise you can't call what vectors bring an "improvement," only a "change." Without a baseline, claims of vector retrieval's effectiveness become unfalsifiable.

The value of this approach is turning subjective impressions into comparable numbers. The assistant "feeling better" isn't an engineering conclusion; the delta above the baseline is.

The Engineering Significance of Evaluation Sets and Baselines: An evaluation set is a collection of questions tagged with "ground truth answers" or "expected hit documents," used to repeatedly measure system performance against a fixed standard. Recall measures the proportion of all relevant documents that should be retrieved that the system actually returns—one of the core metrics for retrieval systems. A baseline is the score of the simplest approach on the same evaluation set, serving as a reference point for later comparisons.

Without an evaluation set, "better" is just a feeling; without a baseline, no new approach's gains can be quantified, nor can you judge whether the cost is worth it. A set of fifty questions is large enough to cover the main query types, yet quick to re-run after every change (typically within minutes)—cheap enough to fold into your daily development loop. This is the concrete expression of the "measurability first" principle in RAG engineering.

Where Vectors Truly Belong

Vector retrieval has its home turf: the vocabulary gap. Baruzzo's example is vivid: a user asks "How do I terminate a lease before it expires," but the document says "early cancellation of a rental agreement." Same meaning, different words—keyword search fails in this scenario.

When the evaluation set clearly exposes this class of errors, hybrid search (keywords plus vectors, results fused) becomes a grounded choice. At that point, the benefits can be measured, and you're in a position to judge whether it's worth the cost. Note the precondition: the evaluation set exposed the problem first, rather than deploying vectors and then hunting for their benefits.

Vocabulary Gap and Hybrid Search: The vocabulary gap refers to missed recalls in keyword search caused by mismatches between query terms and document wording—typically synonyms, hypernyms/hyponyms, and differences between colloquial and formal language. Vector retrieval mitigates this through semantic similarity, but introduces new false positives.

Hybrid search means running keyword retrieval and vector retrieval simultaneously, then fusing the two result sets. Common fusion strategies include Reciprocal Rank Fusion (RRF) and weighted score combination. The value of hybrid search lies in using keywords to guarantee identifier precision and vectors to bridge the semantic gap—usually outperforming either strategy alone on an evaluation set—but it brings higher query latency and more complex tuning costs. It should therefore be triggered by evaluation data, not intuition.

The Real Cost of an Extra Store

A vector index is essentially a copy of your documents. Baruzzo lists the maintenance burden that every team preparing for production should take seriously:

• The copy must stay in sync with the source data
• When the embedding model changes, the entire index must be rebuilt
• The copy must enforce exactly the same access permissions as the original system

The last one is where things most easily go wrong. Someone who can't open a document in the original application shouldn't be able to get it through the assistant either. Permission errors on the copy are often silent and hard to diagnose—and when personal data is involved, they escalate into compliance issues.

Permission Consistency and Compliance Risk: A vector index typically exists as an independent storage system, with access control separate from the original system. This creates the risk of a silent leak—"the original app already restricts access, but the assistant can still recall the document." There are two remedies: either perform a permission check in the original system after retrieval (row-level security filtering), or sync permission metadata into the vector store and enforce filtering at query time. The former guarantees consistency but adds latency; the latter requires a sync mechanism and handling of permission change propagation.

When documents involve personal data, unauthorized recall may constitute a compliance issue under data protection regulations (such as GDPR or PIPL). These problems don't raise errors immediately—they typically surface during audits or user complaints. That's why they must be designed in at the architecture stage, not patched after launch.

A Question of Sequencing, Not a Route Debate

Baruzzo states clearly that these arguments are not against vector retrieval. What he opposes is the sequencing: vectors being treated as the starting point, when they're better suited as a patch for where the baseline fails.

This distinction becomes concrete when models or indexes are upgraded. Teams with a baseline can re-run the question set and judge whether a migration is worth it; teams without one can only say the assistant "seems better now."

His advice lands cleanly: start with boring retrieval, measure it, and let the errors tell you what to add.

In your own assistant, how many questions are lookups in disguise?

One-line Summary

Don't rush to a vector database—use real questions and a baseline to measure where vectors are needed.

Paragraph Summary

Omar Baruzzo argues that when building a RAG assistant, you shouldn't rush to integrate a vector database. Instead, gather and categorize real user questions first—you'll often find that many are essentially precise lookups (IDs, names, dates) better handled by existing full-text search plus structured filtering. Build a measurable baseline with about fifty questions; anything you add afterward must beat that number, or it's just a "change," not an "improvement." Vector retrieval's value concentrates in vocabulary gap scenarios—when query wording and document wording diverge—which is when hybrid search becomes grounded. Meanwhile, a vector index as a document copy introduces three hidden costs: synchronization, rebuilding, and permission consistency, with permission issues becoming compliance risks when personal data is involved. The core claim is about sequencing: vectors should be a patch, not a starting point.