Omar Baruzzo

The method for establishing a baseline is: select about fifty questions from real questions and write out the standard answers, run them through full-text search plus structured filtering and record the number of correct results, and that number is the baseline.

After collecting real user questions and categorizing and counting them, it was found that a considerable proportion of the questions are directed lookups, such as contract numbers, customer names, date ranges, material codes, and person names.

A vector index is essentially a copy of the documents, must be kept in sync with the data source, requires rebuilding the entire index when the embedding model is changed, and must enforce the same access permission controls as the original system.

Without a baseline, adding vector search can only be called a "change", not an "improvement"; any newly added technical solution must be able to beat the baseline to count as an improvement.

Baruzzo's argument is not against vector search itself, but about the order of doing things: first establish a measurable baseline, and let error types drive technical decisions.

When building a RAG assistant, teams usually deploy a vector database as their first reaction, but Omar Baruzzo suggests first laying a baseline with existing search capabilities and letting evaluation data decide when embeddings are truly needed.

Baruzzo explicitly states that these arguments are not against vector retrieval, but against the order of treating vector retrieval as the starting point; vectors are more suitable as a patch for when the baseline fails.

When building a RAG assistant, one should first establish an evaluation baseline using existing full-text search and structured filtering, then decide based on evaluation data whether to introduce vector retrieval, rather than plugging in a vector database from the start.

The method for establishing a baseline is to write down the expected hit answers for about fifty real questions, then run them with full-text retrieval plus structured filtering, and record the number of questions returned correctly as the baseline.

Any retrieval solution added afterward must be able to exceed the baseline number, otherwise it cannot be called an improvement, only a change.

The applicable scenario for vector retrieval is the vocabulary gap, that is, when the user's query wording differs from the document's wording, causing keyword retrieval to miss recalls, for example when the user asks "how to terminate the lease before expiry" while the document says "early termination of the rental contract".

Introducing a vector index adds maintenance burden: the copy must be kept in sync with the source data; the entire index must be rebuilt when the embedding model is changed; and the copy must enforce exactly the same access permissions as the original system.