RAG Accuracy Still Low? Try Metadata Pre-Filtering Before Tweaking Parameters

Stop Obsessing Over Chunk Size and Embedding Models

When building RAG (Retrieval-Augmented Generation) systems, many developers fall into the same trap: the moment the LLM starts hallucinating or retrieving the wrong context, their first instinct is to treat it as a math problem — and start endlessly tuning the vector layer.

One Reddit developer shared their full journey through this mistake. They spent weeks swapping out embedding models, adjusting chunk size from 512 to 1024, tweaking chunk overlap, and experimenting with MMR (Maximal Marginal Relevance) search. The result? "None of it really moved the needle." The model kept pulling outdated information, even mixing up 2021 company policies with 2024 ones.

This story resonates with a huge number of RAG practitioners: we tend to blame retrieval quality on insufficient vector similarity, while overlooking a more fundamental layer — how the data itself is organized and filtered.

The Real Fix: Metadata Pre-Filtering

The answer this developer eventually found was surprisingly simple — properly label your source data, and apply metadata pre-filtering before retrieval.

Their approach introduced a "router" step:

• First, have the LLM identify user intent and extract structured filter conditions from the query — for example, {"year": "2024", "doc_type": "HR_Policy"}
• Apply these conditions as hard filters against the vector database before running semantic similarity search
• Only documents that pass the filters are eligible for the subsequent vector retrieval step

The result: "Overnight, hallucinations dropped to almost zero." The reason is straightforward — the system physically prevents the LLM from ever seeing the wrong documents.

The key shift here is in processing order. The traditional approach is "do semantic retrieval first, then consider relevance." Pre-filtering flips this to "use business logic to narrow the candidate set first, then let vector search do its job within the right scope."

Metadata pre-filtering has native support in all major vector databases. Pinecone, Weaviate, and Qdrant, for instance, all allow structured filter conditions to be attached at query time. The database first uses an inverted index or bitmap index to narrow down matching documents, then performs vector similarity computation within that subset. This "filter first, then search" execution order (pre-filtering) produces significantly better results than "search first, then filter" (post-filtering) — the latter can lead to poor recall because many candidates get discarded after the fact, while the former guarantees that the retrieval scope is business-correct from the start. One important caveat: metadata fields need to be explicitly declared as indexable at ingestion time, otherwise filtering degrades to a full scan and hurts performance. This makes planning your metadata schema during the ingestion pipeline design phase a critical prerequisite for this approach to work in practice.

Why Vector Search Is "Blind" to Business Logic

One line from the original post cuts to the heart of the matter: "Vector search is amazing, but it's completely blind to business logic."

Vector search is fundamentally about semantic similarity. When you ask "What is the vacation policy for 2024?", a 2021 vacation policy document is semantically very similar to your query — both discuss vacation, both are policy documents, and the wording and sentence structure are nearly identical. The vector model has no mechanism to distinguish the hard business-level difference of "which year," because the difference between years may be negligible in semantic vector space.

In other words, when you keep optimizing your embedding model and chunking strategy, you're improving your ability to "find the most semantically similar content." But when the real problem is a temporal mismatch or document type confusion, even the most precise semantic matching won't help — because it's not a similarity problem to begin with.

This explains why parameter tuning "never seems to work." The direction was wrong from the start.

This limitation stems from the nature of vector embeddings: they compress the semantic content of text into a single point in high-dimensional space, where semantically similar texts are close together. But "semantically similar" and "business-relevant" are two entirely different dimensions. The distance between "2021 vacation policy" and "2024 vacation policy" in semantic vector space may actually be much smaller than the distance between "2024 vacation policy" and "2024 travel reimbursement policy" — because the former pair shares nearly identical vocabulary and sentence structure, and year references carry very low weight in a continuous semantic representation. The MMR (Maximal Marginal Relevance) algorithm tries to balance relevance and diversity in results to reduce redundancy, but it also operates purely at the semantic similarity level and cannot recognize business constraints like "this document doesn't match the temporal intent of the query." This explains why MMR also failed to help the developer in this story.

Practical Takeaways for RAG Engineering

This case offers several actionable insights for teams building RAG systems:

Invest first in structured metadata annotation. During ingestion, attach as much metadata as possible to each document chunk: year, document type, department, version, source, etc. The value of these labels often far exceeds another round of chunk size tuning.

Build intent routing into the retrieval pipeline. Having the LLM perform a lightweight intent classification and filter extraction step before retrieval is a low-cost, high-return architectural improvement. It acts as a business logic gate in front of vector retrieval.

Match the tool to the problem type. Semantically ambiguous questions that require understanding meaning belong to vector search. Questions with explicit hard constraints (time, category, permissions) should be handled by filters first. The two are complementary, not interchangeable.

It's worth noting that "80% of accuracy problems can be solved with metadata filtering" is the original author's estimate based on personal experience, not a statistically rigorous finding. In different business contexts, chunking strategy and embedding quality still matter — especially when documents lack metadata dimensions that can be meaningfully labeled. This insight is better understood as a prioritization reminder: before diving into parameter tuning, first check whether your data organization and filtering logic are sound.

Conclusion

This Reddit post resonated so widely because it reveals a common cognitive bias: engineers tend to attribute system problems to algorithm and model issues, because that's the domain we're most familiar with — it feels more "technical." But often, the most effective lever is in the unglamorous fundamentals — like labeling data and adding filter conditions.

The next time your RAG system starts hallucinating, maybe you don't need to rush to swap out the model. Ask yourself first: should it even have been allowed to see that document?