Why I Abandoned Answer Generation After Building a RAG Search Engine for 10,000 Documents

Spend a few minutes browsing GitHub and you'll almost certainly stumble onto yet another "AI-powered knowledge base" — import PDFs, stuff them into a vector database, then confidently answer every question. RAG (Retrieval-Augmented Generation) looks like it's become so accessible that anyone can do it: chunk the text, convert to embeddings, plug in an LLM, done.

Until you fire off your first query and the results leave you completely baffled.

A developer recently shared his experience building the Dalil project for a client. It's a search engine for Russian-language religious documents, with a clear goal: let users ask questions in natural Russian and find relevant religious rulings from thousands of documents — not through keyword matching, but through semantic understanding. Sounds like just another standard RAG project, but during development he discovered there's a massive chasm between "being able to find things" and "search results you can trust."

From Data Cleaning to Structure-Aware Chunking

The project's dataset consisted of religious rulings in JSON format, each containing a title, question, answer, footnotes, original source links, and category information. The developer first did something unglamorous but important: converting the JSON to JSONL format, so records could be processed one at a time rather than loading everything into memory at once.

The data contained both Russian and Arabic, and some records had encoding issues. So he added a text validation step: clearing control characters, handling anomalous Unicode sequences, and outputting unprocessable records to a separate file to avoid disrupting the overall pipeline.

The chunking stage used an interesting strategy. Most RAG tutorials suggest splitting by character count or token count, but Dalil's records already had clear structure. The developer chose to chunk by semantic structure: each record was split into two chunks — the first containing the title and question, the second containing the answer and footnotes. During retrieval, both chunks could be re-associated through the original record ID, so users wouldn't get isolated passages stripped of context.

Embeddings, Retrieval — and the Trap of Semantic Similarity

The indexing stage used OpenAI's text-embedding-3-large model, generating 3072-dimensional vectors stored in Qdrant. Each chunk had its own vector, text, and metadata, with point IDs deterministically generated from the original record ID and chunk number — no need to regenerate identifiers when re-importing the same batch of data. The entire pipeline was: JSON → JSONL → Chunks → Embeddings → Qdrant.

The retrieval side did the reverse: the user enters a question, the system converts it to an embedding, and Qdrant returns the nearest vectors. Early results looked good — the system could find documents that used completely different wording but discussed similar topics.

And that's exactly where the problem lay: semantic similarity doesn't mean a document can answer the question. Two passages might discuss the same religious topic, use the same terminology, describe similar behaviors — yet differ completely in specific circumstances, conditions, or subjects of the ruling. In an ordinary search engine, this is merely annoying; but in contexts involving religious or legal rulings, the consequences can be far more serious.

Why No LLM Was Connected for Answer Generation

At this point, the obvious next step in the standard RAG pipeline is to connect a language model, feed it the retrieved documents, and generate a beautifully worded answer with citations.

But the developer deliberately abandoned the generation stage.

The reason: an LLM can produce a seemingly perfect summary while ignoring an important condition, or cobble together several different rulings into one answer. When the tone of the response is confident enough, users may not even notice that the original meaning has been altered.

So the Dalil system doesn't generate answers to user questions. It only does three things: retrieve existing documents, rank them, and return the original sources.

This project, which started as RAG, essentially became a semantic search engine with hybrid ranking. Technically speaking, "semantic search" is a much more accurate description than "RAG."

Vector Search Isn't Enough: Lessons from Hybrid Retrieval and RRF

The initial implementation relied on dense vectors: queries were converted to embeddings, and Qdrant used cosine similarity to find the nearest documents. Fine for a prototype, but in specialized corpora, certain words and concepts matter not just for their general meaning but for their precise terminological expression. Vector models excel at capturing semantic similarity, but sometimes they "understand too freely"; keyword search can pinpoint exact wording but struggles when users phrase questions differently.

Eventually he moved to hybrid retrieval: dense retrieval (semantic search with OpenAI embeddings) plus sparse retrieval (lexical search using BM25 via FastEmbed), then merging results with RRF (Reciprocal Rank Fusion).

Then an interesting problem surfaced: he had been using RRF scores as a relevance indicator. But RRF measures a document's position in the merged ranking, not the probability that it actually answers the query. The developer used a vivid analogy — imagine a swimming race where none of the three competitors can swim, but one of them will still come in first. That doesn't make them an Olympic champion.

So he had to separate "ranking" from "relevance verification." He first introduced an additional cosine similarity check, with a threshold of 0.32 for regular queries and 0.20 for short queries (since one- or two-word queries often produce lower similarity scores, and overly strict filtering would kill useful results). But filtering alone wasn't enough — reranking was also needed.

BM25 is a classic sparse retrieval algorithm, essentially an improved version of TF-IDF. It scores documents based on term frequency (TF) and inverse document frequency (IDF), normalizes for document length, and uses a saturation function to dampen the unlimited weighting of high-frequency terms. BM25's strength is its sensitivity to exact terminology: if a user's query contains a specialized term, BM25 will almost certainly rank documents containing that term at the top. The trade-off is that it doesn't understand synonyms or alternative phrasings — a query like "how to dissolve a marriage" and a document saying "determination of marital invalidity" appear completely unrelated to BM25. Precisely because dense vectors excel at semantic generalization and BM25 excels at exact term matching, combining the two creates complementarity — which is why hybrid retrieval often significantly outperforms single approaches on specialized corpora.

RRF (Reciprocal Rank Fusion) is an algorithm that merges multiple retrieval result sets into a single ranking. The formula assigns each document a score equal to the sum of 1/(k + rank) across all retrievers, with k typically set to 60. It only cares about a document's rank in each result set, not the raw similarity scores, which naturally sidesteps the problem of "dense retrieval cosine scores and BM25 scores being on different scales and impossible to weight directly." The trade-off is that RRF output is only a relative ranking — it doesn't represent the probability that a document actually answers the query. The top-ranked document might just be "the tallest of the short." This corresponds exactly to the "three swimmers who can't swim, but one still comes in first" analogy from the article.

Reranking and Graceful Degradation

For the reranking stage, he brought in the Cohere Rerank API. Unlike embedding models that independently encode queries and documents into vectors, rerankers evaluate the relationship between a query and each candidate more directly, taking the query and candidate text and producing a relevance score.

The full architecture became: Query → Normalization → Dense/Sparse Retrieval → RRF → Cohere Rerank → Results.

After reranking, there was one more threshold — if the best candidate's score was too low, the system returned an empty result set. The developer emphasized: an empty result isn't necessarily an error. Sometimes the correct behavior for a search engine is to admit it can't find a suitable document — especially compared to the alternative of "confidently showing irrelevant content."

Bringing in an external API also introduced new failure points. Cohere might slow down, return errors, or stop responding entirely. The developer set a 3-second timeout for reranking and implemented a circuit breaker — after consecutive failures, it would temporarily stop calling the problematic service, then try to recover later. If reranking was unavailable, the system could still return Qdrant hybrid retrieval results. Reranking caching and independent status tracking (success, timeout, error, circuit breaker open, skipped) were also added to the monitoring system.

The fundamental difference between a reranker and an embedding model lies in architecture: embedding models encode queries and documents independently into vectors, comparing them only at the end via dot product or cosine similarity — they never "meet" during the encoding stage. Rerankers, on the other hand, are typically cross-encoders that concatenate the query and candidate document and feed them into the model together, allowing attention mechanisms to interact directly between them and capture fine-grained semantic relationships. This yields higher accuracy but means document representations can't be precomputed — every query-document pair requires real-time inference. So it's only suitable for reranking a small number of candidates (like top 50), not for full-corpus retrieval. Cohere Rerank is a representative of this kind of managed service.

A circuit breaker is a fault-tolerance pattern from distributed systems, popularized by Martin Fowler and others. It maintains a state machine: normally in a closed state, requests pass through; when consecutive failures exceed a threshold, it switches to an open state, directly rejecting requests for a period without actually calling the downstream service, preventing thread exhaustion or cascading failures; after a cooldown period, it enters a half-open state, allowing a small number of requests through to test — success closes it, failure keeps it open. The article's 3-second timeout and circuit breaker for Cohere were precisely to prevent external API flakiness from cascading into total search unavailability.

From a Few Thousand to Over 10,000 Documents: A Dataset Migration

As the project progressed, the corpus expanded from a few thousand records to over 10,000 documents. On the surface, this was just a matter of regenerating embeddings and uploading to Qdrant, but by then the project had public documentation pages, categories, internal links, and persistent URLs, with data spread across two systems: Qdrant handled the search index, while PostgreSQL stored the canonical records used by public pages and the application.

What looked like a simple data update turned into a full migration. The developer created a new records_ru_v2 collection, kept the old collection as a fallback, preserved existing slugs, assigned additional URLs to new documents, and prepared a rollback mechanism based on the old Qdrant collection and PostgreSQL backups.

The Unexpected Value of a Dictionary and a Few Lines of Python

Users don't ask questions in the format of a neatly organized dataset. They make typos, use informal expressions, and choose different words for the same concept. So a normalization stage was added before retrieval: cleaning extra whitespace, using a religious synonym dictionary, lemmatization via pymorphy3, and fuzzy matching with rapidfuzz.

The developer summarized a lesson here: not every search problem needs another neural network. Sometimes a decent dictionary and a few lines of Python are enough — not impressive on paper, but it works.

pymorphy3 is a maintained fork of the Russian morphological analysis library pymorphy2, used for lemmatization and morphological tagging of Russian words. Russian is a highly inflected language — a noun has six cases and twelve singular/plural forms, and verbs have tense, aspect, person, and other variations. If you only match on the base form, the query "намаз" and the document's "намаза" or "намазу" would be treated as three different words. Lemmatization reduces each word to its dictionary form, aligning retrieval at the lexical level. Rapidfuzz, meanwhile, is a fast fuzzy string matching library based on algorithms like Levenshtein edit distance, used to tolerate spelling errors in user input — for example, still matching candidates when a transliterated Arabic word is misspelled by a letter or two. These two libraries, combined with a synonym dictionary, form a very low-cost but highly effective query normalization layer.

Measuring Search Quality: An Underrated Step

The more complex the system, the more insufficient it is to only log queries and top results. When users get empty results, you need to understand why: is there really no relevant document? Did dense similarity filtering eliminate candidates? Did the reranker give low scores? Or did Cohere not respond in time?

Search analytics were expanded to record normalized queries, cosine similarities, reranking scores, reranking execution status, and more. One key change was to stop treating RRF's best_score as a quality metric — ranking and relevance are two different things, and confusing them means you could celebrate beautiful numbers for a long time while users keep finding the wrong documents.

He also introduced an independent offline evaluation mechanism, using a set of test queries to validate the retrieval pipeline. This may be one of the most important parts of the entire development process: replacing an embedding model or adding a ranking stage is easy — the hard part is objectively answering a basic question: did search actually get better?

Key Takeaways

Dalil's final search system uses a corpus of over 10,000 documents, structure-aware chunking, dual dense and lexical retrieval in Qdrant, RRF result fusion, additional relevance filtering, and Cohere reranking. The application itself runs on FastAPI, PostgreSQL, Next.js, and a Telegram Mini App.

The developer's most important conclusion: a good search engine needs to know not only how to find answers, but also when to return empty results. You can make an LLM sound extremely persuasive, but search is a different challenge — it doesn't need to sound persuasive, it needs to be accurate. And that took far more development time than the original RAG prototype.