Why RAG Applications Fail: A Development Guide to Chunking, Retrieval, and Evaluation Loops

When a RAG application gives a wrong answer, most people's first instinct is to swap the model. But the real root cause is often hidden before the model ever sees the question — document parsing errors, chunking that cuts through semantics, retrieval that misses key passages, or users seeing text they shouldn't.

The solution is a closed loop: write a golden question set first, change one stage at a time, evaluate retrieval and generation separately, and gate every release with scores.

Why Does RAG Already Fail Before the Model Even Shows Up?

Barnett et al. listed seven failure points across three RAG case studies in research, education, and biomedicine: missing content, relevant documents not ranked high enough, relevant documents not making it into context, the model failing to extract the answer, wrong output format, inappropriate specificity, and incomplete answers. Three of these failure points occur before the model sees a single token.

Context itself has a cost. The "Lost in the Middle" study found that accuracy is highest when relevant passages are at the beginning or end of the input, and drops noticeably in the middle — even for models optimized for long context. Researchers also found that swapping 20 retrieved documents for 50 yields very limited improvement — piling on more content is no substitute for better retrieval.

In a VentureBeat reader survey published in October 2026, 64% of 130 organizations said at least one confident but wrong AI answer could be traced back to missing or inconsistent business context in their own data. The sample is small with clear self-selection bias, but the direction aligns with the failure points above.

The Seven-Step RAG Pipeline

The code below is illustrative. Defaults and library versions change, so adapt and test against your own documents.

Step 1: Write the Golden Question Set First

A golden set is a labeled list of questions paired with the documents needed to answer them. Qdrant's evaluation guide describes three ways to build one: manual annotation, extracting real user queries from logs, and generating questions from sampled documents with an LLM. The guide also warns that synthetic questions tend to be simpler than real ones, inflating scores — a few hundred question-answer pairs are enough to reveal large metric differences.

Start with 50 to 100 real questions, add the odd corner cases users actually ask, and include cases that must return empty results (unanswerable questions, or documents the asking user has no right to read). Version this file alongside your code and record the corpus snapshot it was built against, because once the corpus changes, old labels go stale.

Step 2: Parse and Clean Documents

Most retrieval problems start here. Amazon Bedrock Knowledge Bases' default parser can only read text from files, and AWS recommends advanced parsers when documents contain charts, tables, or images. Open-source tools like Unstructured can split documents into elements such as headings, narrative text, list items, and tables, letting you preserve structure instead of flattening everything into one string.

Remove templated content like repeated headers and footers, keep only one current version per document, and store document ID, owner, date, and access permissions. Handle near-duplicates too — Qdrant lists them as one reason evaluation labels may look incomplete.

Step 3: Chunking with Metadata

Chunking is only necessary when documents exceed the embedding model's input limit, but it also helps when a single vector represents a long page poorly. Microsoft's guide suggests starting at 512 tokens with 25% overlap, while Pinecone suggests starting with fixed-size chunking and iterating once you've measured that it's not good enough.

Defaults vary widely across tools, and units aren't even consistent: LangChain's base text splitter defaults to 4000 characters with 200 overlap in source; LlamaIndex's constant file uses 1024 tokens with 20 overlap; Amazon Bedrock defaults to about 300 tokens; OpenAI's automatic vector store strategy uses 800 tokens with 400 overlap. Treat all of these as starting points for testing, not answers.

The splitter below cuts on headings first so a chunk never spans two topics, then makes sure each chunk retains its heading. To properly compare chunk sizes, Pinecone's recommendation is to index each candidate size separately, run real queries, and compare results.

Step 4: Embedding and Indexing

Embeddings from different models aren't comparable, so store the model name and dimension alongside each chunk. The example below uses Postgres with the pgvector extension, which by default performs exact nearest-neighbor search with perfect recall. Approximate indexes like HNSW trade some recall for speed — only create one when query volume demands it, and measure how much recall you lose against exact search.

1536 dimensions corresponds to OpenAI text-embedding-3-small's default output length; change it to your own model's dimension. The pgvector README lists HNSW defaults: m is 16, ef_construction is 64, hnsw.ef_search is 40 — higher values give better recall but slower speed.

Step 5: Hybrid Retrieval, Filtering, and Reranking

Keyword search is better at product codes, jargon, dates, and names, while vector search finds conceptually similar content with no shared vocabulary. Microsoft's hybrid search documentation describes running both in parallel, then merging the ranked lists with Reciprocal Rank Fusion (RRF), summing 1/(rank + k) for each result, with k set to 60 from the original paper.

Two details matter. First, pgvector applies filters after the index scan, so approximate queries with filters may return fewer rows than the limit; version 0.8.0 added iterative index scans to scan more of the index when needed, and indexing the filtered columns helps too. Second, take the top 20 or so candidates, rerank them with a cross-encoder or a hosted reranking model, then send the best 3 to 8 to the model. Pinecone calls this two-stage pattern one of the simplest ways to improve retrieval quality in a RAG pipeline.

Reciprocal Rank Fusion (RRF) is a method for merging multiple retrieval result lists without any tuning. In hybrid retrieval, vector search and keyword search each return a relevance-ranked document list, but their score scales are completely different and can't be compared or added directly. RRF's solution is to discard raw scores and look only at ranks: for each document, compute 1 / (k + rank) in each result list, then sum them to get a fused score, where rank is the document's position in that list (starting from 1) and k is a smoothing constant set to 60 in the original paper. This way, documents appearing near the top of both lists score highest, while documents ranking high in only one list still get some weight. RRF's advantage is that it's insensitive to score distributions and needs no normalization, which is why it's widely used as the default fusion strategy in production systems.

Step 6: Generation with Citations and Constraints

Only give the model allowed passages, label each with its source, and tell it what to do when it can't find an answer. The refusal path is part of the product, because the golden set should include questions the corpus can't answer.

The last line of the system prompt matters, because OWASP lists indirect prompt injection (external source content changing model behavior) as a risk and notes that RAG doesn't fully mitigate it.

Indirect Prompt Injection is when an attacker hides malicious instructions in external content the model will read — in RAG scenarios, that means retrieved documents, web pages, or database fields. Unlike a user directly typing a malicious prompt, indirect injection doesn't require the attacker to touch the conversation interface; it takes effect as soon as their content is retrieved into context. Common attack goals include making the model ignore its original system prompt, leaking the system prompt or other users' sensitive data, calling tools it shouldn't call, or inserting phishing links into responses. OWASP lists this as LLM01 and explicitly states that RAG cannot fully mitigate this risk, because retrieval itself means the model processes text from untrusted sources. Practical mitigations include: isolating external content from system instructions, format-validating model output, requiring human approval for high-risk actions, and limiting the tools the model can call.

Step 7: Evaluation and Monitoring

Evaluate retrieval and generation separately, because wrong answers can come from either end. On the retrieval side, recall@k is the proportion of known relevant documents appearing in the top k results, and MRR (Mean Reciprocal Rank) averages the reciprocal of the first relevant result's rank. Qdrant recommends recall@k for RAG pipelines, because completely missing a relevant passage hurts more than ranking it low.

The specific threshold numbers are placeholders. Set your own thresholds with a baseline run, then raise them over time. For answer quality, the RAGAS library defines Faithfulness (the share of claims in the answer supported by the retrieved context), Context Precision (the retriever's ability to rank relevant chunks ahead of irrelevant ones), Context Recall (how well the retrieved context covers the reference answer — this always requires a reference answer), and Response Relevancy (formerly answer relevancy, which doesn't check factual accuracy). Run retrieval checks on every CI change, and sample production questions into the golden set over time.

RAGAS is an open-source evaluation framework for RAG pipelines that breaks "is the answer good" into several independently measurable metrics, avoiding gut-feel scoring. Faithfulness measures whether every claim in the answer can be supported by the retrieved context, effectively catching model "fabrication." Context Precision evaluates whether the retriever ranks truly relevant chunks ahead of irrelevant ones. Context Recall measures how well the retrieved context covers the reference answer, so it requires a reference answer to compute. Response Relevancy judges whether the answer is on-topic, but it doesn't check factual accuracy — an on-topic but completely wrong answer can still score high. These metrics must be used in combination to pinpoint whether the problem is on the retrieval side or the generation side.

Permissions and Security

Permissions belong in the retrieval layer, not the prompt. Microsoft's Azure AI Search guide describes document-level authorization as applying a security filter to every query, and Supabase recommends Postgres row-level security over WHERE-clause filtering because policies are always applied — though be sure to measure the latency cost. The schema in Step 5 does the same thing with an allowed-groups array. The principle is simple: if a piece of text never leaves the database and never reaches a user who isn't authorized to open the source document, the model can't possibly repeat it.

Four items in the OWASP Top 10 for LLM Applications (2025) map directly to RAG applications:

• LLM01 Prompt Injection: Retrieved text is untrusted input. Isolate external content, validate output formats, and require human approval for high-risk actions.
• LLM02 Sensitive Information Disclosure: Limit access to sensitive data on a least-privilege basis.
• LLM08 Vector and Embedding Weaknesses: OWASP recommends fine-grained access control and permission-aware vector and embedding stores, warns that context can leak across users in multi-tenant environments, and notes that embeddings can sometimes be reversed to recover source information.
• LLM10 Unbounded Consumption: Rate-limit and set quotas to prevent public endpoints from being used to drive up costs.

Finally, plan for change. MongoDB and Voyage AI's documentation recommends regenerating embeddings for the entire corpus when migrating to a successor model, even if the dimensions are the same, because vectors from different models aren't comparable. Also make sure deletions propagate to the index, not just additions.

Row-Level Security (RLS) is an access control mechanism provided by databases like Postgres that lets you define policies at the table level so different users or roles only see rows matching their permissions. Compared to filtering with WHERE clauses at the application layer, RLS's advantage is that policies are enforced by the database — even if the application layer forgets a filter condition, the database won't return unauthorized data, fundamentally eliminating "forgot to add the condition" errors. In a RAG architecture, modeling document permissions as RLS policies means retrieval queries naturally return only documents the current user is authorized to access, and the model can't obtain text it shouldn't see. Note that RLS adds overhead during query execution, especially when combined with vector indexes for approximate nearest-neighbor search, where the interaction between filter conditions and index scans can affect recall and latency — so measure query performance and recall loss before shipping.

Common Mistakes That Ruin RAG Quality

• No test set: Without one, every adjustment is a guess. Barnett et al. found that validation can only happen during operation, so keep adding real questions.
• Copying default chunking settings: The defaults above differ in both size and units. Test two or three sizes for your questions.
• Flattening tables and scans: If the parser drops a table, no retriever can find it.
• Filtering after generation: Checking permissions on the answer is too late. Filter before candidates enter the prompt.
• Mixing embedding models: Query and document vectors must come from the same model. Store the model name next to each chunk.
• Cramming content into the prompt: More chunks can bury the relevant one, as the Lost in the Middle results have shown. Send fewer after reranking.
• Letting the index go stale: Sync regularly, propagate deletions, and track the age of the oldest indexed version of each document.
• Treating RAG as injection defense: It isn't. Keep minimum privilege for every tool the model can call.

RAG Development Vendors: 5 Teams Worth a Look

If you'd rather hand the pipeline to a specialist team, the following companies include RAG development or AI data engineering in their service lists. The data comes from each company's own website and is self-reported — confirm directly before signing.

Dev Technosys

Founded in 2010, 250+ team members, headquartered in Jaipur, India, holding ISO 27001:2022 and ISO 9001:2015 certifications, CMMI Level 3 appraised. Its five-stage RAG and data engineering process is: data discovery, pipeline design, build and integration, evaluation and hardening, and launch and operations. Core services include custom RAG solutions, RAG pipeline development, vector database integration (Pinecone, Weaviate, pgvector, Milvus), AI knowledge base development, LLM data engineering, and RAG integration with SharePoint, Confluence, Salesforce, Zendesk, and internal APIs.

Its process matches the closed-loop order in this guide — data discovery and pipeline design first, then connecting the model, an evaluation and hardening stage before launch, and ongoing operations after. For teams that want one vendor to own the pipeline, index, and testing, that continuity is a real advantage.

Excellent Webworld

Lists Dubai as headquarters, claims founding in 2011, cites 300+ engineers on its homepage but 350+ on its RAG page. Its RAG page describes semantic chunking for prose, structure-aware chunking for tables and code, hybrid search with reranking, query rewriting and HyDE, and checking permissions before retrieval rather than filtering afterward. The page says it tests retrieval precision, recall, and groundedness on real query sets but doesn't name the evaluation tools. Compliance frameworks like SOC 2 Type II and ISO/IEC 27001 mentioned on its page are "supported by the work," not certifications the company holds. Suited for Gulf region and global buyers.

SCAND

Claims operation since 2000, with offices in New York and Warsaw, 250+ full-time engineers. Its RAG page describes intelligent chunking, hybrid search, cross-encoder reranking, and HyDE, supporting Pinecone, Weaviate, pgvector, Qdrant, Milvus, and Elasticsearch, with cloud or on-premises deployment. It describes role-based access, audit logs, and data isolation, and names LangSmith and Langfuse as monitoring tools but doesn't state evaluation metrics. Claims ISO 9001 certification (a quality, not security, standard), listing GDPR, HIPAA, and SOC 2 only as frameworks its architecture can align with. Suited for buyers wanting on-premises options.

Uvik Software

Founded in 2015, lists two addresses in Tallinn (labeled HQ) and Ipswich (labeled commercial HQ), 50+ senior engineers. Has separate RAG and AI data engineering pages. The RAG page describes hybrid BM25 and vector search with cross-encoder reranking, query-time permission enforcement, and an evaluation framework that assesses context recall, context precision, faithfulness, answer correctness, and citation accuracy on every change. Security is described as "aligned" with ISO/IEC 27001 and SOC 2, with formal proof available on request. Suited for UK and EU buyers wanting an evaluation-first, small senior team.

Kanerika

Claims to have started in 2015, with six offices across the US, India, and Singapore, no headcount published. Its RAG page describes chunking, hybrid search, and reranking, scoring every release with a real-question test set before users see it, and monitoring for drift after launch. It describes role-based access at the retrieval layer, logging on every retrieval, and PII masking before ingestion, supporting cloud or on-premises deployment. Its footer has a CMMI Level 3 mark and lists partnerships with Microsoft, Snowflake, and others, but doesn't claim ISO 27001 or SOC 2. Suited for teams wanting RAG inside a governed data stack on Microsoft Fabric, Azure, or Snowflake.

FAQ

What is RAG? Retrieval-augmented generation supplies relevant passages from your own documents when the model answers, making responses grounded and citable.

How much does RAG development cost? It depends on the volume and quality of sources, the permission model, evaluation depth, and hosting. Dev Technosys says its projects start at $10,000. Most other companies don't publish pricing.

What chunk size should I use? There's no universal answer. Microsoft's suggested starting point is 512 tokens with 25% overlap, but library defaults vary widely, so build two or three variants and compare recall@k on your golden set.

Is RAG better than putting all documents in the prompt? Not by default. Retrieval suits large, frequently changing, or permission-differentiated corpora, while small stable document sets may fit in long context. Test both with the same question set, and remember that long inputs have their own accuracy problems.

One-line Summary

RAG failures usually stem from pre-retrieval document parsing, chunking, and permission filtering — gate them with a golden question set and staged evaluation loops.

Paragraph Summary

The article argues that when RAG applications give wrong answers, the problem often isn't the model itself but pre-model stages: document parsing, chunking that cuts through semantics, retrieval missing key passages, or permission filtering failing. The author lays out a seven-step pipeline: build a version-controlled golden question set, then parse and clean documents, chunk with metadata, embed and index, hybrid-retrieve and rerank, generate with citations, and finally evaluate retrieval and generation separately. The piece stresses that hybrid retrieval with RRF fusion and cross-encoder reranking is one of the most effective ways to improve retrieval quality, and that permission filtering must happen at the retrieval layer, not after generation. It also maps four items from the OWASP LLM Top 10 directly to RAG risks, lists common mistakes and five vendors, with the core methodology being to gate releases with scores.