More Documents, Worse AI? A Benchmark Test of RAG Capabilities Across Open-Source and Paid LLMs

Companies hand a pile of internal documents to an AI and ask: "How many vacation days do I have?" or "Who needs to sign off on this purchase?" Developers call this RAG, but the concept is straightforward — the AI reads the materials you provide and answers from them. A long-standing concern is that giving the AI more documents actually degrades the answers: the right page gets buried, an outdated copy gets selected by mistake, or the AI simply makes something up.

A researcher built a test suite for a Kaggle benchmark challenge specifically designed to probe this assumption. The conclusion was surprising: a "document pile" that maxes out the context window isn't the real risk. What actually breaks smaller models is cross-document reasoning.

How the Test Was Designed

The author invented a fictional company and generated 50 short internal documents: leave policies, travel regulations, vendor contracts, product descriptions, and more. Because the company is made up, no model could have encountered this content during training, eliminating the possibility of models "recalling memorized answers." The author then posed 90 questions to each model, each accompanied by 2 to 50 documents. Models had to do two things: provide an answer and identify which document it came from. If the answer simply wasn't in the pile, the model had to respond with "not found."

Four variables changed from question to question:

• Document pile size: 2, 5, 10, 25, or 50 documents — to test whether "more is worse" holds up.
• Position of the correct document: top, middle, or bottom — to check whether AI "skims" the middle.
• Question type: single-step lookup, two-step reasoning, or unanswerable.
• Distractor documents: irrelevant ones, or ones highly similar to the correct document.

Two design choices deserve special mention. The first is outdated copies: 10 documents had older versions with identical phrasing but different numbers. Hard questions placed both the old and current versions in the same document pile — exactly what a real shared drive looks like. The second is two-step questions: each document ended with a line like "For unresolved questions about this document, escalate to the person who approved the Marrowby Studios annual leave policy." The question would ask, "Who is that person?" To answer correctly, the model had to jump to a second document (the annual leave policy) and find the approver's name. The answer only existed in that second document.

RAG (Retrieval-Augmented Generation) is currently the most common technical approach for enterprise AI Q&A systems. Its core logic: instead of relying on knowledge memorized during training, the system first retrieves relevant passages from a document library at query time, then inserts those passages into the prompt so the model answers solely from the provided materials. The advantage is that enterprise data doesn't require model retraining — updating documents is equivalent to updating the knowledge base. "Lost in the Middle" is an influential early research finding in the RAG field, showing that as the number of documents in the context window increases, a model's attention to content in the middle positions drops significantly, degrading answer quality. This phenomenon was systematically documented in a 2023 paper and has since become a widely accepted baseline concern in the industry. One motivation for this test was to verify whether this assumption still holds at real-world enterprise document scales.

The Seven Models Tested

The author's model selection logic mirrors real enterprise decision-making: large, expensive flagships versus small, cheap lightweight options; paid APIs versus free, self-hostable models.

Model	Category
Claude Sonnet 5.5	Anthropic flagship
Claude Haiku 4.5	Anthropic lightweight
GPT-5.5	OpenAI flagship
GPT-5.4 mini	OpenAI lightweight
Gemini 3.8 Flash	Google fast model
Gemma 4 (31B)	Google open-source, locally deployable
gpt-oss-20b	OpenAI open-source small model, locally deployable

The two comparisons the author cared most about: which is better — a company's large model or its small model — and whether free open-source models can keep pace with paid ones.

Key Findings

More Documents Did Not Degrade Answers

This was the starting premise of the entire test, and the conclusion refuted it. Four models maintained 100% accuracy with 50 documents — identical to their performance with just 2. gpt-oss-20b only dropped to 92% in the 25-document group, which represents a single error across 12 questions. The true bottom performers, Claude Haiku 4.5 and GPT-5.4 mini, were already at just 67% with only 2 documents, which makes clear that document count is not what's holding them back.

Whether the correct document appeared at the top, middle, or bottom also made no difference — at this scale, the "Lost in the Middle" phenomenon did not appear.

The author notes an important boundary here: the documents were short, each under one page, with all 50 totaling roughly 16,500 words or about 35 pages. He can only claim pile size doesn't matter at this scale — extrapolating to ten times the volume would require further testing.

Two-Step Questions Are Where Small Models Fail

Breaking results down by question type completely changes the picture. On single-step questions, almost every model scored near-perfect. But two-step reasoning revealed enormous gaps:

Model	Single-step	Two-step
Claude Sonnet 5.5	30/30	30/30
Gemini 3.8 Flash	29/29	30/30
Gemma 4	30/30	30/30
gpt-oss-20b	30/30	29/30
Claude Haiku 4.5	30/30	6/30
GPT-5.4 mini	29/30	3/30

Every model could locate a single fact within 50 documents, but two of them couldn't follow a chain from one document to another — even when the pile contained only those two documents.

More importantly, they failed in different ways. Claude Haiku 4.5 tended to simply give up — 21 of its 24 errors were "not found" responses. Often it had already read the referral sentence but stopped there without continuing to the second document. GPT-5.4 mini, by contrast, answered but didn't follow through: 20 of its 27 errors involved copying the referral sentence back as the answer, and another 7 fabricated a plausible real-sounding name while citing a genuine document as the source.

The author stresses these two failure modes are not equivalent: when a model says "not found," you know to check yourself. But when it confidently provides a wrong name and cites a real document, you're likely to believe it. The latter is the far more dangerous failure pattern in enterprise settings.

Multi-hop Reasoning in information retrieval refers to answering a question that requires traversing multiple independent documents or knowledge fragments in sequence to arrive at a final answer. Unlike single-step lookup, multi-hop reasoning demands that a model not only locate the first clue but also use it as a new query to continue searching, forming a "reasoning chain." This is a structural challenge for RAG systems: traditional vector retrieval excels at finding passages semantically similar to the question, but indirect references like "the end of Document A points to Document B" are often far from the original question in vector space and may be filtered out during retrieval, meaning the second critical document never reaches the context at all. This test bypassed the retrieval step entirely, feeding all candidate documents directly to the models — which means it reveals the reasoning capability bottleneck of the language models themselves, not a flaw in the retrieval algorithm. This makes the findings more useful for evaluating fundamental model capabilities.

Free Open-Source Models Beat Paid Small Models

This was the most surprising finding for the author. The locally deployable free model Gemma 4 answered all 60 answerable questions correctly and correctly identified sources every time. The equally free gpt-oss-20b answered 59 out of 60. The two paid small models, by contrast, were far behind — Claude Haiku 4.5 answered only 36, and GPT-5.4 mini only 32.

Model	Type	Answerable Questions Correct
Gemma 4	Free open-source	60/60
gpt-oss-20b	Free open-source	59/60
Claude Haiku 4.5	Paid small model	36/60
GPT-5.4 mini	Paid small model	32/60

gpt-oss-20b's only weakness was source attribution — it usually got the answer right but sometimes listed only one of the two required source documents. Out of 60 questions, it fully cited all sources only 52 times.

The author offers two caveats: first, the paid small models' failures may partly stem from instruction-following style — Haiku leans conservative and prefers "not found" over guessing, so rephrasing the prompt might significantly change the score. Second, "free" doesn't mean zero cost: you don't pay per call, but you need your own hardware, and a model the size of Gemma 4 requires a fairly powerful machine.

No Model Hallucinated Answers

One-third of the questions had no answer in the documents at all. Across all models and all document pile sizes, every one of these questions received a "not found" response. The author had expected that under the pressure of a 50-document pile, at least a few fabricated answers would emerge — but none did. The outdated-copy trap also caught almost no one: only one instance across the entire test involved a model using a number from an old version.

What AI generates content that sounds plausible but doesn't actually exist is called hallucination in the field. In RAG scenarios, hallucination typically takes two forms: a model fabricates a response that matches the tone of the question when the answer isn't in the documents, or it blends details from multiple documents into a "composite fact" that doesn't exist in any single source. The former can be suppressed by requiring the model to output "not found," but the latter is harder to detect. Surprisingly, all models in this test honestly reported "not found" on unanswerable questions — contrary to the common public perception that AI loves to make things up. A plausible explanation is that recent reinforcement training on instruction-following has made mainstream models more conservative in formatted tasks. Another explanation is that the documents themselves were written clearly enough to give models sufficient contextual signals to determine "the answer is not here."

What These Results Can't Tell Us

The author is candid about the test's limitations. GPT-5.5 ran out of API quota and left 35 of 90 questions unanswered (including all 50-document questions). The 55 questions it did answer were all correct, but Kaggle counted the unanswered portion as wrong, making its official score appear lower than it should be. Additionally, each model only ran once — there was no repeated testing to measure score variance across runs. Sample sizes per group were small, so a one- or two-question difference is meaningless, though a gap like 30/30 versus 3/30 on two-step questions is unambiguously real. Finally, the documents were fictional and far cleaner than real company files, and with five of seven models scoring near-perfect, the test is too easy to differentiate between top-tier models.

The author outlines directions for future work: scale the document pile well beyond 50 and make individual documents longer to find where scores actually start dropping; add three- and four-hop reasoning to see if top models can be separated; give GPT-5.5 sufficient quota to rerun; and run each model multiple times.

Practical Takeaways for RAG Deployment

Setting aside the experimental limitations, the real-world lessons from this test are clear: at current scales, giving the AI more documents is not where the risk lies. The real risk is asking a small model a question that requires hopping across two documents to answer.

For teams evaluating options, this points to two things. First, if your use cases are mostly single-point lookups, a cheap or open-source small model is perfectly adequate — and open-source models may even outperform paid lightweight alternatives on accuracy. Second, the moment your questions involve "Document A points to Document B" style multi-hop reasoning, you need to carefully evaluate small models, and especially watch out for the failure mode where a model confidently provides a wrong answer while citing a real source — that's far more dangerous than honestly saying "not found."