Open-Source LLMs Fail at Hadith Recitation: Zero Exact Matches Across 30,000 Answers

A new benchmark testing open-source LLMs' ability to recall religious texts delivers a striking conclusion: when "Hadith" is treated as a verifiable recall task, currently deployable open-weight models fail almost universally. The study's title puts it bluntly—they know where the passage is, but not what it says.

Why Hadith recall is a "hardcore" test scenario

Hadith are texts recording the words and deeds of the Prophet Muhammad, holding quasi-canonical status in Islamic jurisprudence and tradition. Unlike open-ended Q&A, Hadith recall has a critical property: it is verifiable.

Given a bibliographic citation—a specific collection, chapter, and narration number—the corresponding text is uniquely determined. The model either reproduces that text or it fabricates. And fabricated "Hadith" are hard to distinguish in appearance from authentic ones; readers may mistake forged content for authoritative scripture.

This is where the severity lies. Ordinary hallucination at most makes someone double-check a source, but high-confidence hallucination in religious texts can flow directly into sermons, teaching, and even legal discussions.

The research team accordingly argues that Hadith recall is not an open-ended generation task but should be treated as a high-risk, verifiable information retrieval problem.

How the test was designed: 30,000 answers, citations only, no source text

Model selection: five open-weight, deployable models

The benchmark covers five open-weight, locally deployable language models: Mistral-7B, Gemma3-4B, Jais-7B, ALLaM-7B, and Phi4-mini-3.8B. The combination deliberately balances general-purpose models with Arabic-oriented ones (such as Jais and ALLaM) to observe whether language adaptation actually improves religious text recall.

Corpus and sampling: full coverage of the six major Hadith collections

Test data covers the six canonical Sunni Hadith collections: Sahih al-Bukhari, Sahih Muslim, Sunan Abi Da'ud, Jami' al-Tirmidhi, Sunan an-Nasa'i, and Sunan Ibn Majah. Each collection contributes 500 Hadith. Each language and model pair accounts for 3,000 entries; across two languages (English/Arabic), each model contributes 6,000 entries, yielding 30,000 answers in total across all models.

Prompting: deliberately no retrieval support

In each test, the model receives only bibliographic citation information and is asked to reproduce the exact text of the Hadith, without retrieval augmentation (RAG). This design choice is critical: it measures the model's parametric memory, not the combined capability of "model plus retrieval pipeline."

Parametric memory refers to knowledge from pretraining that the model "compresses" into weight parameters; at inference it no longer accesses original documents but generates answers directly from parameters. In contrast, non-parametric memory dynamically retrieves original text from an external corpus at inference time.

The inherent limitation of parametric memory is that it is lossy compression. High-frequency, repeated content is more easily remembered, while low-frequency, long-tail, verbatim-precision text—especially details like religious scriptures and legal clause numbers—is hard to retain stably in weights. Models often remember "roughly what this Hadith is about" but lose the exact wording. RAG (Retrieval-Augmented Generation) exists precisely to compensate for this deficiency: first retrieve relevant passages from a verified corpus, then have the model organize an answer based on retrieved results, transforming a "memory" problem into a "retrieval + reading comprehension" problem. This study deliberately disables RAG to measure the upper bound of the parametric memory pathway alone.

Scoring: dual-track lexical and semantic evaluation

Evaluation has two dimensions: lexical fidelity measured by BLEU, semantic fidelity measured by multilingual cosine similarity, with refusals flagged separately. The dual-track approach matters because a model may paraphrase with similar meaning, or be literally close yet lose key semantics—both must be considered.

BLEU (Bilingual Evaluation Understudy) was originally designed for machine translation as an automatic metric: it counts n-gram overlap between model output and reference translation, with a penalty for overly short outputs. It is fast and reproducible, but essentially surface string matching: swap a synonym or adjust word order and BLEU drops noticeably even if meaning is identical. Thus high BLEU doesn't mean correct meaning, and low BLEU doesn't necessarily mean the model "doesn't know."

Multilingual cosine similarity typically uses multilingual sentence embedding models to encode answer and source into dense vectors, then computes the cosine of the angle between them. It measures semantic closeness, more tolerant of paraphrase and synonym substitution, but may score "topically relevant but factually wrong" text highly. This is why the study juxtaposes both: one watches the letter, the other the meaning—only dual-high counts as true reproduction. It also explains why threshold settings significantly affect the final "pass rate."

Results: 0% of answers pass both lexical and semantic gates

This is the most unsettling number in the entire study. Across all five models and two languages—30,000 answers in total—0% achieved dual-high lexical and semantic fidelity, meaning not a single answer constituted an exact, verifiable match to the source text.

This 0% is not statistical noise but a consistent result across five architectures of different scales. It includes both general small models and models specifically optimized for Arabic, yet none demonstrated usable exact recall on this task.

Arabic answers perform worse overall

A counterintuitive finding: Arabic answers generally performed worse than English answers. This is especially noteworthy because the Hadith source texts are in Arabic. Theoretically, Arabic models or models with higher proportions of Arabic training data should excel at source reproduction.

According to the study's unified conclusion, language adaptation did not confer the expected exact-reproduction advantage. This suggests the problem is not merely "insufficient Arabic capability" but that parametric memory itself cannot stably carry verbatim-level canonical texts.

This counterintuitive result can be understood from tokenization and language resources. Arabic morphology is complex with rich inflection; a single root can spawn many surface forms, and tokenizers often split words more finely, making n-gram overlap harder to align during verbatim reproduction. Moreover, mainstream open-source models (such as Mistral, Gemma, Phi) have predominantly English pretraining corpora with relatively limited Arabic share; even Arabic-oriented models like Jais and ALLaM may not have sufficient verbatim coverage of Hadith texts in their training data.

More critically, language capability and verbatim memory are different things. A model can fluently discuss Hadith topics in Arabic yet resort to "reasonable paraphrase" when asked to reproduce exact text. This shows language adaptation improves expressive fluency, not precise storage of canonical texts. This distinction has general reference value for evaluating faithfulness in any non-English domain.

Interpretations and disagreements across multiple perspectives

Discussion in the technical community around this benchmark shows high consensus along with several points of divergence.

Consensus (cross-validated across sources): pure parametric generation without retrieval support is unsuitable for high-risk text tasks requiring verbatim accuracy. The study itself, its public code repository, and community discussion all point to the same conclusion—deployment scenarios should mandate retrieval-augmented generation over verified corpora rather than relying on memory in model weights.

Divergence (presented as-is): some argue the 0% result depends heavily on how scoring thresholds are set; BLEU and cosine similarity penalize "high-quality paraphrase" quite harshly, so the number is better read as "no verbatim-level match" rather than "the model has no idea what the content is." Another view emphasizes that Hadith are not general common sense and rarely appear in high-frequency pretraining regions; evaluating religious texts with general benchmark methodology may underestimate the potential of domain-fine-tuned models. The research team's response to the latter is preserved in the conclusion: even with fine-tuning, verbatim-level fidelity still requires external verification as a fallback.

Single-source insight: the study explicitly argues that model deployment for Islamic scholarly research should treat "retrieval-augmented generation + verified corpus" as a hard requirement, not an option. This position is currently advanced mainly by this research team.

Implications for AI application development

Although this study focuses on religious texts, its conclusions transfer to broader AI application scenarios.

First, verifiable tasks must follow the retrieval path. Any scenario with a unique correct answer that can be checked verbatim—legal clauses, drug labels, standard provisions, financial disclosures—is unreliable when relying purely on model memory. Such tasks should be explicitly classified as RAG problems, not prompt engineering problems.

Second, language adaptation does not equal domain reliability. Arabic models performing worse on Arabic texts breaks the intuition that "right language means all is well." Domain reliability comes from data verification and retrieval pipelines, not language coverage.

Third, high-risk scenarios need refusal mechanisms. The study separately flags refusals, which is itself a responsible design. A model explicitly refusing when uncertain is far safer than generating a perfectly formatted but fabricated scripture. Product teams should treat "I cannot verify" as a first-class response type.

Open resources and reproduction

Commendably, the study's full code, prompts, evaluation pipeline, and scored outputs are all public at github.com/dayam8696/Hadith-Hallucination-Benchmark. This means the community can replace scoring thresholds, expand the model roster, add RAG control groups, and independently verify the robustness of this 0% conclusion.

For teams building domain Q&A products, the value of this benchmark lies not only in its conclusions but in providing a reproducible evaluation skeleton: how to construct verifiable samples, how to do bilingual comparison, how to score lexical and semantic dimensions separately, and how to count refusals separately. This methodology transfers directly to other high-fidelity text scenarios.

The core judgment can be summed up in one sentence: the model remembers "which chapter of Bukhari that Hadith is in," but cannot reproduce what it says. In domains requiring verbatim accuracy, this gap cannot be closed by simply adding more parameters.

One-line Summary

Five open-source models, 30,000 Hadith recalls, zero verbatim matches—exposing the hard limits of parametric memory.

Paragraph Summary

A new benchmark tested five locally deployable open-source models (Mistral-7B, Gemma3-4B, Jais-7B, ALLaM-7B, Phi4-mini) on verbatim recall of the six canonical Sunni Hadith collections, providing only bibliographic citations without retrieval augmentation, yielding 30,000 answers. Results show 0% achieved dual-high lexical (BLEU) and semantic (multilingual cosine similarity) fidelity, with no answer constituting an exact match to source text. Arabic answers performed worse than English overall, despite the Hadith originals being in Arabic. The study argues that deployment for verifiable, high-risk scenarios like religious texts should mandate retrieval-augmented generation over verified corpora rather than relying on memory in weights, and has open-sourced all code and evaluation pipelines for reproduction.