RAG Evaluation and Hallucination Guardrails: How to Detect and Block AI Hallucinations

RAG (Retrieval-Augmented Generation) pipelines are powerful, but they introduce new failure points. The model's performance depends entirely on what you feed it, and when it receives the wrong content, it won't say "I got confused"—it will confidently give an answer.

Here's a simple analogy: an open-book exam only works if the student flips to the right page. Hand them the wrong chapter, and they'll still produce a fluent, persuasive, but completely wrong answer.

More critically, if the retrieval step pulls the wrong documents, the LLM will still write a fluent response based on them. The result is confident and wrong—meaning hallucinations can originate in the retrieval step, not the model itself.

Measuring the Two Stages Separately

When evaluating a RAG system, you need to score its two stages separately, because they fail in different ways.

Context relevance examines whether the retrieved documents actually contain the answer to the user's question. This tests the retrieval step.

Faithfulness examines whether the generated answer comes strictly from the retrieved context, or whether the model introduced external information. This tests the generation step.

These two metrics need to be viewed separately because a RAG system has distinctly different failure modes: retrieval failure means "wrong book," while generation failure means "right book but fabricated answer." If you evaluate them together, you won't know which stage the problem lies in.

Context relevance is related to but not identical to the classic information retrieval concepts of recall and precision. Recall measures "whether the documents needed for the answer were retrieved," precision measures "how many of the retrieved documents are relevant," while context relevance is closer to a holistic judgment: whether the retrieval results as a whole are sufficient to answer the user's question. In practice, insufficient recall often leads to "missing material" in answers, while insufficient precision lets large amounts of irrelevant noise into the context, interfering with the generation model.

Faithfulness corresponds to the concept of Factual Consistency, which has been extensively studied in natural language generation evaluation. It differs subtly from "truthfulness": truthfulness means whether the answer aligns with objective world facts, while faithfulness only requires that the answer does not go beyond the given context. If a document itself contains incorrect information, a model that faithfully restates it has high faithfulness but low truthfulness. Therefore, in RAG scenarios, faithfulness is a more measurable and controllable metric, while truthfulness also depends on the reliability of the retrieval source itself.

Identifying Unfaithful Answers

Faithfulness checking is done claim by claim.

For example: if the context says a REIT paid dividends in Q3, and the answer additionally mentions its yield is 7.4%, then that second claim is not in the context. It was fabricated—that's a hallucination, even if it sounds completely plausible.

This claim-by-claim comparison is more precise than holistic scoring. An answer might have 90% of its content verifiable, but mixed with 10% fabricated information, it overall looks "pretty good"—but that 10% hallucination is enough to mislead users into making wrong decisions.

Guardrails: Using an LLM as Judge

This check can be automated. A small auxiliary LLM acts as a judge in the pipeline. Before the final answer reaches the user, the judge compares it against the retrieved text and decides whether to pass or block it.

When the check fails, you can either retry retrieval or simply return "I don't know."

Here's a short Python example. The llm_judge function represents calling a small model, using a prompt to have it compare the answer against the context:

def guarded_answer(question: str) -> str:
    chunks = search(embed(question), top_k=3)
    answer = generate(question, chunks)

    # Judge: is every claim in the answer supported by retrieved text?
    verdict = llm_judge(
        "Reply SUPPORTED or UNSUPPORTED. Is every claim in the answer "
        "backed by the context?\n\n"
        f"Context: {chunks}\n\nAnswer: {answer}"
    )
    if verdict.strip().upper().startswith("SUPPORTED"):
        return answer
    return "I could not find a reliable answer in the documents."

The core logic is clear: retrieve → generate → judge review → pass or block. If the judge returns "SUPPORTED," output the answer; otherwise return the fallback message.

The judge itself can also make mistakes, so its decisions should be sampled and reviewed. This is often overlooked—many teams deploy a judge mechanism and assume it's always correct, but in reality judge models also need evaluation and calibration.

Using an LLM as a judge is commonly called LLM-as-a-Judge, and its core assumption is: a powerful language model, given clear criteria and context, can judge quality close to human annotators at a much lower cost. It has been used to evaluate summarization quality, dialogue helpfulness, and many other tasks.

But this approach has known biases: judge models may prefer longer answers, outputs similar to their own style, or show position bias toward options that appear first. Therefore, in practical deployment, it's typically necessary to fix prompt templates, randomize option order, and periodically calibrate against human annotations. Common reliability metrics include the agreement rate between judge and human judgments; when agreement is too low, consider switching to a stronger judge model or increasing the proportion of human review. These details determine whether the guardrail is a real safeguard or just for show.

Building a Regression Test Set

Maintain a small test set containing real questions and known answers, and run it every time you change the chunking strategy, embedding model, or prompts.

This is how you catch regressions before deployment. RAG system parameter tuning often has ripple effects—changing chunk size can affect retrieval quality, swapping embedding models can change similarity rankings, adjusting prompts can affect generation style. Without regression tests, you're using users as test subjects.

The test set doesn't need to be large, but it should cover typical scenarios and edge cases. The key is to run it with every change, making it a habit.

A regression test set is essentially a Golden Dataset, i.e., a set of input samples with reference answers or reference judgments, used to quickly verify that behavior hasn't degraded after system changes. It follows the same philosophy as unit testing: don't pursue coverage, but pursue continuous protection of critical paths.

In RAG scenarios, the test set typically needs to cover three types of cases: normal questions (verifying basic capability), edge questions (such as multi-hop reasoning, questions requiring cross-document synthesis), and adversarial questions (such as cases where the retrieval source lacks the answer, or contains distractor documents). The latter two most easily expose coordination defects between retrieval and generation. It's recommended to record both "expected retrieved documents" and "expected answer" for each test sample, so a single run can simultaneously evaluate both context relevance and faithfulness, making the regression signal more complete.

Key Takeaways

Preventing hallucinations in RAG systems requires two dimensions: evaluate retrieval and generation separately, using context relevance and faithfulness metrics to locate problems; implement automated hallucination guardrails through an LLM judge to intercept answers before they reach users; and continuously maintain a regression test set to ensure system changes don't introduce new problems.

Additional Background

In engineering practice, you can also introduce open-source evaluation frameworks like RAGAS and TruLens, which encapsulate metrics such as context relevance, faithfulness, and answer relevance into callable evaluation functions, reducing the cost of building measurement logic from scratch. At the same time, making evaluation and guardrails fixed steps in the pipeline (such as in CI workflows) can turn "run tests on every change" from a manual habit into an automated constraint, further reducing regression risk.

One-line Summary

RAG hallucinations can start in retrieval rather than the model—evaluate stages separately and build guardrails with LLM judges and regression tests.

Paragraph Summary

The article points out that RAG pipelines introduce an often-overlooked failure point: when retrieval pulls the wrong documents, the LLM still writes a fluent and confident answer based on the wrong context, so the source of hallucination may lie in retrieval rather than the model itself. To locate the problem, you must split the system into two stages and evaluate them separately—context relevance tests retrieval quality, and faithfulness tests whether generated content comes strictly from the context. Faithfulness checking should compare claim by claim, identifying fabricated information that seems plausible but isn't in the retrieved text. The author recommends using a small LLM as a judge to pass or block answers before they reach users, while emphasizing that the judge itself also needs sampling review. Additionally, maintaining a regression test set with real questions and known answers, re-run every time chunking, embedding models, or prompts are adjusted, is a key means of preventing parameter changes from introducing new problems.