Cosine Similarity Isn't Enough: rag-critic, an Adversarial Fact Auditor for LangGraph

Why Cosine Similarity Can't Save RAG from "Confident Hallucinations"

Production RAG systems keep tripping over the same pitfall: the retriever returns top-k chunks with cosine similarity above 0.85, yet the LLM wanders beyond that context and even fabricates numbers that appear nowhere in the reference documents. This is the "confident hallucination"—the model sounds certain, but the basis doesn't hold up.

The structure of a traditional single-pass RAG pipeline is Query → Vector DB → LLM Generation → Output to user, with no self-correction step in between. Vector similarity measures semantic closeness, not factual entailment. A passage being close to the question in embedding space doesn't mean it supports every concrete claim the model makes.

rag-critic, released by a developer on Reddit, targets exactly this gap: it's a lightweight Python package that acts as an adversarial verification node before the response reaches the user. Installation is a single line, pip install rag-critic, it's MIT-licensed, and it's published on PyPI.

Split, Entail, Score: The Four Mechanisms of rag-critic

According to the author's explanation in the Reddit post, the core pipeline of rag-critic has four steps.

Atomic proposition decomposition: rather than evaluating a whole paragraph, it breaks compound sentences into independent claims. When a passage mixes three facts—two verifiable and one fabricated—scoring the paragraph as a whole only masks the problem.

Entailment matrix: each proposition is checked against the retrieved context chunks using an NLI classifier, labeled as SUPPORTED, CONTRADICTED, or UNGROUNDED. This is the most fundamental difference from cosine similarity—similarity gives only a continuous value, while entailment judgment gives a directional factual relationship.

Continuous grounding score: outputs a numeric confidence between 0.0 and 1.0, making it easy to set thresholds for engineering decisions.

Actionable rewrite instructions: when the grounding score drops below the threshold (the author uses 0.85 as an example), the graph pauses and sends specific diagnostics to the iterative rewrite agent, such as "Remove unsupported claim: 'Platform raised $50M in London'."

NLI (Natural Language Inference) is a foundational task in NLP: given a "premise" text and a "hypothesis" sentence, determine whether the hypothesis stands in an entailment, contradiction, or neutral/unrelated relationship to the premise. NLI classifiers are typically built by fine-tuning pretrained models such as BERT, DeBERTa, or RoBERTa on large-scale datasets like MNLI and SNLI. In RAG auditing, the retrieved context chunk serves as the "premise" and the model-generated atomic proposition serves as the "hypothesis," and the classifier output is the grounding verdict for that proposition. Whereas cosine similarity can only measure semantic distance, NLI directly models the direction of the logical relationship between two texts, so it can distinguish cases like "semantically related but factually opposite" that similarity can't capture. Note that general-purpose NLI models lose accuracy on numerical reasoning, nested negation, and long-tail entities—one of the core bottlenecks in audit quality.

Code Example: What a Rejected Answer Looks Like

In the standalone Python usage the author provides, the draft answer contains two sentences: one about DeepTrace detecting document tampering using ELA at Q=95, and another claiming the platform closed a $50M Series A in London. The retrieved context supports only the first sentence.

from rag_critic import RAGCritic

critic = RAGCritic(provider="groq", model="llama-3.3-70b-versatile")
result = critic.evaluate(draft_answer, retrieved_context)
print("Status: ", result.status)              # REJECTED_HALLUCINATION
print("Grounding Score:", f"{result.grounding_score:.1%}")  # 50.0%

The output status is REJECTED_HALLUCINATION, the grounding score is 50%, and the feedback explicitly points out that the funding proposition can't be found in the reference paragraphs. This example is telling: the passage as a whole is highly semantically related to the context, and cosine similarity would likely be high, yet half of its claims are ungrounded.

On evaluation latency, the author claims sub-second performance via Groq, OpenAI, Ollama (100% offline capable), or a heuristic mode.

LangGraph Integration: create_critic_node and critic_router

For teams running multi-agent workflows, rag-critic offers native LangGraph integration: create_critic_node() produces the verification node, and critic_router handles routing based on the audit result. Once integrated, the graph structure shifts from linear to a loop with conditional branches—if the audit fails, the flow returns to the rewrite agent instead of dumping output to the user.

This effectively turns "evaluation" from an offline metric into part of the online control flow. Most teams' RAG evaluation still amounts to running a batch of test sets and checking scores, whereas rag-critic's approach is to gate every single generation.

LangGraph is an orchestration framework from the LangChain team for building stateful, multi-agent workflows. It models an application as a graph: nodes are execution units, edges define the direction of flow, and conditional edges allow the next step to be chosen dynamically based on runtime state. This structure naturally suits loops and branches—for example, a closed loop of "generate → audit → return to rewrite on failure." create_critic_node() returns a callable that conforms to the LangGraph node signature, receiving and updating shared state; critic_router is a routing function that reads the audit result and decides whether to proceed to the output node or fall back to the rewrite node. Compared with a linear chain, LangGraph's gated graph structure makes "evaluation as control flow" possible without hard-coding verification logic into business code.

What It Solves, and What It Doesn't

To its credit, the problem identification is spot on. Cosine similarity is fine as a retrieval recall metric, but treating it as a factual guarantee is a category error. Atomic proposition decomposition plus NLI entailment judgment upgrades "is this passage relevant" to "is this sentence supported," offering finer granularity and better explainability—the diagnostic feedback can tell the rewrite agent exactly which sentence to delete, which is far more useful than a vague low score.

The caveats are equally obvious. The original post provides no benchmark data: which datasets were used, how the grounding-score threshold is calibrated, and what the NLI classifier's error rate is are all unaddressed. The 0.85 threshold is just the author's example; appropriate thresholds may vary widely by domain. Moreover, the NLI classifier's own performance on long-tail facts, numerical comparisons, and negations directly affects audit quality—a point the original post doesn't elaborate on.

For engineers running production LangGraph or evaluation clusters, this package is worth a try—it's lightweight, cheap to integrate, and turns the "adversarial audit" idea into a reusable node. But before relying on its gate for release decisions, I'd recommend testing it against a set of failure cases from your own domain to see how well grounding scores align with human judgment.

RAGAS (Retrieval Augmented Generation Assessment) is a currently popular RAG evaluation framework offering metrics such as faithfulness, answer relevancy, and context precision/recall, aimed primarily at offline batch evaluation: given a question set and generated results, it computes aggregate scores and does regression comparisons. Its faithfulness metric also performs proposition-level decomposition, but the output is dataset-level statistics rather than a per-request pass/block signal. rag-critic and RAGAS aren't substitutes: the former is a runtime gate, the latter a development-time quality dashboard. A common combination in practice is using RAGAS for iterative evaluation and rag-critic for online interception—both share the "atomic proposition + entailment judgment" methodology, but serve different goals: one optimizes long-term metrics, the other safeguards individual outputs.

FAQ

How is rag-critic different from evaluation frameworks like RAGAS? RAGAS leans toward offline batch evaluation, while rag-critic is positioned as a runtime node—a failed audit blocks the graph and triggers a rewrite, making it an online control-flow component.

Does it require an internet connection? No. The author states it supports local Ollama models, can run 100% offline, and also has a heuristic mode.

What grounding score counts as passing? The original post uses 0.85 as an example, but that's an illustrative value, not a recommendation—you need to calibrate it for your domain.