rag-critic
A lightweight Python package for adversarial factual auditing of RAG systems. It detects and corrects unsupported claims in LLM-generated text through atomic proposition decomposition, NLI entailment judgments, and continuous grounding scores.
Timeline (last 90 days)
rag-critic is a lightweight Python package that acts as an adversarial verification node auditing responses before they reach the user; it can be installed via pip install rag-critic, uses the MIT license, and has been published to PyPI.
rag-critic's core workflow consists of four steps: atomic proposition decomposition, an entailment matrix (NLI judging SUPPORTED/CONTRADICTED/UNGROUNDED), a continuous grounding score (0.0-1.0), and actionable rewrite instructions.
When the grounding score falls below a threshold (the author gives 0.85 as an example), rag-critic pauses graph execution and issues specific diagnostic instructions to an iterative rewrite agent.
rag-critic offers native LangGraph integration: create_critic_node() generates the verification node, and critic_router routes based on the audit result, turning the graph structure from linear into a loop with conditional branches.
In the author's standalone Python usage example, a draft answer containing one grounded claim and one ungrounded funding claim was judged REJECTED_HALLUCINATION, with a grounding score of 50.0%.
rag-critic's evaluation latency can be sub-second via Groq, OpenAI, Ollama or heuristic mode.
rag-critic supports local Ollama models and can run 100% offline, and also offers a heuristic mode.
rag-critic is positioned as a runtime gate and RAGAS as a development-time quality dashboard; the two are not substitutes and can be used in combination.
The original text does not provide any benchmark test data, including datasets, threshold calibration methods, and NLI classifier misjudgment rates.
All Facts (9)
rag-critic is a lightweight Python package that acts as an adversarial verification node auditing responses before they reach the user; it can be installed via pip install rag-critic, uses the MIT license, and has been published to PyPI.
50%Unverifiedrag-critic's core workflow consists of four steps: atomic proposition decomposition, an entailment matrix (NLI judging SUPPORTED/CONTRADICTED/UNGROUNDED), a continuous grounding score (0.0-1.0), and actionable rewrite instructions.
50%UnverifiedWhen the grounding score falls below a threshold (the author gives 0.85 as an example), rag-critic pauses graph execution and issues specific diagnostic instructions to an iterative rewrite agent.
50%Unverifiedrag-critic offers native LangGraph integration: create_critic_node() generates the verification node, and critic_router routes based on the audit result, turning the graph structure from linear into a loop with conditional branches.
50%UnverifiedIn the author's standalone Python usage example, a draft answer containing one grounded claim and one ungrounded funding claim was judged REJECTED_HALLUCINATION, with a grounding score of 50.0%.
50%Unverifiedrag-critic's evaluation latency can be sub-second via Groq, OpenAI, Ollama or heuristic mode.
50%Unverifiedrag-critic supports local Ollama models and can run 100% offline, and also offers a heuristic mode.
50%Unverifiedrag-critic is positioned as a runtime gate and RAGAS as a development-time quality dashboard; the two are not substitutes and can be used in combination.
50%UnverifiedThe original text does not provide any benchmark test data, including datasets, threshold calibration methods, and NLI classifier misjudgment rates.
50%