rag-eval

rag-eval is an open-source RAG evaluation tool that has zero dependencies, requires no API key, and is framework-agnostic.

rag-eval accepts evaluation data via standard .jsonl files and is not tied to specific frameworks such as Haystack, LangChain, or LlamaIndex.

rag-eval includes two types of local scoring methods built in: a lexical-level claim-overlap scorer and set-based retrieval metrics, both of which run entirely locally and are free.

rag-eval supports enabling a large language model as a judge for semantic validation via the --judge openai parameter, as an optional feature.

rag-eval provides two usage modes—a CLI and a FastAPI microservice—where the CLI outputs rich-text terminal tables with JSON export support, and FastAPI exposes an /evaluate endpoint.

rag-eval can be installed via the command pip install rag-eval.

Integrating evaluation into CI/CD pipelines—comparing key metrics against preset thresholds and blocking merges when tolerance ranges are exceeded—is a key practice for automating RAG quality assurance.

rag-eval employs a layered evaluation strategy of using default local free metrics for initial screening and enabling LLM Judge precision evaluation only at critical checkpoints, in order to control costs.

The source code of rag-eval is hosted on GitHub in the Mustaaabadshah/rag-eval repository.