promptfoo
An open-source LLM testing and evaluation framework that supports automated evaluation, red-team testing, and prompt comparison for LLM applications. It can be deployed as a self-hosted solution.
Timeline (last 90 days)
promptfoo integrates with CI via CLI and YAML, while DeepEval integrates via pytest.
The score range accepted by default by promptfoo's llm-rubric assertion is a continuous value from 0-1, and when using a 1-5 Likert scale the conversion method must be explicitly written in the prompt.
Across three re-runs, only 4 responses in promptfoo had their pass/fail conclusion flip.
After cleaning up the anchors and prompts, promptfoo's agreement rate rose from 34/40 to 37/40, and DeepEval's rose from 35/40 to 36/40.
promptfoo's first-run configuration took 25 minutes, running 40 data items took 3 minutes 12 seconds, the cost was $0.94, and the agreement rate with manual annotation was 34/40.
promptfoo does not raise an error when the sum of scoring rubric weights is not 1.0; instead it computes a weighted average using the given weights.
The evaluation and testing category is the largest category, containing 33 tools, with representative projects including promptfoo, deepeval, and openevals.
promptfoo supports multiple assertion types (including keyword matching, regex matching, semantic similarity, and LLM-as-a-Judge), and after running the promptfoo eval command, it automatically calls models in batch and generates a visual report.
promptfoo is a command-line and YAML-driven AI evaluation tool that supports CLI, batch invocation, and Python custom extensions, and can cover three layers of evaluation targets: LLMs, knowledge bases, and Agents.
All Facts (9)
promptfoo integrates with CI via CLI and YAML, while DeepEval integrates via pytest.
50%UnverifiedThe score range accepted by default by promptfoo's llm-rubric assertion is a continuous value from 0-1, and when using a 1-5 Likert scale the conversion method must be explicitly written in the prompt.
50%UnverifiedAcross three re-runs, only 4 responses in promptfoo had their pass/fail conclusion flip.
50%UnverifiedAfter cleaning up the anchors and prompts, promptfoo's agreement rate rose from 34/40 to 37/40, and DeepEval's rose from 35/40 to 36/40.
50%Unverifiedpromptfoo's first-run configuration took 25 minutes, running 40 data items took 3 minutes 12 seconds, the cost was $0.94, and the agreement rate with manual annotation was 34/40.
50%Unverifiedpromptfoo does not raise an error when the sum of scoring rubric weights is not 1.0; instead it computes a weighted average using the given weights.
50%UnverifiedThe evaluation and testing category is the largest category, containing 33 tools, with representative projects including promptfoo, deepeval, and openevals.
50%Unverifiedpromptfoo supports multiple assertion types (including keyword matching, regex matching, semantic similarity, and LLM-as-a-Judge), and after running the promptfoo eval command, it automatically calls models in batch and generates a visual report.
50%Unverifiedpromptfoo is a command-line and YAML-driven AI evaluation tool that supports CLI, batch invocation, and Python custom extensions, and can cover three layers of evaluation targets: LLMs, knowledge bases, and Agents.
50%