LLM Judge Rubrics in Practice: promptfoo vs DeepEval

Why Rubrics Matter More Than the Eval Framework

A developer ran an experiment with 40 customer support replies: he manually labeled 29 as passing and 11 as failing, then ran the same rubric through both promptfoo and DeepEval. The finding: fixing the rubric itself improved agreement far more than switching frameworks.

The experiment used a 312-word refund policy document covering the 30-day window, prorated refunds on annual plans, and no refunds on used add-on credits. The rubric had three dimensions: policy accuracy (weight 0.5, scored 1-5), next-step guidance (weight 0.3, scored 1-5), and tone (weight 0.2, pass/fail). A weighted score of 0.7 or above counted as a pass.

promptfoo: Fast and cheap, but it silently swallows config errors

In promptfoo, you configure each dimension as an llm-rubric assertion with a weight. Initial setup took 25 minutes, the 40-item run took 3 minutes 12 seconds and cost $0.94, and agreement with the manual labels was 34/40.

The problem was a typo: the tone weight was written as 0.3 instead of 0.2, making the three weights sum to 1.1. promptfoo didn't complain at all—it just computes the weighted average using whatever weights you give it. The developer ended up writing a 50-line Python validation script that checks whether the weights sum to 1.0 and whether each dimension's anchor descriptions match the scale it declares.

Another friction point: promptfoo's llm-rubric only accepts 0-1 scores, so using a 1-5 Likert scale means spelling out the conversion in the prompt, which is a bit awkward.

Converting between llm-rubric assertions and Likert scales

In promptfoo, the unit of rubric configuration is the "assertion": each dimension becomes an llm-rubric assertion with a weight field. An assertion is essentially a prompt handed to the judge model, which scores the output according to that prompt.

One easy trap is the scale: llm-rubric accepts continuous 0-1 scores by default, while the article's policy accuracy and next-step guidance dimensions use 1-5 Likert scales (a common graded rating method where 1 is worst and 5 is best). To make promptfoo handle 1-5 scores, you must spell out the conversion in the prompt—something like "Please output an integer from 1 to 5; we will map it to 0-1 via (score-1)/4." Omit that instruction and the model may return raw 1-5 scores, which promptfoo then treats as if they were in the 0-1 range, shifting the entire weighted score. That's why the article calls it "a bit awkward": a scale mismatch doesn't raise an error, it just quietly distorts the scores.

DeepEval: Richer rationales, but noticeable drift under defaults

DeepEval uses GEval metrics. If you don't pass evaluation_steps, it first generates steps automatically from your criteria, then scores against those steps. Setup took about 40 minutes, the run took 4 minutes 48 seconds and cost $1.31, and agreement with manual labels was only 31/40.

But G-Eval produced the best rationales, pinpointing exactly which policy a reply violated. More telling was the drift: across three reruns, 7 replies flipped their pass/fail verdict at least once. The developer suspects the steps were regenerated on each run with slightly different wording. After manually pinning evaluation_steps, flips dropped to 3 and agreement rose to 35/40. By comparison, promptfoo had only 4 flips across three reruns.

How G-Eval and evaluation_steps work

DeepEval's core metric is G-Eval (GEval), which comes from a paper on "having LLMs score step by step." You supply a criteria (e.g., "Does the reply accurately explain the refund policy?"), and there are two modes:

• No evaluation_steps passed: DeepEval first calls the model once to auto-generate a set of scoring steps from your criteria, then hands those steps plus the criteria to the judge model to score item by item. Convenient, but each run generates slightly different wording, so the same reply can get opposite verdicts on a rerun—the "drift" the article describes.
• Manually pinned evaluation_steps: You write a stable set of scoring steps yourself and pass them in, reusing the same steps on every run. Steps are no longer randomly generated, the rubric is anchored, and rerun flips drop from 7 to 3 while agreement rises from 31/40 to 35/40.

This explains why the article advises pinning evaluation_steps "on day one": under the default mode, step instability quietly contaminates your eval results, and it's hard to see from the final scores alone.

The rubric builder: validate first, evaluate later

The developer's builder doesn't call models or run evals. It does one thing: validation at the rubric-writing stage. It checks whether weights sum to 1.0 and whether each dimension's anchor descriptions match its declared scale.

When rebuilding the refund rubric, it caught two problems. The 1.1 weight sum was known; the other was that the tone dimension declared pass/fail but its description had five Likert anchors copied into it. Both promptfoo and DeepEval accepted this contradiction without complaint for days of runs.

After pasting the cleaned anchors and prompts back into both frameworks, promptfoo's agreement went from 34 to 37 and DeepEval's from 35 to 36. The developer guesses promptfoo leans more on the rubric text itself, while G-Eval's step structure partially compensates for rough rubric text.

What weighted scores and weight validation mean

The "weights" the article keeps referencing are the core mechanism of a rubric: when a rubric has multiple dimensions (policy accuracy, next-step guidance, tone), each gets a weight representing its relative importance in the overall score. The final score is the weighted average of all dimension scores. If the weights are 0.5, 0.3, and 0.2, they must sum to 1.0—otherwise the weighted average's scale breaks and every reply's score is systematically inflated or deflated, potentially pushing replies near the 0.7 pass line into the wrong bucket.

The subtler risk is that frameworks usually don't error when weights don't sum to 1. Both promptfoo and DeepEval simply "compute with the weights you give them," with no sanity check. That's why the developer wrote an extra 50-line validation script: it does just two small things—checks whether weights sum to 1.0, and checks whether each dimension's anchor descriptions match its declared scale. This kind of validation calls no models and runs no evals, costs almost nothing, and yet intercepts the most fatal class of errors at the rubric-writing stage.

Dimension-by-dimension trade-offs

Dimension	promptfoo	DeepEval
Setup to first result	25 minutes	40 minutes
First-run agreement	34/40	31/40
Agreement after cleaning anchors	37/40	36/40
Flips across three reruns	4	7 (3 after pinning steps)
Cost for 40 items	$0.94	$1.31
Checks weight sum	No	No
Rationale quality	Short but usable	Longest, most specific
CI integration	CLI and YAML	pytest

How to actually choose

If you only pick one, promptfoo is the better fit: cheaper, faster, fewer flips under defaults, and already in his CI pipeline. If you lean heavily on pytest and don't have a YAML-driven eval setup, pick DeepEval—but pin evaluation_steps on day one.

His current approach: write and validate the rubric first, then run it in promptfoo for CI. If a dimension fails heavily and he needs to understand why, he takes that piece to DeepEval with pinned steps, since G-Eval's rationales are better for debugging.

The core lesson

After spending days comparing frameworks, he found the weakest link was the rubric itself. A 0.3 written as 0.2, a pass/fail dimension carrying five anchors—the accuracy loss from these outweighed the differences between frameworks. Of course, this is 40 items labeled by one person, so it's anecdotal, not a benchmark. If your scoring dimensions are fuzzier than "did they get the refund policy right," drift could be worse.

What "anchors" and scale consistency mean

The article repeatedly mentions conflicts between "anchors" and scales. A scale is the definition of the rating levels: "pass/fail" is a binary scale, "1-5" is a five-point Likert scale. An anchor is the text description for each level of the scale, e.g., "1: makes no mention of the refund policy" and "5: accurately states all applicable rules."

For a rubric to work, anchors must correspond strictly to the scale. A binary scale can only have two anchors (pass, fail); a five-point scale must have five. The article's tone dimension declares "pass/fail" but copies five Likert anchors into its description, leaving the judge model at a loss: it doesn't know whether to output two values or five, and may score under a different interpretation on each run. Neither promptfoo nor DeepEval detects this contradiction, so it runs for days without error, but the noise it introduces directly drags down agreement. Validating that anchors match the scale, like validating weights, is the easiest step to miss when writing a rubric—and one of the costliest.

One-line Summary

After comparing promptfoo and DeepEval, fixing errors in the rubric itself improved agreement with manual labels more than switching eval frameworks.

Paragraph Summary

A developer ran a controlled experiment with 40 customer support replies and a 312-word refund policy document, running the same three-dimension rubric through promptfoo and DeepEval. promptfoo configured faster (25 minutes), cost less ($0.94), and had fewer rerun flips by default, with a first-run agreement of 34/40—but it never warned that the weights summed to 1.1. DeepEval gave more specific rationales but its auto-generated evaluation_steps caused 7 verdict flips, dropping to 3 after pinning steps, with agreement rising from 31/40 to 35/40. The developer also wrote a builder that validates weights and anchors without calling any model, and it caught that the tone dimension declared pass/fail while carrying five Likert anchors. After cleanup, promptfoo's agreement rose to 37/40 and DeepEval's to 36/40—rubric flaws hurt agreement more than framework differences.