Hands-On Arize Phoenix Evaluation: LLM Judge Disagreement in Small Samples and Choosing an Automated Evaluator

In LLM application development, reliably judging the quality of model outputs has long been a thorny problem. One developer's practice of setting up prompts, evals, and datasets in Arize Phoenix offers a concrete window into this problem.

Small-sample experiment reveals judge disagreement

According to a practice-sharing post on the DEV community, the author used 24 Jev invocation samples to configure prompts, evals, and a dataset in Arize Phoenix. The most direct finding from the experiment was that disagreement among different judges was larger than expected.

This result is not surprising, but it reminds us with a concrete number: in LLM evaluation, inconsistency in the judging criteria themselves may be an underestimated source of error. Although 24 samples is a very small scale, it is enough to expose systematic differences among judges. For any team that relies on automated evaluation to make iteration decisions, this means conclusions from a single evaluation need to be treated with caution.

The original author says they plan to expand the experiment and obtain labeled data for a small multiple-heads classification model. This direction is worth watching—moving from purely prompt-based judging to a supervised small model may be a path to reducing evaluation noise.

LLM-as-a-Judge refers to the evaluation paradigm of using another large language model to score or rank the outputs of a target model. It usually uses carefully designed prompts to have the judging model give scores or preferences along dimensions such as relevance, factuality, and completeness. This method is low-cost and scalable, but it has known biases: the judging model may prefer outputs similar in style to its own (self-preference bias), be sensitive to position (position bias), and be sensitive to length. Therefore, the same batch of samples may receive inconsistent scores under different judging models or under the same model with different prompts, which is one of the root causes of the "judge disagreement" mentioned in the article. In practice, human annotation is often introduced for calibration, or multiple judges are used to vote in order to reduce single-point noise.

A multiple-heads classification model refers to attaching multiple independent classification output layers on top of a shared encoder (backbone), where each "head" is responsible for a subtask or prediction along one dimension. In evaluation scenarios, a shared text encoder can simultaneously predict multiple dimensions such as "relevance," "factuality," and "fluency," thereby obtaining multi-dimensional scores in a single forward pass. Compared with pure prompt-based judging, this kind of supervised small model has fewer parameters, lower inference cost, and stable, reproducible outputs, and it can correct biases in a targeted way through labeled data. The cost is that it requires a certain amount of high-quality annotation, and its generalization ability is limited by the training distribution. The article's plan to obtain labeled data to train such a model is precisely aimed at replacing randomness with determinism and reducing evaluation noise.

Choosing the automated daily evaluator: Sol 6.1

For judge selection, the author currently chooses Sol 6.1 as the default judge for automatic daily eval and plans to do more spot checks later.

The logic behind this choice is not elaborated in detail in the original text, but from an evaluation engineering perspective, daily automatic evaluation requires stability and reproducibility rather than absolute best performance. When judges disagree substantially, choosing one baseline and continuously tracking changes in its behavior is more helpful for building a trustworthy evaluation baseline than frequently switching judges. The author's practice of retaining spot checks also shows that the results of automatic evaluation still need human oversight as a backstop.

Automatic daily eval is a continuous monitoring mode in evaluation engineering: run the evaluation once a day with a fixed dataset and fixed judge, record scores, and track trends. It does not pursue the absolute accuracy of a single evaluation but emphasizes baseline stability—only when the judge is fixed, the data is fixed, and the prompt is fixed can score changes be attributed to changes in the model or application under test. If judges are changed frequently, historical scores lose comparability and the trend line breaks. This is also why the author chooses one default judge and retains manual spot checks: automatic evaluation is responsible for coverage and frequency, while manual spot checks are responsible for calibration and detecting systematic drift.

Implications for evaluation workflows

The value of this practice case lies in showing a pragmatic evaluation iteration loop: first use small samples to quickly expose problems, then plan a supervised approach based on disagreement data, while choosing one judge to maintain continuity in daily operations.

For teams building evaluation systems on Arize Phoenix or similar tools, two references can be drawn from it: first, judge disagreement itself should be recorded and analyzed as part of the evaluation data, rather than focusing only on final scores; second, going from small samples to labeled data and then to a small classification model is a controllable evolution path, suitable for gradual progress when resources are limited.

The original article does not disclose the specific measurement method for disagreement, detailed performance data for Sol 6.1, or the distribution of the 24 samples. These gaps mean the current conclusions are more suitable as directional reference than as configuration advice that can be directly reused. The subsequent multiple-heads model experiment is worth continued attention.

One-line Summary

A small-sample experiment exposed disagreement among LLM judges; the author uses a fixed judge to maintain daily evaluation and plans a supervised small model.

Paragraph Summary

A developer used 24 Jev samples to set up prompts, evals, and a dataset in Arize Phoenix and found that disagreement among different judges was larger than expected, indicating that inconsistent judging criteria are an underestimated source of error in LLM evaluation. The author chose Sol 6.1 as the default judge for automatic daily eval and retained manual spot checks as a backstop, prioritizing stability and reproducibility over absolute optimality. The next step is to obtain labeled data and train a small multiple-heads classification model, moving from pure prompt-based judging to supervised evaluation to reduce noise. The article argues that judge disagreement should be recorded as part of the evaluation data, and that going from small samples to labeled data and then to a small model is a controllable evolution path under limited resources, but the current conclusions are only suitable as directional reference.