Rubricon: Automatically Generating Rubrics to Boost LLM Evaluation and Training Efficiency

In LLM evaluation and training, developing rubrics has always been a challenge: manual authoring is costly and struggles to cover all dimensions of complex tasks. The recently proposed Rubricon pipeline can automatically generate discriminative rubrics without human preference labels, scoring annotations, or generator model training, offering a new automated path for LLM evaluation and training.

Why do we need automatic rubric generation?

For tasks without verifiable answers, rubrics are key to assigning rewards. However, writing rubrics from scratch manually is both expensive and time-consuming, and it is easy to miss key requirements. Existing synthesis methods often rely on human preference data or require training generative models, limiting their generality. Rubricon's goal is to generate rubrics that are discriminative across model responses using only any off-the-shelf LLM, through an iterative process of sampling, selection, supplementation, and pruning.

A rubric is essentially a checklist of criteria describing "what makes a good answer," based on which an evaluator model or reward model scores each dimension of a response item by item. It stands in contrast to "verifiable answers" (such as the standard result of a math problem or a code test case): the latter can be judged right or wrong directly by rules, whereas open-ended tasks (writing, medical consultation, multi-turn dialogue) have no unique answer and can only approximate human judgment through a set of criteria. In paradigms such as RLHF, reinforcement learning beyond verifiable rewards (RLVR), and LLM-as-a-Judge, the quality of the rubric directly determines the signal-to-noise ratio of the reward signal—criteria that are too coarse lead to reward saturation and fail to distinguish good from bad, while criteria that are too fine-grained or misaligned introduce noise.

Existing methods for synthesizing rubrics fall roughly into two categories: one relies on human preference pairs or expert annotations as supervision signals, which is costly and hard to scale; the other trains dedicated generator models (such as the rubric generator in RRD), requiring additional training data and compute. Rubricon's difference lies in treating rubric generation as a training-free inference-time search problem, borrowing only the sampling capability of any existing LLM, thereby avoiding the two barriers of annotation and model training.

How does Rubricon work?

At the core of Rubricon is a four-step loop: sample multiple candidate rubrics, select the most discriminative candidate, add criteria that capture overlooked requirements or response flaws, and prune misaligned or shallow criteria. To guide refinement, Rubricon relies not only on LLM self-critique and test-time scaling, but also on criterion pass rates observed on example model responses—these pass rates provide a data-driven signal that helps identify which criteria truly distinguish responses of different quality.

This design allows Rubricon to self-improve without any human annotation. According to the paper, the pipeline can be used with any LLM and is not limited to a specific model architecture.

"Criterion pass rate" is the key signal for understanding the method. For a given candidate criterion, judging a set of example model responses one by one against whether they satisfy it yields the pass ratio of that criterion on this batch of responses. If almost all responses pass, the criterion is too shallow and lacks discriminative power; if almost none pass, it may be misaligned with the task or ambiguously worded. A truly useful criterion should show a spread in pass rate across responses of different quality.

The value of this signal is that it is data-driven rather than relying solely on LLM self-critique—the latter easily produces criteria that are self-consistent but non-discriminative. Combined with test-time scaling (i.e., improving output quality by increasing sampling and filtering compute without updating parameters), Rubricon effectively turns "writing good rubrics" into an iterative search process with external feedback.

How well does it work? Cross-benchmark validation

The research team evaluated Rubricon on five professional, consumer, clinical, and general LLM benchmarks. Results show that scores assigned by Rubricon-generated rubrics to frontier LLM responses achieve 9% higher alignment with labor-intensive expert-written rubrics than the best alternative synthesis method. Comparison methods include RRD, QSRG, OpenRubrics, and agentic generation via tools such as Codex.

On HealthBench, post-training the Qwen3.5-9B model with Rubricon rubrics outperforms training with the strongest alternative synthetic rubric generator, RRD, by 4.5%. This shows that automatically generated rubrics can be used not only for evaluation but also to effectively improve model training.

The post-training mentioned here refers to further optimizing a base model using reward signals constructed from rubrics, commonly in forms such as rejection sampling fine-tuning, DPO, or rubric-based reinforcement learning. The key here is "the rubric is the reward": the same set of criteria can be used both to evaluate model outputs and to serve as a scoring function during training, thereby connecting evaluation and training. HealthBench is a benchmark for real-world medical dialogue scenarios, whose questions are mostly open-ended consultations without a unique correct answer, so it places high demands on the dimensional coverage and discriminability of rubrics—making it a suitable testing ground for the practicality of automatic generation methods.

Multi-source perspectives and limitations

Current discussion of Rubricon comes mainly from the paper itself. In the rss feed, the work is described as a pipeline requiring no human intervention, emphasizing its scalability and compatibility with existing LLMs. However, some views point out that automatically generated rubrics may still depend on the quality of the initial LLM, and may require additional calibration in certain highly specialized domains. The 9% and 4.5% improvements reported in the paper were achieved on specific benchmarks, and generalization still needs further validation.

Overall, Rubricon offers a new approach to automating LLM evaluation and training, reducing reliance on manual rubrics and potentially lowering evaluation costs and accelerating model iteration. In the future, with testing on more benchmarks and feedback from practical applications, such methods may become a standard component of the LLM development workflow.

One-line Summary

Rubricon automatically generates discriminative rubrics via iterative sampling and criterion pass rates, without human annotation.

Paragraph Summary

The article introduces the Rubricon pipeline: for tasks without verifiable answers, it uses only any off-the-shelf LLM to automatically generate high-quality rubrics through a four-step loop of sampling candidate rubrics, selecting the most discriminative ones, supplementing overlooked requirements, and pruning misaligned or shallow criteria—all without human preference labels, scoring annotations, or generator model training. Its refinement signal includes not only LLM self-critique but also a data-driven metric: criterion pass rates on example responses. On five professional, consumer, clinical, and general benchmarks, Rubricon's scores achieve 9% higher alignment with expert standards than the best alternative synthesis method; post-training Qwen3.5-9B with its rubrics on HealthBench outperforms RRD by 4.5%. The authors also note limitations such as dependence on the quality of the initial LLM and generalization yet to be validated.