Rubricon
An inference-time search pipeline that automatically generates LLM scoring rubrics without human annotation. Through an iterative loop of sampling, selection, supplementation and pruning, it uses criterion pass-rate signals to improve the discriminative power of the rubrics and can be used with any LLM.
Timeline (last 90 days)
Rubricon is a pipeline that can automatically generate scoring rubrics without human preference labels, rating annotations, or training of a generator model, and can be used with any LLM.
The core of Rubricon is a four-step iterative loop: sample multiple candidate rubrics, select the most discriminative candidate, add criteria that capture overlooked requirements, and prune misaligned or superficial criteria.
Rubricon uses the criterion pass rates observed on example model responses as a data-driven signal to guide rubric refinement, rather than relying solely on LLM self-critique.
Rubricon was evaluated on five professional, consumer, clinical, and general-purpose LLM benchmarks.
The alignment of scores that Rubricon-generated rubrics assign to frontier LLM responses with expert-written rubrics is 9% higher than that of the best alternative synthesis method, with compared methods including RRD, QSRG, OpenRubrics, and agent-based generation methods such as Codex.
On HealthBench, post-training Qwen3.5-9B with Rubricon rubrics yields performance 4.5% higher than training with RRD, the strongest alternative synthetic rubric generator.
Automatically generated rubrics may still depend on the quality of the initial LLM, and may require additional calibration in certain highly specialized domains; the 9% and 4.5% improvements were achieved only on specific benchmarks, and generalization capability still needs verification.
All Facts (7)
Rubricon is a pipeline that can automatically generate scoring rubrics without human preference labels, rating annotations, or training of a generator model, and can be used with any LLM.
50%UnverifiedThe core of Rubricon is a four-step iterative loop: sample multiple candidate rubrics, select the most discriminative candidate, add criteria that capture overlooked requirements, and prune misaligned or superficial criteria.
50%UnverifiedRubricon uses the criterion pass rates observed on example model responses as a data-driven signal to guide rubric refinement, rather than relying solely on LLM self-critique.
50%UnverifiedRubricon was evaluated on five professional, consumer, clinical, and general-purpose LLM benchmarks.
50%UnverifiedThe alignment of scores that Rubricon-generated rubrics assign to frontier LLM responses with expert-written rubrics is 9% higher than that of the best alternative synthesis method, with compared methods including RRD, QSRG, OpenRubrics, and agent-based generation methods such as Codex.
50%UnverifiedOn HealthBench, post-training Qwen3.5-9B with Rubricon rubrics yields performance 4.5% higher than training with RRD, the strongest alternative synthetic rubric generator.
50%UnverifiedAutomatically generated rubrics may still depend on the quality of the initial LLM, and may require additional calibration in certain highly specialized domains; the 9% and 4.5% improvements were achieved only on specific benchmarks, and generalization capability still needs verification.
50%