Scoring Rubric
A structured set of scoring rules in the evaluation domain used to decompose complex outputs into multiple assessable dimensions; in PMR it is learned automatically through co-evolution rather than specified manually.
Core Facts
Scoring criteria (Rubrics) originally referred to detailed tables used in teaching to evaluate assignments along multiple dimensions, and in recent years have been introduced into the large model field, referring to breaking down vague overall quality into several independently judgeable items, each with clear judgment conditions.
A rubric is a list of criteria describing "what makes a good answer," used to score each dimension of model responses in open-ended tasks item by item.
Timeline (last 90 days)
PMR's rubrics are not manually set but co-evolve with the memory policy: the general layer captures common failure modes across users, while the user-adaptive layer learns individual preferences from a small amount of user interaction data.
The quality of the rubric directly determines the verification ceiling, and omitted requirements cannot be recovered by subsequent exploration, so constructing the rubric in the first stage is the most critical step in the entire pipeline.
VeriCUA's pipeline has two stages: first, it checks the available context and builds a task rubric; then, for requirements not yet resolved, it chooses GUI inspection or code inspection and synthesizes the findings of both under explicit evidence rules.
A rubric is a list of criteria describing "what makes a good answer," used to score each dimension of model responses in open-ended tasks item by item.
For non-verifiable data, the Microsoft team devoted senior manpower to building scoring rubrics; a single open question may correspond to 50 detailed criteria, each of which must be clear, structured and scorable.
The researchers released the scoring rubric, judge prompts, and evaluation code, providing the community with a reproducible risk assessment tool.
In the blind review results, 96 rows showed differing interpretations of the scoring criteria.
Agreement metrics themselves only measure whether the results are the same, and do not distinguish whether that sameness stems from a shared understanding of the criteria or from mutual influence among reviewers.
Practical recommendations include: conducting regular blind reviews, clarifying scoring criteria, analyzing disagreement cases, and rotating reviewers.
If the judge model and the evaluated model come from the same vendor or share training data, there may be systematic bias.
7 more timeline events
All Facts (16)
Scoring criteria (Rubrics) originally referred to detailed tables used in teaching to evaluate assignments along multiple dimensions, and in recent years have been introduced into the large model field, referring to breaking down vague overall quality into several independently judgeable items, each with clear judgment conditions.
75%Partially VerifiedA rubric is a list of criteria describing "what makes a good answer," used to score each dimension of model responses in open-ended tasks item by item.
75%UnverifiedPMR's rubrics are not manually set but co-evolve with the memory policy: the general layer captures common failure modes across users, while the user-adaptive layer learns individual preferences from a small amount of user interaction data.
50%UnverifiedThe quality of the rubric directly determines the verification ceiling, and omitted requirements cannot be recovered by subsequent exploration, so constructing the rubric in the first stage is the most critical step in the entire pipeline.
50%UnverifiedVeriCUA's pipeline has two stages: first, it checks the available context and builds a task rubric; then, for requirements not yet resolved, it chooses GUI inspection or code inspection and synthesizes the findings of both under explicit evidence rules.
50%UnverifiedFor non-verifiable data, the Microsoft team devoted senior manpower to building scoring rubrics; a single open question may correspond to 50 detailed criteria, each of which must be clear, structured and scorable.
50%UnverifiedAgreement metrics themselves only measure whether the results are the same, and do not distinguish whether that sameness stems from a shared understanding of the criteria or from mutual influence among reviewers.
50%UnverifiedIn the blind review results, 96 rows showed differing interpretations of the scoring criteria.
50%UnverifiedPractical recommendations include: conducting regular blind reviews, clarifying scoring criteria, analyzing disagreement cases, and rotating reviewers.
50%UnverifiedIf the judge model and the evaluated model come from the same vendor or share training data, there may be systematic bias.
50%UnverifiedThe author suggests the minimum viable path is: first pick a task for which someone is genuinely willing to pay, first write out the scoring criteria, and then build the commitment mechanism.
50%UnverifiedThe test scoring used a weighted formula: 0.20×readability + 0.35×design execution and style match + 0.25×typography and completeness + 0.20×attractiveness, rounded to one decimal place.
50%UnverifiedThe quality of the scoring criteria themselves depends on the judgment of their designers, and how to control the completeness and bias of the decomposed dimensions remains a question requiring long-term validation.
50%UnverifiedAccuracy loss caused by errors in the scoring rubric itself (such as wrong weights or anchors not matching the scale) is greater than the differences between evaluation frameworks.
50%UnverifiedThe rubrics generated by this framework are grounded in real evidence retrieved for answering the question (evidence-based) and employ Retrieval-Augmented Generation (RAG) technology
50%UnverifiedThe researchers released the scoring rubric, judge prompts, and evaluation code, providing the community with a reproducible risk assessment tool.
50%