[KongchangAI]
ConceptRubric

Scoring Rubric

A structured set of scoring rules in the evaluation domain used to decompose complex outputs into multiple assessable dimensions; in PMR it is learned automatically through co-evolution rather than specified manually.

Core Facts

Timeline (last 90 days)

Nov 11

PMR's rubrics are not manually set but co-evolve with the memory policy: the general layer captures common failure modes across users, while the user-adaptive layer learns individual preferences from a small amount of user interaction data.

Unverified50%
Oct 30

The quality of the rubric directly determines the verification ceiling, and omitted requirements cannot be recovered by subsequent exploration, so constructing the rubric in the first stage is the most critical step in the entire pipeline.

Unverified50%
Oct 30

VeriCUA's pipeline has two stages: first, it checks the available context and builds a task rubric; then, for requirements not yet resolved, it chooses GUI inspection or code inspection and synthesizes the findings of both under explicit evidence rules.

Unverified50%
Oct 30

A rubric is a list of criteria describing "what makes a good answer," used to score each dimension of model responses in open-ended tasks item by item.

Partially Verified75%
Oct 27

For non-verifiable data, the Microsoft team devoted senior manpower to building scoring rubrics; a single open question may correspond to 50 detailed criteria, each of which must be clear, structured and scorable.

Unverified50%
Oct 20

The researchers released the scoring rubric, judge prompts, and evaluation code, providing the community with a reproducible risk assessment tool.

Unverified50%
Oct 18

In the blind review results, 96 rows showed differing interpretations of the scoring criteria.

Unverified50%
Oct 18

Agreement metrics themselves only measure whether the results are the same, and do not distinguish whether that sameness stems from a shared understanding of the criteria or from mutual influence among reviewers.

Unverified50%
Oct 18

Practical recommendations include: conducting regular blind reviews, clarifying scoring criteria, analyzing disagreement cases, and rotating reviewers.

Unverified50%
Oct 16

If the judge model and the evaluated model come from the same vendor or share training data, there may be systematic bias.

Unverified50%

7 more timeline events

All Facts (16)

Verified

Scoring criteria (Rubrics) originally referred to detailed tables used in teaching to evaluate assignments along multiple dimensions, and in recent years have been introduced into the large model field, referring to breaking down vague overall quality into several independently judgeable items, each with clear judgment conditions.

75%
Partially Verified

A rubric is a list of criteria describing "what makes a good answer," used to score each dimension of model responses in open-ended tasks item by item.

75%
Unverified

PMR's rubrics are not manually set but co-evolve with the memory policy: the general layer captures common failure modes across users, while the user-adaptive layer learns individual preferences from a small amount of user interaction data.

50%
Unverified

The quality of the rubric directly determines the verification ceiling, and omitted requirements cannot be recovered by subsequent exploration, so constructing the rubric in the first stage is the most critical step in the entire pipeline.

50%
Unverified

VeriCUA's pipeline has two stages: first, it checks the available context and builds a task rubric; then, for requirements not yet resolved, it chooses GUI inspection or code inspection and synthesizes the findings of both under explicit evidence rules.

50%
Unverified

For non-verifiable data, the Microsoft team devoted senior manpower to building scoring rubrics; a single open question may correspond to 50 detailed criteria, each of which must be clear, structured and scorable.

50%
Unverified

Agreement metrics themselves only measure whether the results are the same, and do not distinguish whether that sameness stems from a shared understanding of the criteria or from mutual influence among reviewers.

50%
Unverified

In the blind review results, 96 rows showed differing interpretations of the scoring criteria.

50%
Unverified

Practical recommendations include: conducting regular blind reviews, clarifying scoring criteria, analyzing disagreement cases, and rotating reviewers.

50%
Unverified

If the judge model and the evaluated model come from the same vendor or share training data, there may be systematic bias.

50%
Unverified

The author suggests the minimum viable path is: first pick a task for which someone is genuinely willing to pay, first write out the scoring criteria, and then build the commitment mechanism.

50%
Unverified

The test scoring used a weighted formula: 0.20×readability + 0.35×design execution and style match + 0.25×typography and completeness + 0.20×attractiveness, rounded to one decimal place.

50%
Unverified

The quality of the scoring criteria themselves depends on the judgment of their designers, and how to control the completeness and bias of the decomposed dimensions remains a question requiring long-term validation.

50%
Unverified

Accuracy loss caused by errors in the scoring rubric itself (such as wrong weights or anchors not matching the scale) is greater than the differences between evaluation frameworks.

50%
Unverified

The rubrics generated by this framework are grounded in real evidence retrieved for answering the question (evidence-based) and employ Retrieval-Augmented Generation (RAG) technology

50%
Unverified

The researchers released the scoring rubric, judge prompts, and evaluation code, providing the community with a reproducible risk assessment tool.

50%

Source Articles