RubricAtlas: Mining Type-Level Rubrics from Expert Reports to Evaluate Deep Research Agents

Deep research agents synthesize information into analytical reports through multi-step web retrieval. When evaluating this kind of output, merely checking whether the relevant facts are present is far from enough—you also need to judge what analysis they actually deliver. To this end, researchers propose RubricAtlas, an evaluation framework that automatically mines "type-level" analytical requirements from expert reports and organizes its checks around recurring analytical questions.

Why Task-Level Checklists Fall Short

Binary rubrics written for a single query can make each requirement explicit, but task-level checklists often leave the overall analytical structure "under-examined"—a checklist may cover the factual points while missing the core analytical questions that this kind of research is supposed to answer.

RubricAtlas's approach is this: rather than writing criteria separately for each task, it distills shared analytical requirements from expert reports on different subjects within the same research type. These requirements give tasks of the same kind a unified yardstick for "what should be analyzed" without prescribing specific answers, thereby avoiding turning scoring into a mechanical comparison against some reference report.

How Type-Level Rubrics Are Constructed

The framework's construction scale is clearly stated in its paper: it is based on 3,079 Chinese financial research reports, covering 99 research types.

Mechanistically, RubricAtlas links binary checkpoints to specific passages in the reports, and evaluates "completeness" and "argumentative validity" separately. This point is crucial: a report may cover all the analytical points but fail to make a sound argument, or it may argue rigorously yet miss a key analytical dimension—the two should not be conflated into a single score.

According to the paper, reference-report selection experiments show that a single report covers, on average, only about half of the analytical questions of its type. This means that if evaluation criteria were extracted from just one reference report, the other half of the requirements would be systematically missed.

The distinction between "type-level" and "task-level" deserves elaboration. Task-level rubrics are written for a specific query—for example, "analyze the reasons for a certain company's 2023 gross margin change"—and their checkpoints are tied to that company in that year. Type-level rubrics, by contrast, are induced from multiple expert reports on different subjects within the same research type (e.g., "attribution analysis of a listed company's profitability"), distilling analytical requirements that are independent of the specific subject, such as "does it decompose the volume-price factors of the gross margin change?" or "does it distinguish industry-wide factors from company-specific factors?"

The direct consequence is that the criteria are transferable: the same set of checkpoints can be used to evaluate reports on any subject within that type, without having to rewrite a scoring sheet for each new query. At the same time, because the requirements come from the commonalities across multiple reports, it naturally avoids the bias of "aligning to just one reference answer." The cost is that the criteria are more abstract and need to be linked to specific evidence passages in the reports to be grounded as decidable binary checkpoints—which is precisely why RubricAtlas links checkpoints to passages.

Evaluation Reliability: Cross-Validation of Two Judge Pipelines

In tests on ten agents, two judge pipelines achieved κ = 0.71 checkpoint agreement and Kendall τ = 0.87 agent-ranking agreement. These two metrics correspond respectively to "how accurately individual checkpoints are judged" and "how stable the overall ranking is," indicating that the framework has usable consistency at both the fine-grained judgment and the macro-level ordering.

Another notable finding is that selecting requirements from different reports changes evaluation scores and agent rankings. This suggests that the choice of rubric source is itself a variable that needs to be taken seriously, rather than an inconsequential implementation detail.

Cohen's κ (Cohen's kappa) is a statistic measuring the degree of agreement between two raters (or two judge pipelines) on categorical judgments. Its calculation removes the "chance agreement" portion: if both raters randomly assigned "yes/no" to checkpoints, there would still be some proportion of accidental matches, and κ subtracts this from the observed agreement rate. Values typically range from −1 to 1, with 0 meaning no better than chance; 0.6–0.8 is generally considered "substantial agreement." Thus κ = 0.71 means the two pipelines agree on whether individual checkpoints are satisfied at a rate clearly above chance.

Kendall τ (Kendall's rank correlation coefficient) is used to compare the consistency of two sets of rankings, ranging from −1 (completely opposite) to 1 (completely consistent). In this case, it measures whether the two judge pipelines give matching rankings to the ten agents. τ = 0.87 indicates that even if individual checkpoint judgments differ, the two pipelines remain highly stable in their macro-level ordering of "who is better and who is worse." The two are complementary: κ reflects the reliability of fine-grained judgments, while τ reflects the robustness of the overall evaluation conclusion.

Controlled Experiments: The Different Effects of Deletion and Padding

The paper uses controlled experiments to test whether the rubrics truly have discriminative power.

• Controlled deletions: after removing the relevant analysis from a report, the score on the corresponding dimension drops, showing that the checkpoints do capture the target content.
• Padding: after adding irrelevant content to a report, no score increase was detected, showing that the framework is not easily fooled by word-count-padding filler.

Together, the results of these two directions support one conclusion: the rubrics measure analytical completeness, not text length or surface richness.

Implications for Deep Research Evaluation

RubricAtlas's core contribution is to move the anchor of evaluation up from "a single reference report" to "the research type." The benefit is that analytical completeness can be compared across tasks of the same kind, while every specific judgment can still be traced back to evidence passages in the report.

For teams building or using deep research agents, this offers a way of thinking: rather than pursuing similarity to some "standard answer," it is better to specify which analytical questions this kind of research should answer, and then examine coverage and argumentative quality separately.

It should be noted that the framework's construction corpus is concentrated in Chinese financial research reports, and its applicability to other domains and languages has not yet been concluded in the original material.

One-line Summary

RubricAtlas mines type-level analytical requirements from expert reports to evaluate the analytical completeness of deep research agents.

Paragraph Summary

The article introduces RubricAtlas, a framework for evaluating deep research agents. To address the problem that task-level binary checklists tend to miss the overall analytical structure, the framework distills shared type-level analytical requirements from multiple expert reports within the same research type, links checkpoints to evidence passages in the reports, and evaluates completeness and argumentative validity separately. The framework is built on 3,079 Chinese financial research reports covering 99 research types; a single reference report covers, on average, only about half of the analytical questions of its type. On ten agents, two judge pipelines achieved κ = 0.71 checkpoint agreement and Kendall τ = 0.87 ranking agreement. Controlled deletion experiments lower scores on the corresponding dimensions, while padding with irrelevant content yields no detectable score increase, indicating that the framework measures analytical completeness rather than text length. The article also notes that the choice of reference reports changes scores and rankings, and that the framework's applicability to other domains and languages remains to be verified.