New Benchmark for LLM Evaluation Across the Full Cancer Care Pathway: Strong Diagnosis Does Not Equal Strong Downstream Performance

Introduction

Large language models (LLMs) show promise in specialty medicine, but existing evaluations mostly focus on isolated clinical scenarios rather than tracking how clinical responsibility evolves along a longitudinal care pathway. A new study proposes NPC-RubricBench, a stage-aware benchmark for evaluating multi-stage clinical reasoning across the full nasopharyngeal carcinoma (NPC) care pathway. Based on longitudinal records from 405 patients at two cancer centers, the study constructed 941 patient-stage units spanning six consecutive care stages, along with 17,688 patient-specific atomic criteria across five evaluation dimensions, with each assessment strictly limited to the clinical evidence available at the decision moment.

Why is longitudinal, stage-aware evaluation needed?

Most existing LLM medical evaluations remain at the "single-point diagnosis" level—for example, giving a diagnostic conclusion based on a history or an imaging report. However, real cancer care is a long chain: from initial workup, anatomic staging, and treatment decisions to response assessment and follow-up, each stage places different demands on model capabilities and involves different error types. The core contribution of NPC-RubricBench is that it expands the evaluation unit from "a single Q&A" to a "patient-stage" sequence, allowing tracking of the same patient's model performance across different stages of disease. This design is closer to clinical reality and can expose stability issues in models along the temporal dimension.

Nasopharyngeal carcinoma (NPC) is a malignant tumor originating from the mucosal epithelium of the nasopharynx. Its incidence is significantly higher in southern China and Southeast Asia and is highly associated with Epstein-Barr virus (EBV) infection. Its care pathway is typically stage-dependent: early-stage patients are treated primarily with radiotherapy, locally advanced disease often requires concurrent chemoradiotherapy, and recurrent or metastatic disease involves chemotherapy, targeted therapy, and immunotherapy. Because the nasopharynx is anatomically deep and adjacent to the skull base and important neurovascular structures, staging (especially T staging) places extremely high demands on imaging interpretation, and response assessment after induction chemotherapy requires dynamic judgment combining nasopharyngoscopy, MRI, and EBV DNA load. This cycle—"staging determines treatment, and treatment in turn changes the basis for subsequent staging"—makes NPC an ideal disease model for testing LLMs' longitudinal reasoning ability: the model must not only answer correctly at a given time point but also understand how decisions at a previous stage constrain subsequent options.

Key finding: disconnection between diagnostic performance and downstream care

The study evaluated 30 general-purpose and medical-specific LLMs and reached a key conclusion: high diagnostic performance does not automatically translate into better downstream care performance. In other words, a model may achieve considerable accuracy in anatomic staging but frequently omit key steps in initial workup or post-treatment follow-up. This finding directly challenges the naive assumption that "higher diagnostic accuracy means a more reliable model."

Failure modes reverse across stages

A more fine-grained analysis showed that dominant failure modes reverse across stages: the anatomic staging stage was dominated by explicit decision errors, accounting for 48.2%; whereas the initial workup and post-treatment follow-up stages were dominated by decision omissions, accounting for 33.4% and 30.4%, respectively. This means that the same model has distinct weaknesses upstream and downstream in the chain, and stage-adaptive safety mechanisms are more targeted than global optimization.

Omission problem in medically adapted models

Among the evaluated model set, medically adapted models had a higher omission rate than general-purpose models. One possible explanation is that medical fine-tuning may make models more inclined to give concise "plausible-looking" answers, thereby skipping certain necessary workup or follow-up steps. This counterintuitive result suggests that medical domain adaptation does not necessarily improve reliability across the full workflow and requires fine-grained diagnosis combined with stage characteristics.

Atomic criterion refers to decomposing a complete clinical assessment into the smallest independently judgeable single requirement—for example, "whether the staging result of nasopharyngeal MRI was recorded" or "whether hearing and mouth opening were assessed before radiotherapy." Compared with overall scoring or coarse grading, atomic criteria have the advantages of attributability and reproducibility: if a model output satisfies only part of the requirements, evaluators can precisely locate which step was omitted. NPC-RubricBench generates patient-specific atomic criteria for each patient at a specific stage, meaning the criteria vary with the patient's history, staging, and prior treatment rather than applying a generic checklist. This approach transforms "scoring" into "item-by-item verification of a responsibility checklist," allowing differences among models to be tied to specific clinical actions and enabling separate accounting of "omissions" and "errors"—the methodological basis for the study's finding that failure modes reverse across stages.

Key trade-off in prompt granularity

The study also examined the impact of query granularity on model performance and revealed a key specification trade-off: broad open-ended requests reduced decision quality across all 30 models; more detailed checklist-style prompts improved decision quality but reduced evidence faithfulness and safety compliance. In other words, the more "hand-holding" the prompt, the more likely the model is to give a seemingly complete decision, but the more likely it is to depart from the given evidence or ignore safety boundaries. This imposes practical constraints on prompt design in clinical deployment: one cannot simply pursue decision quality at the expense of faithfulness and safety.

Evidence faithfulness measures whether claims in model outputs are supported by the information provided in the input medical records and examination reports, rather than coming from the model's own parametric memory or being fabricated. In clinical settings, this metric is particularly critical: if a model gives seemingly correct recommendations but cites nonexistent examination results or treats past history as current evidence, it may mislead subsequent decisions. The reason checklist-style prompts reduce evidence faithfulness is that itemized requirements induce the model to "complete" each item even when the original records contain no corresponding basis. Like safety compliance, faithfulness is a dimension that is difficult to indirectly reflect through downstream task accuracy and must be measured separately—which is also the reason the stage-aware benchmark sets multi-dimensional atomic criteria in addition to decision quality.

Implications for multi-stage medical AI

The conclusions of NPC-RubricBench point to a clear direction: static diagnostic benchmarks are insufficient to ensure consistent clinical reliability of models along the temporal dimension. Multi-stage medical AI requires stage-adaptive safeguards, such as differentiated omission detection, evidence constraints, and safety verification rules for different care stages. For teams hoping to embed LLMs into cancer care workflows, the evaluation focus should shift from "single-point accuracy" to "the distribution of error types across the full responsibility chain," and targeted intervention strategies should be designed accordingly.

Using nasopharyngeal carcinoma as a well-defined disease context, the benchmark's methodology—patient-specific atomic criteria, decision-moment evidence constraints, and stage-aware evaluation—has the potential to be extended to other cancer types and chronic disease management scenarios across the full care pathway.

Key takeaways

• NPC-RubricBench covers 405 patients, 941 patient-stage units, and 17,688 atomic criteria, enabling longitudinal stage-aware evaluation.
• Evaluation of 30 LLMs shows that strong diagnostic performance does not mean good downstream care performance, and failure modes reverse across stages.
• Medically adapted models have higher decision omission rates than general-purpose models; medical fine-tuning does not equal full-workflow reliability.
• There is a trade-off in prompt granularity: open-ended requests reduce decision quality, while checklist-style prompts improve decision quality but reduce evidence faithfulness and safety compliance.
• Multi-stage medical AI requires stage-adaptive safety mechanisms; static diagnostic benchmarks are insufficient to ensure long-term clinical reliability.

One-line Summary

A new benchmark reveals that strong LLM diagnostic performance does not mean reliable full-pathway care, and failure modes reverse across care stages.

Paragraph Summary

The study proposes NPC-RubricBench, a stage-aware benchmark for the full care pathway of nasopharyngeal carcinoma. Based on longitudinal records of 405 patients at two cancer centers, it constructs 941 patient-stage units spanning six consecutive stages and 17,688 patient-specific atomic criteria, with each assessment strictly limited to clinical evidence visible at the decision moment. Evaluation of 30 general-purpose and medical-specific LLMs shows that high diagnostic performance does not translate into better downstream care performance; failure modes reverse across stages, with the staging stage dominated by explicit decision errors (48.2%) and the initial workup and follow-up stages dominated by decision omissions (33.4%, 30.4%). Medically adapted models have higher omission rates than general-purpose models, suggesting that medical fine-tuning does not necessarily improve full-workflow reliability. Prompt granularity also involves a trade-off: open-ended requests reduce decision quality across all models, while checklist-style prompts improve decision quality but reduce evidence faithfulness and safety compliance.