[KongchangAI]
· 2 min read· 1,358 words

Can Decision Models Extract Cancer Pathology Reports? Accuracy and Failure Mode Evaluation

Can Decision Models Extract Cancer Pathology Reports? Accuracy and Failure Mode Evaluation

Decision models match same-scale generative models in accuracy for pathology report extraction and are four times faster, but local calibration and complex fields remain limited.

The study systematically evaluated two open-source decision models (Nimble 9B, Tev1 4B) and two generative models (local Qwen3.5 9B, cloud GLM-5.3-flash) on a 25-field extraction task using 1,113 oral cancer pathology reports. Tev1 achieved an average field accuracy of 0.917, on par with local Qwen3.5 but about four times faster; Nimble scored 0.862, while cloud GLM-5.3-flash reached 0.986. Decision models were comparable to the cloud model on simple binary fields but weaker on binned measured values, fields where the correct answer was not reported, and complex decisions. Probabilities were well calibrated overall but could not flag systematic errors within a single field. Routing answers with probability below 0.94 to the cloud model raised Tev1's accuracy to 0.982, with a 38% escalation rate, constituting a quantifiable local-cloud collaboration scheme.

Decision models are a new class of language models: instead of generating free text, they select an answer for each question from a fixed list of options and return a probability for each option. These models had not previously been systematically evaluated on clinical text. A study on oral cancer pathology reports fills this gap, comparing two open-source decision models with two generative models on a structured field extraction task.

Screenshot of the decision model evaluation study source

Study Design and Comparison Subjects

The study selected 1,113 oral cancer pathology reports from a single center, asked the models to extract 25 fields, and compared results against a prospectively maintained clinical database. The models evaluated included:

  • Decision models: Nimble 9B and Tev1 4B, both open-weight
  • Generative models: Qwen3.5 9B (run locally) and GLM-5.3-flash (cloud API)

What makes this design notable is that it does not stop at "which model is more accurate" but examines accuracy, speed, probability calibration, and failure modes within a single framework. For clinical deployment, the latter two are often more critical than raw accuracy alone.

Decision models differ fundamentally from generative language models in their output space: generative models sample step by step over a vocabulary and freely generate text, whereas decision models directly score a fixed set of candidate answers for a given question and normalize them into a probability distribution—essentially turning "extraction" into "constrained classification." This design has two direct consequences: first, outputs are inherently structured and can be mapped directly to database fields, eliminating parsing and post-processing; second, each answer comes with a probability, making it easy to set thresholds for human review or routing. The cost is limited flexibility—when the correct answer is not among the candidates, or when meta-information such as "not mentioned in the report" needs to be expressed, the model lacks the room for free generation, which is precisely the deeper reason why decision models performed poorly on missing-value fields in this study.

Accuracy and Speed: The True Level of Decision Models

Tev1 achieved an average field accuracy of 0.917 (95% CI 0.913–0.920), statistically indistinguishable from locally run Qwen3.5 9B (0.918, p = 0.26), but about four times faster—15 seconds versus 61 seconds per report. Nimble's accuracy was 0.862, while cloud-based GLM-5.3-flash reached 0.986.

These numbers reveal an easily overlooked fact: decision models can match generative models of the same scale in accuracy, and local deployment brings a clear speed advantage. But if cloud large models are used as the upper-bound reference, open-source decision models still lag behind.

One detail: both decision models performed comparably to GLM-5.3-flash on simple binary-type fields. This suggests their weakness lies not in "making judgments" per se, but in specific types of tasks.

Failure Modes: Where They Are Weak

The study explicitly identified three scenarios where decision models performed poorly:

  1. Measured values binned into categories—errors are common when mapping continuous values to discrete intervals
  2. Fields where the correct answer is not reported—when information is missing from the report, the model struggles to correctly express "not mentioned"
  3. Some complex decisions, such as whether margins are involved

Among these, the most valuable finding concerns the diagnosis of Nimble. Researchers found that Nimble's poorer performance stemmed mainly from the configuration of "asking 25 questions in a single request": when changed to asking only two questions per request, its accuracy improved. This suggests decision models are sensitive to the complexity of a single request, and that configuration itself is a performance variable.

Binning refers to mapping continuous measured values (such as tumor size or depth of invasion) into discrete categories according to predefined intervals. For decision models, this step is especially error-prone: the model must select one interval among the candidates, and determining which bin a value near a boundary falls into requires precise numerical comparison—precisely a weakness of language-model-style architectures; moreover, if the binning rules themselves are not clearly expressed among the candidates, the model has no way to infer interval boundaries. In contrast, generative models can directly transcribe the raw value and leave binning to downstream programs, thereby bypassing this source of error. This suggests that in structured extraction pipelines, continuous variables should retain their raw values and be discretized by a rule engine rather than having the model directly output categories.

Probability Calibration: Globally Trustworthy, Locally Unreliable

A major selling point of decision models is returning a probability for each option. The study found that, when aggregated by field, probabilities were well calibrated, with expected calibration errors of 0.022 and 0.026, respectively. But the problem is that these probabilities cannot flag systematic errors within a single field—that is, a high probability assigned to an answer for a given field does not mean that field's answer is reliable.

This limitation directly affects the practical value of probabilities. For scenarios requiring human review, if high confidence does not correspond to high accuracy, threshold-based routing strategies will fail.

Expected Calibration Error (ECE) is a common metric for measuring the trustworthiness of a model's output probabilities: it divides predicted probabilities into several intervals (bins), compares the gap between the average predicted probability and the true accuracy within each interval, and then takes a sample-size-weighted sum. The lower the ECE, the more it means that "when the model says it is 90% confident, it is actually right about 90% of the time." Values of 0.022 and 0.026 are relatively low, meaning probabilities are trustworthy overall. But ECE is an aggregate metric: it reflects only overall average deviation and inherently cannot reveal local issues such as "whether a high probability for a specific field is reliable." Therefore, in this study, good overall calibration and the inability to flag systematic errors within a single field are not contradictory—the former is a macro-level statistical property, while the latter is a micro-level interpretability gap, and the two require different evaluation methods (such as field-stratified calibration and confidence-accuracy curves) to characterize separately.

A Feasible Path for Human-AI Collaboration

The study simulated a strategy: routing answers with probability below 0.94 to GLM-5.3-flash for processing. As a result, Tev1's accuracy rose to 0.982, while 38% of answers were escalated.

The significance of this result is that it provides a quantifiable compromise: use the local decision model to quickly process most fields and hand uncertain parts to a stronger cloud model. The 38% escalation rate means most work can still be done locally, preserving the speed advantage while bringing accuracy close to that of the cloud model.

Implications for Clinical Deployment

The study's recommendations are quite pragmatic: decision models can quickly extract explicitly stated findings from pathology reports on local hardware and return useful probabilities. But the prerequisite is validation under the actual configuration in use, limiting the scope of application to fields where the model is reliable, and handling numerical information in conjunction with other methods.

For healthcare AI practitioners, the core message of this study is: model selection cannot rely on average accuracy alone. Field-level reliability, request configuration, and the interpretable scope of probability calibration jointly determine whether a model can enter clinical workflows. Decision models are not a simple substitute for generative models, but a tool that needs to be tailored to the scenario.

One-line Summary

Decision models match same-scale generative models in accuracy for pathology report extraction and are four times faster, but local calibration and complex fields remain limited.

Paragraph Summary

The study systematically evaluated two open-source decision models (Nimble 9B, Tev1 4B) and two generative models (local Qwen3.5 9B, cloud GLM-5.3-flash) on a 25-field extraction task using 1,113 oral cancer pathology reports. Tev1 achieved an average field accuracy of 0.917, on par with local Qwen3.5 but about four times faster; Nimble scored 0.862, while cloud GLM-5.3-flash reached 0.986. Decision models were comparable to the cloud model on simple binary fields but weaker on binned measured values, fields where the correct answer was not reported, and complex decisions. Probabilities were well calibrated overall but could not flag systematic errors within a single field. Routing answers with probability below 0.94 to the cloud model raised Tev1's accuracy to 0.982, with a 38% escalation rate, constituting a quantifiable local-cloud collaboration scheme.

#decision models#pathology report extraction#clinical NLP#generative models#probability calibration#human-AI collaboration#cancer pathology
Share:

Related articles