Cone Beam CT Report Generation: Clinical Reasoning Pitfalls Under a Partially Observed Objective

A Medical AI Task Distorted by Evaluation Rules

Automatically generating maxillofacial imaging reports from Cone Beam Computed Tomography (CBCT) scans might appear to be yet another routine multimodal medical AI application. But a new arXiv paper reveals a deeper problem: when only part of the evaluation objective is visible to developers, where does model optimization end up?

The researchers designed a composite scoring system that places 80% of the weight on a large language model's judgment of factual entailment, with only 20% on lexical overlap. The critical detail is that during development, only the 20% lexical component is visible — this is what the title means by a "partially observed objective." What developers can see happens to be precisely the part that doesn't drive the score.

This setup directly mirrors a common real-world dilemma: the proxy metrics we use to train and tune models (such as BLEU and METEOR) often don't align with the clinical quality goals we actually care about.

Chasing Lexical Overlap Actually Hurts Factual Accuracy

The paper's most compelling empirical findings come from a comparative experiment on 622 public cases. The researchers first reproduced the scorer's BLEU-4 and METEOR routines in pure Python, achieving machine-precision agreement with the reference implementation. They also built an offline entailment proxy model capable of distinguishing "a report written for patient A" from "a report written for patient B" with an AUC of 0.987, making direct optimization of the composite objective computationally tractable.

The results are strikingly ironic:

• Reports selected by visible lexical ranking achieved a composite score of only 0.2909;
• Reports selected by the full composite objective rose to 0.4122.

The root cause of this gap is that blindly chasing n-gram overlap drags entailment precision down from 0.522 to 0.266. In other words, making a report textually more "similar" to the reference answer actually makes it less factually reliable. This is a stark warning about the applicability of purely lexical metrics in medical report generation.

Models Learn Dictation Habits, Not Anatomical Structure

Even more alarming is the finding about what lexical metrics are actually rewarding. The researchers trained a 29-million-parameter encoder that, after fine-tuning on public data, achieved a popularity-weighted cross-fold AUC of only 0.486 across 985 statements — indistinguishable from the corpus prior, effectively learning nothing useful.

Yet nine numbers read directly from the imaging file header predicted mandibular coverage with an AUC of 0.945 and condylar coverage at 0.872. More telling still: the single variable of acquisition centre predicted sentence selection with an accuracy of 0.718 — actually higher than the image-derived model's 0.663.

This means that what lexical metrics truly reward is each institution's dictation conventions, not the anatomical information in the images. Physicians at different centers have their own fixed phrasing habits; a model that simply figures out "which center this report came from" can score highly on lexical matching — and this has nothing to do with clinical reasoning ability.

A Delivery System That Trades Constraints for Credibility

Faced with these pitfalls, the system the researchers delivered adopts a conservative, structured strategy. It outputs 8 unconditional statements, along with 5 gated statements conditioned on file-header geometry and constrained by polarity, laterality, and tooth-level consistency.

This system achieved a METEOR score of 0.3542 on 50 held-out cases from an unseen center. The number itself isn't spectacular, but given that it was achieved under the stringent condition of cross-center generalization — and built on a clear-eyed understanding of metric shortcomings — its credibility is actually higher for it.

The dataset and code have been made publicly available on GitHub (GIND123/CBCT-Clinical-Reasoner) to facilitate replication and validation by future researchers.

Implications for Medical AI Evaluation

The value of this work extends well beyond CBCT report generation. It uses rigorous quantitative evidence to illuminate several broadly applicable problems:

First, the visibility of proxy metrics shapes the direction of optimization. When developers can only optimize the visible portion of an objective, and that portion is misaligned with the true goal, models easily drift toward superficial compliance with actual failure.

Second, lexical overlap metrics can be dangerous in medical text. They reward formatting and phrasing conventions rather than factual correctness, and may even be negatively correlated with factual accuracy.

Third, simple baselines are essential sanity checks. Using "leaky features" like file header metadata and acquisition center as controls effectively exposes whether a model is genuinely performing clinical reasoning or simply exploiting statistical shortcuts.

For any team building medical imaging report generation systems, this paper is worth treating as a methodological reference — before chasing high scores, first figure out what the score is actually measuring.