TALON Framework: Temporal Awareness Makes Radiology Report Generation Truly Understand "Change"

The "Memory Deficit" in Radiology Report Generation

Radiology Report Generation (RRG) has long been a key research focus in medical AI. Ideally, an AI system should work like an experienced radiologist — not only interpreting the current image but also cross-referencing a patient's historical records to determine whether a lesion is persistent, improving, or newly developed.

In practice, however, most mainstream RRG models generate descriptive reports based on a single exam, or at most one prior study. This "short-term memory" limits the model's ability to perform accurate, meaningful longitudinal comparisons and makes it difficult to capture subtle interval changes — which are often the most clinically critical information.

Even recent methods that attempt to incorporate multiple prior studies typically aggregate a fixed-length history in a straightforward manner, without explicitly modeling the "role relevance" of each prior exam before fusion. In other words, the model treats all historical records equally, unable to distinguish which prior images are more valuable for the current assessment.

Radiology Report Generation (RRG) refers to the task of using deep learning models to automatically analyze medical images (such as chest X-rays and CT scans) and generate structured written reports. Early approaches borrowed from image captioning techniques, extracting visual features with convolutional neural networks before feeding them into recurrent networks or Transformer decoders for text generation. More recently, large vision-language models (such as LLaVA and MedPaLM) have also been introduced to this field. Although strong performance has been achieved on NLP metrics (BLEU, ROUGE), the core clinical value of medical reports lies in longitudinal comparison — a single-image description cannot answer the most critical clinical question: "What has changed compared to last time?" MIMIC-CXR is currently the largest publicly available chest X-ray report dataset, containing approximately 220,000 studies from Beth Israel Deaconess Medical Center, with most patients having multiple follow-up records, providing a natural training and evaluation environment for longitudinal research.

TALON: Enabling Models to Adaptively Interpret Patient History

To address these pain points, the research team proposed TALON (Temporally Aware LONgitudinal RRG framework) — a longitudinally-aware radiology report generation framework. Its core breakthrough lies in its ability to adaptively integrate variable-length patient histories, no longer constrained by fixed-length historical windows.

This means that regardless of whether a patient has one or many prior studies, TALON can flexibly handle them and filter out genuinely valuable longitudinal evidence.

Dual-Channel Temporal Fusion Module (DCTFM)

The technical foundation of TALON is the Dual-Channel Temporal Fusion Module (DCTFM). It compares the current study against each prior study through two complementary channels:

• Similarity channel: Captures persistent findings — characteristics that have consistently appeared across multiple studies.
• Change channel: Specifically identifies interval changes that occurred between time points, which represents the most clinically meaningful aspect of longitudinal comparison.

This design of "watching for what stays the same while watching for what changes" precisely mirrors how radiologists actually think when reading images: confirming that known issues remain stable while staying alert to newly emerging or worsening signs.

In the Transformer architecture, cross-attention is the standard mechanism for fusing multi-source information: the current study's features serve as queries, while historical study features serve as keys and values, allowing the model to learn "what to look up from history that is most relevant to the present." The two channels of DCTFM can be understood as a specialized decomposition of this mechanism — the similarity channel's attention weights tend to align stable anatomical structures and lesions across time points, while the change channel concentrates attention on regions where feature distributions have shifted through differential feature transformations (such as feature subtraction or contrastive loss). This "division-of-labor fusion" design is inspired by the standard practice of comparison reading in radiology: before issuing a report, physicians must simultaneously reference both current and prior images, systematically checking the stability and changes across each anatomical region.

Relevance Estimation and Adaptive Gating

Beyond the dual channels, TALON incorporates two elegant mechanisms to address the question of "which historical study to trust":

Channel-specific attention estimates the relevance of each prior study, judging its degree of contribution to the current report generation.

A learned prior-specific gate adaptively integrates valuable longitudinal evidence while suppressing redundant information. This step is critical — when a patient's history is long, large amounts of repetitive or irrelevant historical information can actually interfere with the model's judgment, and the gating mechanism serves as a "noise-filtering" process.

Experimental Validation: More History, Better Performance

The research team validated TALON on the widely-used MIMIC-CXR dataset. Results show that TALON outperforms current state-of-the-art methods on multiple clinical efficacy metrics and graph-based evaluation metrics.

A particularly noteworthy finding is that TALON's performance continues to improve as more prior studies become available. This trend directly validates DCTFM's advantage in modeling longitudinal RRG — faced with longer and more complex patient histories, TALON is not bogged down by redundant information; instead, it extracts more valuable temporal cues.

This stands in sharp contrast to traditional approaches: fixed-length aggregation methods often see performance stagnate or even decline as history grows longer, while TALON's adaptive architecture achieves a virtuous cycle of "richer history, more accurate judgment."

Automatic evaluation of radiology reports is one of the major challenges in this field. Traditional NLP metrics (BLEU, ROUGE, METEOR) measure lexical overlap, but the same condition in medical reports can be described in many equivalent ways, causing lexical metrics to often underestimate a model's true capability. Clinical efficacy metrics extract clinical entities (such as "pneumonia" or "pleural effusion") and their attributes (present/absent/uncertain) from reports, converting report generation into a classification task and computing precision, recall, and F1 — more directly measuring the diagnostic information content of reports. Graph-based evaluation metrics (such as RadGraph F1) parse imaging reports into entity-relationship graphs and use graph matching to assess the consistency between generated and reference reports at the level of medical knowledge structure. Both types of metrics are more clinically relevant than lexical metrics and represent the mainstream trend in current RRG evaluation.

Implications for Medical AI Deployment

TALON's value extends beyond improving benchmarks. It reflects an important direction in the evolution of medical imaging AI — from "single-frame recognition" to "longitudinal reasoning."

In real clinical settings, physicians never make diagnoses by looking at a single image in isolation; they synthesize a patient's complete visit history for comprehensive judgment. An AI system capable of understanding "change" is far closer to the true demands of clinical decision-making and has genuine potential to add value in high-impact scenarios such as follow-up monitoring and disease progression assessment.

Of course, as a frontier study published on arXiv, TALON currently remains at the academic validation stage. Whether its performance on MIMIC-CXR can generalize to more diverse clinical data, different imaging modalities, and real-world deployment environments remains to be tested. Nevertheless, its approach to temporal modeling undoubtedly offers an inspiring technical paradigm for longitudinal radiology report generation.