Fusing Heterogeneous Echocardiography AI Outputs in Language Space with Supervised LLMs

Echocardiography AI models often specialize—some excel at a particular view, others at a specific measurement—and used individually their predictive power is limited, so combining them is a natural idea. But heterogeneous models produce a wide variety of output formats: binary decisions, probability values, confidence scores, fixed-length feature vectors. Getting them to "talk" to each other is not easy. A new study proposes using a supervised large language model (LLM) for late fusion, aggregating these heterogeneous outputs directly in language space, eliminating the need for model-specific output conversion and medical feature engineering.

The Challenge of Fusing Heterogeneous Outputs

Multi-view, multi-target echocardiography models each capture complementary information. The problem is that traditional ensemble learning requires outputs to be unified into machine-processable numerical forms: a binary classification result, a probability, a confidence score, or a fixed-length feature vector.

This approach comes at a considerable cost. Converting structured predictions into numerical values typically requires medical knowledge to design conversion rules and feature engineering, and the conversion process itself may lose information. The original paper notes that heterogeneous models "typically require" such conversions—and that is precisely the step this research aims to bypass.

Late Fusion is a class of strategies in ensemble learning where each sub-model first completes its prediction independently, and results are then merged at the output level; in contrast, early fusion concatenates multi-source information at the feature level and trains jointly. The advantage of late fusion is that each model can be heterogeneous and developed and updated independently, which suits the reality of medical imaging where models are built separately by view or by measurement. The cost is that the fusion layer must handle outputs in inconsistent formats—common approaches include voting, weighted averaging, and stacking—and the weights or meta-classifiers typically require additional annotated data to learn. This paper replaces the traditional numerical fusion with a supervised LLM; it fundamentally remains within the late fusion framework, except that merging happens in language space rather than numerical space.

Patient-level cross-validation differs from the common image- or study-level splits: a single patient may have multiple study records, and if studies are randomly split, images from the same patient could appear in both the training and test sets, causing information leakage and overestimated performance. Patient-level splitting ensures that all studies from the same patient fall into the same fold, so evaluation results better reflect the generalization performance expected in real clinical deployment. This paper uses three folds rather than the more common five or ten, which is usually related to limited sample size (646 patients); fewer folds allow more training data per fold, but the variance of a single estimate is also larger.

Letting the LLM Fuse Directly in Language Space

The research team built a late fusion module based on a supervised LLM. Its input is not numerical vectors but the "predicted finding statements" and "predicted measurements" output by each view model—that is, conclusions in natural language form—from which the module learns a case-level fusion function.

The elegance of this design lies in language serving as a unified interface. As long as the outputs of different upstream models can be expressed in sentences, they can enter the same framework without designing a separate converter for each model. The fusion function is obtained through task-specific training rather than relying on the zero-shot capabilities of a general-purpose LLM.

Zero-shot capability refers to a model directly completing a new task using only general knowledge acquired from pre-training, without having seen task-specific training samples. General-purpose LLMs perform well in text understanding and reasoning, which can easily lead to the misconception that they can serve as fusion modules "out of the box." But a fusion function is essentially a mapping that must be learned from data: which finding statements corroborate each other, which measurements should be prioritized, how to weigh conflicts—these patterns all depend on the specific task and data structure, and the priors of a general-purpose model may not align. The comparative experiments in this paper precisely place "task-specific supervised training" alongside "zero-shot invocation of a general-purpose LLM," thereby attributing the performance gain to supervised learning rather than model scale or general reasoning ability. This is also a common experimental design when evaluating LLM applications—distinguishing whether "capability comes from pre-training" or "capability comes from fine-tuning."

What the Validation Results Show

The study performed patient-level three-fold cross-validation on 859 studies from 646 patients. Results showed that the method integrated heterogeneous outputs within a single framework, performing comparably to the best reference strategy evaluated and outperforming a general-purpose LLM without task-specific training.

The latter point is especially critical: it shows that fusion capability comes from supervised training, not from the LLM's own general reasoning. In other words, "fusing with an LLM" is not an out-of-the-box solution—task-specific learning remains indispensable.

Implications for Medical AI Engineering

The value of this work is not in achieving higher metrics but in providing a more convenient engineering path: upstream models can be heterogeneous, outputs can remain in natural language form, and the fusion layer is handled by a supervised LLM, thereby reducing dependence on model-specific output conversion and medical-knowledge-driven feature engineering.

Of course, the validation scale is limited, and the robustness of the conclusions still needs to be tested with larger datasets and more model combinations. But it suggests a direction—when medical AI systems are assembled from multiple specialty models, language space may be a more natural glue layer than numerical vectors.

Medical-knowledge-driven feature engineering in this context refers to the manual design process in which clinical experts convert model outputs into usable numerical values—for example, specifying how a confidence score maps to a weight, or how a measurement exceeding a threshold is encoded as a categorical flag. Such rules are highly interpretable but costly to design, difficult to transfer to new model combinations, and discretization itself discards continuous information. Having the LLM directly read conclusions in natural language form amounts to letting the model learn "how to interpret a predicted statement" on its own during training, thereby shifting expert knowledge from handcrafted rules to supervisory signals in the data. It should be noted that this does not eliminate the need for medical knowledge but moves it forward to the data annotation and task definition stages.