Dual Task Vector Merging: Boosting Low-Resource Alzheimer's Detection with Sentiment and ASR Knowledge

Title

Dual Task Vector Merging: Boosting Low-Resource Alzheimer's Detection with Sentiment and ASR Knowledge

Meta Description

The paper proposes Dual Task Vector Merging, injecting task vectors from sentiment and ASR fine-tuning into a frozen wav2vec 2.0, updating less than 0.1% of parameters and outperforming frozen probes, LoRA, and full fine-tuning on ADReSS, ADReSSo, and PROCESS, with stable gains in low-resource settings.

Article Content

Detecting Alzheimer's disease (AD) from speech relies on both paralinguistic and linguistic cues, but model training is often constrained by insufficient data. Fine-tuning large pre-trained speech encoders easily leads to overfitting, while frozen probes fail to fully exploit the task-relevant knowledge embedded in the encoder. A new paper proposes Dual Task Vector Merging, a parameter-efficient method that adapts a frozen wav2vec 2.0 backbone by injecting knowledge from sentiment and ASR tasks, significantly improving AD detection performance in low-resource scenarios.

Background: The Data Dilemma in AD Speech Detection

AD speech detection faces a classic small-sample challenge. Common datasets such as ADReSS and ADReSSo are limited in scale, making direct fine-tuning of large pre-trained models (e.g., wav2vec 2.0) highly prone to overfitting; conversely, completely freezing the encoder and training only a classification head wastes the rich speech knowledge contained in the encoder. Existing parameter-efficient fine-tuning methods like LoRA can alleviate some issues, but they do not introduce external prior knowledge tailored to the AD task.

The paper's core insight is that sentiment recognition and automatic speech recognition (ASR) tasks share part of the speech representation foundation with AD detection. The sentiment task captures paralinguistic features (e.g., prosody, voice quality), while the ASR task captures linguistic content—together corresponding exactly to the two information dimensions required for AD detection. If the knowledge from these tasks can be injected into the AD model in a lightweight manner, it could provide effective regularization when data is scarce.

Method: How Task Vectors Are Merged

Constructing Task Vectors

A task vector refers to the difference in model weights before and after fine-tuning, encoding the knowledge learned on a specific task. The authors fine-tune wav2vec 2.0 separately on sentiment and ASR datasets, compute the weight increments for each Transformer layer, and obtain two task vectors.

wav2vec 2.0 is a self-supervised speech representation model proposed by Meta (formerly Facebook) in 2020. It consists of a multi-layer convolutional feature encoder and 12 (base) or 24 (large) Transformer layers. It is first pre-trained on massive unlabeled speech via contrastive learning to predict masked speech segments, thereby learning general speech representations. Its key property: even without any fine-tuning, intermediate layer outputs have been shown to linearly probe various types of information such as phonemes, speakers, and sentiment—which is precisely why it is used as a frozen backbone in low-resource tasks. The backbone used in this paper is wav2vec 2.0 base, whose each Transformer layer has an output dimension of 768, and the task vectors operate on these layers.

Layer-Wise Merging with Extremely Low Parameter Updates

During AD training, the wav2vec 2.0 backbone is frozen, and the two task vectors are injected into each Transformer layer with layer-wise learnable merging coefficients. The entire training process optimizes only these merging coefficients and a linear classification head, with trainable parameters less than 0.1% of full fine-tuning. This design preserves the general capabilities of the pre-trained encoder while introducing sentiment and ASR task knowledge at extremely low cost.

LoRA (Low-Rank Adaptation) is a mainstream parameter-efficient fine-tuning method. It does not modify pre-trained weights; instead, it attaches a pair of low-rank matrices A (dimension reduction) and B (dimension increase) in parallel to each weight matrix, with the forward output being the original weight result plus B·A·x. Since the rank r is typically 4–16, the number of new parameters is minimal, and at inference time B·A can be merged back into the original weights without adding latency. This paper compares or combines it with task vector merging: task vectors inject external priors from sentiment and ASR, while LoRA learns AD-specific adaptation directions. The two have different mechanisms and are therefore complementary.

Experimental Results: Outperforming All Baselines

Main Binary Classification Experiments

Speaker-independent experiments on ADReSS and ADReSSo show that Dual Task Vector Merging outperforms frozen probes, LoRA, and full fine-tuning. The best results are achieved when combined with LoRA, indicating that the two parameter-efficient methods are complementary—task vectors provide external knowledge priors, while LoRA provides task-specific adaptation capability.

Three-Class and Low-Resource Validation

On the PROCESS dataset, when paired with LoRA, the advantage transfers to the HC/MCI/dementia three-class task. More importantly, in reduced experiments using only 20%, 40%, and 80% of the training data, the performance gains persist stably, confirming its reliability in low-resource scenarios.

The differences among ADReSS, ADReSSo, and PROCESS are worth noting. ADReSS and ADReSSo are INTERSPEECH challenge datasets, both binary classification (AD vs. healthy control), and they emphasize speaker-independent splits to avoid optimistic bias caused by the same speaker appearing in both training and test sets. PROCESS, on the other hand, is a larger-scale multi-class corpus annotated with three levels: healthy control (HC), mild cognitive impairment (MCI), and dementia. It is closer to the real needs of clinical screening—because MCI is the prodromal stage of AD, and distinguishing MCI from early dementia is precisely the key difficulty in early identification. Three-class classification is significantly harder than binary classification, so this result is stronger evidence for the method's generalization ability.

Practical Implications

The value of this work lies in providing a general approach: when target-domain data is insufficient, task vectors can be extracted from related tasks as a knowledge injection source rather than fine-tuning from scratch. The layer-wise learnable merging coefficients also bring interpretability—different layers' reliance on sentiment and ASR knowledge may reveal the hierarchical distribution of AD speech features. For AD speech research teams with limited resources, this method opens a channel to leverage external task knowledge while maintaining extremely low training costs.

Future directions worth exploring include: combinations of more source tasks (e.g., depression detection, dysarthria recognition), and the compatibility of task vector merging with larger-scale speech foundation models.

One-line Summary

Dual Task Vector Merging improves low-resource AD speech detection with less than 0.1% trainable parameters.

Paragraph Summary

This article introduces a parameter-efficient method called Dual Task Vector Merging for detecting Alzheimer's disease from speech. To address the dilemma that AD speech datasets are small and fine-tuning large pre-trained models easily overfits while frozen probes waste representational knowledge, the authors leverage the insight that sentiment recognition and ASR tasks correspond to paralinguistic and linguistic cues respectively. They first fine-tune wav2vec 2.0 to obtain two task vectors, then inject them into each Transformer layer of a frozen backbone with layer-wise learnable coefficients, optimizing only the merging coefficients and a linear classification head, with trainable parameters less than 0.1% of full fine-tuning. It outperforms frozen probes, LoRA, and full fine-tuning on ADReSS and ADReSSo binary classification, with the best results when combined with LoRA; PROCESS three-class and 20%–80% reduced-data experiments validate stable gains in low-resource settings.