RASO: Role-Aware Server Optimization for Multimodal Federated Learning with Missing Modalities

Multimodal Federated Learning (MFL) faces the common challenge of missing client modalities while preserving data privacy. When some clients lack certain modality data, they can only activate the available modality encoders and share the prediction block (fusion and classifier). This creates two key problems on the server side: which clients should contribute updates for each component, and how should these updates be aggregated across rounds? A recent paper proposes Role-Aware Server Optimization (RASO), which significantly improves model performance in missing-modality scenarios by coordinating contributor selection with role-specific server updates.

Server-Side Challenges Under Missing Modalities

In multimodal federated learning, clients typically possess data from multiple modalities such as images, text, and audio. However, in real-world scenarios, clients may contain only a subset of modalities. The conventional approach has clients separately adapt active modality encoders and a shared prediction block, but since different clients have different available modality subsets, encoder updates vary.

The server faces a dilemma during aggregation: availability-based aggregation may ignore updates for missing modality encoders, since that encoder received no corresponding input in the current round. But the paper observes that even when a modality is missing, the encoder can still adapt through missing-input handling mechanisms—meaning its updates should not simply be discarded. On the other hand, prediction block updates come from all clients and should be retained and jointly learned.

Another easily overlooked factor is server momentum, i.e., the weighted history of aggregated updates. Momentum helps stabilize training, but its effectiveness depends on the encoder's update strategy. The paper finds that prediction block momentum may improve or degrade performance, depending on the choice of encoder update strategy.

Server momentum in federated learning is a weighted history mechanism for aggregated updates, typically implemented as an exponential moving average (EMA): after each round of aggregation, the server does not directly adopt the current round's average gradient but blends it with the previous round's model parameters at a certain weight. Momentum smooths the training trajectory and reduces oscillation caused by client data heterogeneity, and has been shown to aid convergence in classical algorithms such as FedAvg. However, the effectiveness of momentum is highly dependent on the statistical properties of parameter updates: for parameters that receive stable gradient signals every round, momentum acts as a denoiser; for parameters with sparse or intermittent updates, historical momentum may lag behind the current distribution and instead drag down performance. In multimodal federated learning, encoders and prediction blocks have different update frequencies and sources, so the applicability of momentum must be evaluated separately rather than uniformly enabled or disabled globally.

RASO's Core Design: Role-Aware Coordination Mechanism

To address the above issues, RASO proposes a role-aware server optimization framework, whose core idea is to refine server-side aggregation into two roles: the encoder role and the prediction block role, with separate update strategies for each.

Specifically, RASO achieves coordination through the following mechanisms:

• Contributor selection: Contributors are selected based on prediction dependency and participation in local optimization. Prediction dependency measures the degree of a client's influence on the shared prediction block, while participation reflects whether the client actually optimized that component during local training. This selection approach ensures that the most relevant client updates are chosen for each component.
• Encoder momentum preservation: For the encoder role, RASO preserves server momentum to stabilize the encoder's adaptation process, even when certain modalities are missing from some clients.
• Prediction block synchronization: For the shared prediction block, RASO synchronizes it to the current aggregation result without using momentum, thereby avoiding the potential negative effects of momentum.

This role-separation design enables the server to more finely control the update rhythm of different components, rather than adopting a one-size-fits-all aggregation strategy.

Prediction dependency is a core metric in RASO's contributor selection, used to characterize a client's influence on the shared prediction block. Intuitively, if a client's local update significantly moves the prediction block parameters in a certain direction, or if its gradient contribution to the prediction loss is large, then the client is highly relevant to the prediction task. Common implementations compute the cosine similarity or norm ratio between the client-uploaded prediction block gradients and the server-aggregated gradients, or estimate prediction consistency on a validation set. Compared with conventional aggregation weighted solely by data volume, prediction dependency can identify clients that, despite small data volume, make critical contributions to the classification boundary, thereby avoiding dilution of the prediction block's learning signal by weakly relevant clients when missing modalities cause uneven client capabilities.

Missing-input handling mechanisms refer to designs that allow multimodal models to still forward-propagate and produce gradients when a certain modality input is unavailable. Common approaches include: replacing the modality's embedding with a learnable missing token, zero-padding followed by masked attention to mask invalid positions, or introducing modality presence embeddings so the model is aware of which modalities are available. It is precisely this mechanism that enables encoders to still participate in computation when clients lack the corresponding modality, with their parameters receiving gradients from the missing-token pathway. Therefore, if the server discards encoder updates on the grounds that "no input of that modality exists this round," it wastes this adaptation signal and may even cause discontinuous encoder updates across rounds. RASO's emphasis on retaining encoder updates and pairing them with momentum is built on the premise that missing-input handling keeps encoders always trainable.

Experimental Validation and Results

The paper conducts experiments on different datasets and fusion architectures, validating RASO's effectiveness in missing-modality scenarios. Experimental results show that by coordinating contributor selection with role-specific server updates, RASO can significantly improve model performance under missing-modality conditions.

As a detail, RASO does not simply discard or retain momentum but dynamically adjusts based on component roles. This flexibility enables it to adapt to different fusion architectures and data distributions, offering new ideas for the practical deployment of multimodal federated learning.

Implications for Multimodal Federated Learning

RASO reveals two important principles for server-side optimization in missing-modality scenarios: role separation and dynamic coordination. Traditional federated learning aggregation often treats all parameters equally, but in multimodal scenarios, different components (encoders vs. prediction blocks) have vastly different needs regarding momentum and contributors.

In the future, this role-aware approach may extend to more complex federated learning architectures, such as personalized federated learning or cross-modal retrieval tasks. For developers committed to leveraging multimodal data under privacy protection, RASO provides a reusable server-side optimization paradigm.

As multimodal applications become widespread, the missing-modality problem will become increasingly prominent. Through careful server-side design, RASO offers a practical solution to this challenge and drives federated learning toward directions that better reflect real-world scenarios.

One-line Summary

RASO improves missing-modality federated learning performance through role-aware contributor selection and differentiated momentum strategies.

Paragraph Summary

The article introduces the client missing-modality problem in multimodal federated learning: some clients can only activate available modality encoders and share the prediction block, leaving the server facing two major challenges—"which clients should contribute updates for each component" and "how should updates be aggregated across rounds." The paper's proposed Role-Aware Server Optimization (RASO) splits server aggregation into the encoder role and the prediction block role, selecting contributors based on prediction dependency and local optimization participation; it preserves server momentum for encoders to stabilize adaptation, while synchronizing the prediction block to the current aggregation result to avoid momentum's negative effects. Experiments show that this coordination mechanism significantly improves model performance in missing-modality scenarios across different datasets and fusion architectures, providing a reusable server-side optimization paradigm for privacy-preserving multimodal learning.