Modality Dropout
A multimodal training strategy that randomly masks certain input modalities during training, forcing the model to learn cross-modal redundant representations in order to support consistent inference under any subset of input modalities.
Timeline (last 90 days)
ProTok uses product quantization to assign an independent codebook group to each modality; during training, after randomly dropping the input of a certain modality, the shared codebooks must still produce meaningful discrete tokens, so that a single checkpoint can support flexible inference over any subset of modalities without retraining.
Modality dropout draws on the missing modality training paradigm in multimodal learning, and ProTok's innovation lies in combining this strategy with a quantized tokenizer: after randomly dropping the input of one modality during training, the shared codebook must still produce meaningful discrete tokens, thereby ensuring inference consistency under any subset of modality inputs.
All Facts (2)
ProTok uses product quantization to assign an independent codebook group to each modality; during training, after randomly dropping the input of a certain modality, the shared codebooks must still produce meaningful discrete tokens, so that a single checkpoint can support flexible inference over any subset of modalities without retraining.
50%UnverifiedModality dropout draws on the missing modality training paradigm in multimodal learning, and ProTok's innovation lies in combining this strategy with a quantized tokenizer: after randomly dropping the input of one modality during training, the shared codebook must still produce meaningful discrete tokens, thereby ensuring inference consistency under any subset of modality inputs.
50%