ProTok
A unified protein tokenizer that uses product quantization to assign independent codebook groups for the three modalities of sequence, structure, and function, and employs modality dropout training to enable multimodal inference from a single checkpoint.
Timeline (last 90 days)
ProTok is a unified protein tokenizer that maps the three modalities of sequence, structure, and function into a unified discrete vocabulary through modular multi-codebook quantization.
ProTok uses product quantization to assign an independent codebook group to each modality; during training, after randomly dropping the input of a certain modality, the shared codebooks must still produce meaningful discrete tokens, so that a single checkpoint can support flexible inference over any subset of modalities without retraining.
Modality dropout draws on the missing modality training paradigm in multimodal learning, and ProTok's innovation lies in combining this strategy with a quantized tokenizer: after randomly dropping the input of one modality during training, the shared codebook must still produce meaningful discrete tokens, thereby ensuring inference consistency under any subset of modality inputs.
On the seven benchmarks, 14 tasks and 28 test splits of StructTokenBench, ProTok achieved an average rank of 1.4 among pure structure tokenizers, ranking first on 21 splits and entering the top two on 27 of the 28 splits.
ProTok's tokens achieve 0.939 lDDT on the CATH 4.2 dataset.
ProTok's structure tokens comprehensively lead baseline structure tokenizers on zero-shot function retrieval, covering the CATH, EC and GO label hierarchies.
A compact language model trained from scratch jointly generates backbone and sequence on ProTok tokens, with 52% of the sampled backbones being designable, whereas the parameter-matched GeoBPE baseline achieves only 21%.
The scale currently validated for ProTok is limited, and the generation task also uses a relatively small language model; whether the unified vocabulary can scale to larger parameter sizes and more complex multimodal protein tasks remains an open question.
All Facts (8)
ProTok is a unified protein tokenizer that maps the three modalities of sequence, structure, and function into a unified discrete vocabulary through modular multi-codebook quantization.
50%UnverifiedProTok uses product quantization to assign an independent codebook group to each modality; during training, after randomly dropping the input of a certain modality, the shared codebooks must still produce meaningful discrete tokens, so that a single checkpoint can support flexible inference over any subset of modalities without retraining.
50%UnverifiedModality dropout draws on the missing modality training paradigm in multimodal learning, and ProTok's innovation lies in combining this strategy with a quantized tokenizer: after randomly dropping the input of one modality during training, the shared codebook must still produce meaningful discrete tokens, thereby ensuring inference consistency under any subset of modality inputs.
50%UnverifiedOn the seven benchmarks, 14 tasks and 28 test splits of StructTokenBench, ProTok achieved an average rank of 1.4 among pure structure tokenizers, ranking first on 21 splits and entering the top two on 27 of the 28 splits.
50%UnverifiedProTok's tokens achieve 0.939 lDDT on the CATH 4.2 dataset.
50%UnverifiedProTok's structure tokens comprehensively lead baseline structure tokenizers on zero-shot function retrieval, covering the CATH, EC and GO label hierarchies.
50%UnverifiedA compact language model trained from scratch jointly generates backbone and sequence on ProTok tokens, with 52% of the sampled backbones being designable, whereas the parameter-matched GeoBPE baseline achieves only 21%.
50%UnverifiedThe scale currently validated for ProTok is limited, and the generation task also uses a relatively small language model; whether the unified vocabulary can scale to larger parameter sizes and more complex multimodal protein tasks remains an open question.
50%