MRL
A representation learning framework that jointly optimizes multiple nested prefix dimensions of the same representation vector, so that any prefix sub-vector can be used independently for downstream tasks, supporting flexible representation sizes and compressed deployment.
Timeline (last 90 days)
Experiments on MTEB show that SpecMRL outperforms MRL at all evaluated reduced dimensions.
SpecMRL's gains over MRL are larger under more aggressive compression.
SpecMRL surpasses full-dimensional MRL using only half the embedding dimensions.
MRL only requires that each prefix be usable on its own, and no mechanism guarantees that the first few dimensions are exactly the most important directions.
Prefix readout is the core mechanism of MRL: during training, classification or retrieval heads are attached to the first d dimensions of the same representation vector and jointly optimized, so that any prefix can be used independently.
By placing the readout operation in a spectral space ordered by importance, SpecMRL makes prefix truncation naturally correspond to the most informative directions, alleviating the lack of explicit constraints in MRL.
All Facts (6)
Experiments on MTEB show that SpecMRL outperforms MRL at all evaluated reduced dimensions.
50%UnverifiedSpecMRL's gains over MRL are larger under more aggressive compression.
50%UnverifiedSpecMRL surpasses full-dimensional MRL using only half the embedding dimensions.
50%UnverifiedMRL only requires that each prefix be usable on its own, and no mechanism guarantees that the first few dimensions are exactly the most important directions.
50%UnverifiedPrefix readout is the core mechanism of MRL: during training, classification or retrieval heads are attached to the first d dimensions of the same representation vector and jointly optimized, so that any prefix can be used independently.
50%UnverifiedBy placing the readout operation in a spectral space ordered by importance, SpecMRL makes prefix truncation naturally correspond to the most informative directions, alleviating the lack of explicit constraints in MRL.
50%