DenseTwins
A multimodal self-supervised learning framework that explicitly disentangles the dense feature space into cross-modally shared components and modality-specific components. It jointly optimizes dense cross-modal correspondences together with a redundancy reduction objective, and is used to learn dense representations that preserve spatial structure.
Timeline (last 90 days)
DenseTwins is a multimodal self-supervised framework that learns dense representations by explicitly decoupling the dense feature space into shared components and modality-specific components.
DenseTwins extends the idea of redundancy reduction from the channel dimension to the modality dimension, using it to constrain modality-specific components not to be redundant with one another.
DenseTwins jointly optimizes two objectives: an intra-modal objective (acting between augmented views to maintain consistency across different augmentations of the same modality) and a cross-modal objective (aligning shared features and decorrelating modality-specific information).
DenseTwins is evaluated on multiple multimodal benchmarks and sensing settings, and the results achieve state-of-the-art or competitive performance compared with multimodal self-supervised baselines.
DenseTwins's evaluation covers different sensing configurations, showing that the method is not tied to any particular modality combination.
The DenseTwins code is open-sourced at https://github.com/CarloSgaravatti/DenseTwins .
DenseTwins explicitly writes decoupling into the objective function rather than expecting the network to learn the division of labor on its own, which is more controllable than implicit fusion designs.
All Facts (7)
DenseTwins is evaluated on multiple multimodal benchmarks and sensing settings, and the results achieve state-of-the-art or competitive performance compared with multimodal self-supervised baselines.
50%UnverifiedDenseTwins extends the idea of redundancy reduction from the channel dimension to the modality dimension, using it to constrain modality-specific components not to be redundant with one another.
50%UnverifiedDenseTwins jointly optimizes two objectives: an intra-modal objective (acting between augmented views to maintain consistency across different augmentations of the same modality) and a cross-modal objective (aligning shared features and decorrelating modality-specific information).
50%UnverifiedDenseTwins is a multimodal self-supervised framework that learns dense representations by explicitly decoupling the dense feature space into shared components and modality-specific components.
50%UnverifiedDenseTwins's evaluation covers different sensing configurations, showing that the method is not tied to any particular modality combination.
50%UnverifiedDenseTwins explicitly writes decoupling into the objective function rather than expecting the network to learn the division of labor on its own, which is more controllable than implicit fusion designs.
50%UnverifiedThe DenseTwins code is open-sourced at https://github.com/CarloSgaravatti/DenseTwins .
50%