BLIP-2
BLIP-2 is a vision-language pretraining framework proposed by Salesforce that introduces the Q-Former bridging module to align the vision and language modalities.
Timeline (last 90 days)
Q-Former is a bridging module proposed by Salesforce in BLIP-2 that uses learnable query vectors to extract information from the output of a visual encoder through cross-attention, compressing it into a fixed number of tokens.
Alignment between visual embedding vectors and text embedding vectors is typically accomplished through a projection layer, with common implementations including a linear layer, MLP, Q-Former (as used in BLIP-2), or Perceiver Resampler (as used in Flamingo).
Q-Former extracts a fixed number of visual features from a frozen visual encoder through a set of learnable query vectors, achieving information bottleneck-style compression.
Common implementations of the cross-modal alignment module include a linear projection layer (used by LLaVA), Q-Former (used by BLIP-2), and cross-attention mechanisms.
Flamingo was proposed by DeepMind in 2022, and BLIP-2 was proposed by Salesforce in 2023.
BLIP-2 uses a Q-Former structure to extract a fixed number of visual features through learnable query vectors.
All Facts (6)
BLIP-2 uses a Q-Former structure to extract a fixed number of visual features through learnable query vectors.
60%UnverifiedQ-Former is a bridging module proposed by Salesforce in BLIP-2 that uses learnable query vectors to extract information from the output of a visual encoder through cross-attention, compressing it into a fixed number of tokens.
50%UnverifiedAlignment between visual embedding vectors and text embedding vectors is typically accomplished through a projection layer, with common implementations including a linear layer, MLP, Q-Former (as used in BLIP-2), or Perceiver Resampler (as used in Flamingo).
50%UnverifiedQ-Former extracts a fixed number of visual features from a frozen visual encoder through a set of learnable query vectors, achieving information bottleneck-style compression.
50%UnverifiedCommon implementations of the cross-modal alignment module include a linear projection layer (used by LLaVA), Q-Former (used by BLIP-2), and cross-attention mechanisms.
50%UnverifiedFlamingo was proposed by DeepMind in 2022, and BLIP-2 was proposed by Salesforce in 2023.
50%