VQ-VAE
A Vector Quantized Variational Autoencoder proposed by DeepMind in 2017 that maps continuous latent representations to a finite discrete codebook. It is widely used in generative modeling and sequence modeling.
Core Facts
Timeline (last 90 days)
NCM extends the VQ-VAE mechanism to multiple codebook layers, with each layer responsible for a different level of abstraction granularity: coarse-layer codebook entries are few and semantically broad, while fine-layer entries are numerous and express specific details.
Codebook collapse is a common failure mode of VQ-VAE, meaning only a few codebook entries are used while the rest become dead codes; common mitigations include exponential moving average updates, codebook resets and diversity regularization.
NCM uses VQ-VAE to compress text into multi-level discrete codes, with coarse layers corresponding to paragraph- or semantic-topic-level abstraction and fine layers corresponding to more specific expressive details.
The core of VQ-VAE is a learnable codebook containing a number of discrete vectors; after the encoder outputs continuous representations, each position is replaced with the nearest vector in the codebook (nearest-neighbor quantization).
During VQ-VAE training, a straight-through estimator passes gradients back from the decoder to the encoder, and a commitment loss constrains the encoder output to stay close to the selected codebook vector.
Common mitigation measures for codebook collapse include exponential moving average updates to the codebook, restarting inactive codewords, and introducing entropy regularization or commitment loss for codebook usage.
VQ-VAE is a class of generative models that discretizes continuous representations; its core mechanism is to maintain a learnable codebook, replace continuous vectors with discrete vectors via nearest-neighbor lookup, and then reconstruct the input through a decoder.
VQ-VAE uses a straight-through estimator to pass gradients back from the decoder to the encoder, and the training objective typically includes three terms: reconstruction loss, codebook alignment loss, and commitment loss.
VQ-VAE training is sensitive to codebook size and initialization and is prone to codebook collapse, where only a few codewords are activated and most codewords sit idle.
NCM's tokenizer (VQ-VAE) and generator (Transformer) are trained separately and independently, which may limit overall performance.
2 more timeline events
All Facts (14)
Inverse rendering is fundamentally an ill-posed optimization problem, as the same photograph can be produced by infinitely many combinations of lighting and materials, requiring strong prior constraints.
75%UnverifiedNCM extends the VQ-VAE mechanism to multiple codebook layers, with each layer responsible for a different level of abstraction granularity: coarse-layer codebook entries are few and semantically broad, while fine-layer entries are numerous and express specific details.
50%UnverifiedNCM uses VQ-VAE to compress text into multi-level discrete codes, with coarse layers corresponding to paragraph- or semantic-topic-level abstraction and fine layers corresponding to more specific expressive details.
50%UnverifiedThe core of VQ-VAE is a learnable codebook containing a number of discrete vectors; after the encoder outputs continuous representations, each position is replaced with the nearest vector in the codebook (nearest-neighbor quantization).
50%UnverifiedDuring VQ-VAE training, a straight-through estimator passes gradients back from the decoder to the encoder, and a commitment loss constrains the encoder output to stay close to the selected codebook vector.
50%UnverifiedCodebook collapse is a common failure mode of VQ-VAE, meaning only a few codebook entries are used while the rest become dead codes; common mitigations include exponential moving average updates, codebook resets and diversity regularization.
50%UnverifiedVQ-VAE training is sensitive to codebook size and initialization and is prone to codebook collapse, where only a few codewords are activated and most codewords sit idle.
50%UnverifiedNCM's tokenizer (VQ-VAE) and generator (Transformer) are trained separately and independently, which may limit overall performance.
50%UnverifiedCommon mitigation measures for codebook collapse include exponential moving average updates to the codebook, restarting inactive codewords, and introducing entropy regularization or commitment loss for codebook usage.
50%UnverifiedVQ-VAE is a class of generative models that discretizes continuous representations; its core mechanism is to maintain a learnable codebook, replace continuous vectors with discrete vectors via nearest-neighbor lookup, and then reconstruct the input through a decoder.
50%UnverifiedVQ-VAE uses a straight-through estimator to pass gradients back from the decoder to the encoder, and the training objective typically includes three terms: reconstruction loss, codebook alignment loss, and commitment loss.
50%UnverifiedVQ-GAN introduced adversarial training and perceptual loss on top of VQ-VAE, becoming the technical foundation for the visual tokenizer in models such as DALL-E and Parti.
50%UnverifiedDiscrete tokenization schemes typically rely on quantization encoders such as VQ-VAE or VQ-GAN, which sacrifice visual fidelity during the image generation process.
50%UnverifiedVQ-VAE（Vector Quantized Variational Autoencoder）由DeepMind于2017年提出，将连续潜在表征量化映射至有限离散码本
50%