EnCodec
A neural network audio codec developed by Meta that compresses audio into discrete token sequences and reconstructs it. It is widely used as the acoustic encoding layer in AI music generation models.
Timeline (last 90 days)
Audio conditioning techniques typically rely on pretrained audio encoders (such as CLAP or EnCodec) to compress reference audio into embedding vectors, which are then injected into the generative network via cross-attention or AdaLN mechanisms.
Discrete token schemes (such as EnCodec, SoundStream, and Mimi) require predicting a set of residual vector quantization codebooks at each timestep, and in Mistral's scenarios having more than 30 codebooks significantly increases generation latency.
Some research attempts to encode audio as sequences of discrete tokens (such as EnCodec and SoundStream), enabling language models to read and write audio.
All Facts (3)
Audio conditioning techniques typically rely on pretrained audio encoders (such as CLAP or EnCodec) to compress reference audio into embedding vectors, which are then injected into the generative network via cross-attention or AdaLN mechanisms.
50%UnverifiedDiscrete token schemes (such as EnCodec, SoundStream, and Mimi) require predicting a set of residual vector quantization codebooks at each timestep, and in Mistral's scenarios having more than 30 codebooks significantly increases generation latency.
50%UnverifiedSome research attempts to encode audio as sequences of discrete tokens (such as EnCodec and SoundStream), enabling language models to read and write audio.
50%