SoundStream
A neural audio codec based on residual vector quantization and multiple codebooks that supports low-bitrate streaming audio compression.
Timeline (last 90 days)
Traditional TTS pipelines rely on an acoustic tokenizer (such as discrete codecs, with common implementations including SoundStream and EnCodec) to discretize audio through residual vector quantization.
Discrete token schemes (such as EnCodec, SoundStream, and Mimi) require predicting a set of residual vector quantization codebooks at each timestep, and in Mistral's scenarios having more than 30 codebooks significantly increases generation latency.
Some research attempts to encode audio as sequences of discrete tokens (such as EnCodec and SoundStream), enabling language models to read and write audio.
All Facts (3)
Traditional TTS pipelines rely on an acoustic tokenizer (such as discrete codecs, with common implementations including SoundStream and EnCodec) to discretize audio through residual vector quantization.
50%UnverifiedDiscrete token schemes (such as EnCodec, SoundStream, and Mimi) require predicting a set of residual vector quantization codebooks at each timestep, and in Mistral's scenarios having more than 30 codebooks significantly increases generation latency.
50%UnverifiedSome research attempts to encode audio as sequences of discrete tokens (such as EnCodec and SoundStream), enabling language models to read and write audio.
50%