WaveGen
An end-to-end text-to-speech synthesis model that generates waveforms directly from text based on a diffusion model, without intermediate representations or an external decoder. Its core design includes Spectral Forcing, a decoupled diffusion head, and a multi-scale spectral v-loss, enabling single-stage training.
Timeline (last 90 days)
WaveGen is a diffusion model that generates waveforms directly from text, without relying on any external decoder, duration predictor, or self-supervised semantic representation.
The key design of WaveGen is Spectral Forcing, which has a single model learn both the model's internal spectral trajectory and the target waveform trajectory simultaneously, with the spectral trajectory serving as internal scaffolding to help stabilize the diffusion process under high compression ratios.
WaveGen adopts a disentangled diffusion heads architecture and a multiscale spectral v-loss training objective; text-speech alignment is done inside the model, and the duration predictor is removed.
WaveGen scales the model from 0.1B parameters to 0.6B parameters, validating the scalability of end-to-end TTS.
On the Emilia training dataset, WaveGen reports state-of-the-art performance.
On the Seed-en benchmark, WaveGen achieves a word error rate (WER) of 1.47 and a speaker similarity (SIM) of 0.75.
WaveGen's training pipeline is a single architecture, open-source data, and one-stage training, requiring no auxiliary optimization methods such as REPA.
The WaveGen project is open-sourced on GitHub (sh-lee-prml/WaveGen), with both code and checkpoints released.
The WaveGen internal spectral trajectory is essentially still an implicit intermediate representation, except that it is no longer solidified by external modules, and its interpretability and intervenability are currently open questions.
All Facts (9)
The WaveGen internal spectral trajectory is essentially still an implicit intermediate representation, except that it is no longer solidified by external modules, and its interpretability and intervenability are currently open questions.
50%UnverifiedThe key design of WaveGen is Spectral Forcing, which has a single model learn both the model's internal spectral trajectory and the target waveform trajectory simultaneously, with the spectral trajectory serving as internal scaffolding to help stabilize the diffusion process under high compression ratios.
50%UnverifiedWaveGen adopts a disentangled diffusion heads architecture and a multiscale spectral v-loss training objective; text-speech alignment is done inside the model, and the duration predictor is removed.
50%UnverifiedWaveGen scales the model from 0.1B parameters to 0.6B parameters, validating the scalability of end-to-end TTS.
50%UnverifiedOn the Emilia training dataset, WaveGen reports state-of-the-art performance.
50%UnverifiedOn the Seed-en benchmark, WaveGen achieves a word error rate (WER) of 1.47 and a speaker similarity (SIM) of 0.75.
50%UnverifiedWaveGen's training pipeline is a single architecture, open-source data, and one-stage training, requiring no auxiliary optimization methods such as REPA.
50%UnverifiedWaveGen is a diffusion model that generates waveforms directly from text, without relying on any external decoder, duration predictor, or self-supervised semantic representation.
50%UnverifiedThe WaveGen project is open-sourced on GitHub (sh-lee-prml/WaveGen), with both code and checkpoints released.
50%