[KongchangAI]
· 2 min read· 1,291 words

WaveGen: Truly End-to-End Waveform Generation, No Intermediate Representations

WaveGen: Truly End-to-End Waveform Generation, No Intermediate Representations

WaveGen uses a single diffusion model for direct text-to-waveform generation without external decoders, simplifying the TTS pipeline.

WaveGen is an end-to-end TTS method using a single diffusion model to generate waveforms directly from text, eliminating mel-spectrograms, neural audio codecs, or VAE latents. Its Spectral Forcing technique learns spectral and waveform trajectories internally, with disentangled diffusion heads and multi-scale spectral v-loss, removing duration predictors and external semantic modules. Scaling from 0.1B to 0.6B parameters and trained on Emilia, it reaches WER 1.47 and SIM 0.75 on Seed-en, showing feasibility and scalability. Code and checkpoints are open-sourced.

WaveGen paper title

Speech synthesis has long relied on a multi-stage pipeline of "text → intermediate representation → waveform," where the intermediate representation is either mel-spectrograms plus a vocoder, discrete tokens from a neural audio codec, or VAE latents. Each of these modules is trained and tuned separately, errors accumulate stage by stage, and engineering complexity remains high. A new paper proposes WaveGen, which attempts to tear down this pipeline entirely: a single diffusion model generates waveforms directly from text, without relying on any external decoder, duration predictor, or self-supervised semantic representation.

Why "end-to-end" is easier said than done

Truly end-to-end TTS faces a fundamental contradiction: text and waveforms differ enormously in time scale—a single sentence can correspond to tens of thousands of waveform samples, while the text has only a few dozen tokens. A diffusion model must model directly at such high dimensionality and also perform text-speech alignment on its own during generation, which is far more difficult than first generating a low-frame-rate intermediate representation and then upsampling.

In reality, most systems claiming to be "end-to-end" make compromises. Common approaches include: compressing waveforms into discrete tokens with a neural audio codec, two-stage synthesis with mel-spectrograms plus a vocoder, or introducing BERT text embeddings, Wav2Vec 2.0 semantic features, and REPA-style semantic distillation to stabilize training. These methods are indeed effective, but each external module means an additional training stage, additional data dependencies, and additional deployment burden.

WaveGen's core claim is that these compromises are not necessary.

Diffusion models are a class of generative models whose core idea is to generate data through a gradual denoising process: starting from pure noise, the model learns how to remove noise step by step until it recovers the target sample. In speech synthesis, diffusion models can model the distribution of waveforms or spectrograms. Compared with traditional generative models, diffusion models train stably and can generate high-quality samples, but inference requires multiple iterations and is computationally expensive. WaveGen uses a diffusion model to generate directly in waveform space; the challenge is the extremely high dimensionality of waveforms, requiring carefully designed training objectives to stabilize the high-dimensional diffusion process.

Neural audio codecs are models that compress waveforms into discrete tokens, typically consisting of an encoder, quantizer, and decoder, such as SoundStream, EnCodec, etc. They can greatly reduce the waveform sequence length, making it easier for language models or diffusion models to model. However, the codec itself must be trained separately, and the quantization process introduces information loss that may affect synthesis quality. WaveGen deliberately avoids such external codecs to eliminate their additional dependencies on training data and deployment.

REPA (Representation Alignment) is a training auxiliary technique that accelerates diffusion model convergence by aligning the model's internal representations with features from a pretrained self-supervised model (such as Wav2Vec 2.0). Similarly, Wav2Vec 2.0 is a self-supervised speech representation model that can extract rich semantic and acoustic features. Although these methods can improve training stability, they introduce dependencies on external pretrained models, increasing training complexity and data licensing restrictions. WaveGen's goal is to abandon these auxiliary means and achieve simpler one-stage training.

Spectral Forcing: Learning spectral trajectories into the model itself

WaveGen's key design is Spectral Forcing. It lets a single model learn two trajectories simultaneously: one is the model's internal spectral trajectory, and the other is the target waveform trajectory. The spectral trajectory acts as an "internal scaffold" to help the diffusion process remain stable under high compression ratios, but it is generated entirely within the model and does not require an external vocoder or codec.

Complementing this are the disentangled diffusion heads architecture and the multi-scale spectral v-loss training objective. The disentangled heads allow different diffusion stages to each perform their own role, while the multi-scale spectral loss constrains generation quality at multiple frequency resolutions. Text-speech alignment is also completed within the model, and the duration predictor is completely removed.

The direct benefit of this design is a simplified training process: a single architecture, open-source data, one-stage training, and no need for auxiliary optimization methods like REPA.

Scaling and empirical performance

The paper validates the scalability of end-to-end TTS, scaling the model from 0.1B to 0.6B parameters. On the Emilia training dataset, WaveGen reports state-of-the-art performance; on the Seed-en benchmark, word error rate (WER) is 1.47 and speaker similarity (SIM) is 0.75.

These two numbers deserve context. WER 1.47 means the intelligibility of the synthesized speech is already above the practical threshold; SIM 0.75 indicates considerable fidelity in voice cloning. What you may not have noticed is the conditions under which these metrics were achieved: no duration predictor, no external semantic module, no multi-stage training. This weakens the conventional impression that "end-to-end necessarily sacrifices quality."

Of course, numbers on a single benchmark are not enough to cover all scenarios. Whether the same holds across languages, emotional control, and long-text stability still requires further validation by the community based on the open-source code and checkpoints.

Word error rate (WER) is a metric in speech recognition that measures recognition accuracy, computed as the ratio of substituted, inserted, and deleted words to the total number of words; lower is better. In TTS evaluation, an ASR system is typically used to transcribe the synthesized speech, and then WER is computed to reflect intelligibility. Speaker similarity (SIM) uses a speaker verification model to extract embeddings and computes the cosine similarity between the synthesized speech and the target speaker's speech, measuring the fidelity of voice cloning; higher is better. The Seed-en benchmark is an English test set specifically designed for evaluating TTS system performance, containing multiple speakers and texts. WaveGen achieves WER 1.47 and SIM 0.75 on it, indicating that its synthesized speech is both clear and close to the target timbre.

What this means for TTS engineering practice

If WaveGen's approach is reproducible, the deployment form of TTS systems may change. In traditional pipelines, the vocoder, codec, and alignment module each have their own version management and inference overhead; an end-to-end single model consolidates all of these into one inference graph. For scenarios requiring low latency or edge deployment, this simplification has practical value.

Another layer of impact is on the data side. Eliminating dependence on BERT embeddings and Wav2Vec 2.0 features means training is no longer limited by the language coverage and licensing conditions of these pretrained models; in theory, one can train with purer target-language data.

However, one should be clear-eyed: the internal spectral trajectory is still essentially an implicit intermediate representation, only it is no longer fixed by an external module. Whether it can be intuitively debugged and intervened in like mel-spectrograms remains an open question. The reduced interpretability brought by end-to-end design is a cost this approach must pay.

The project is open-sourced on GitHub (sh-lee-prml/WaveGen), with both code and checkpoints released. It is worth following and reproducing for researchers and engineers interested in the evolution of TTS architectures.

One-line Summary

WaveGen uses a single diffusion model to achieve direct text-to-waveform generation without external decoders, simplifying the TTS pipeline.

Paragraph Summary

WaveGen proposes an end-to-end text-to-speech synthesis method that uses a single diffusion model to generate waveforms directly from text, abandoning intermediate representations such as mel-spectrograms, neural audio codecs, or VAE latents in traditional multi-stage pipelines. Its core is the Spectral Forcing technique, which lets the model learn both spectral and waveform trajectories internally, combined with disentangled diffusion heads and multi-scale spectral v-loss to stabilize training, while removing the duration predictor and external semantic modules. The model scales from 0.1B to 0.6B parameters and is trained on the Emilia dataset, achieving WER 1.47 and SIM 0.75 on the Seed-en benchmark, demonstrating the feasibility and scalability of end-to-end TTS. The work open-sources code and checkpoints, providing new ideas for the evolution of TTS architectures.

#WaveGen#end-to-end TTS#diffusion model#Spectral Forcing#text-to-speech#waveform generation
Share:

Related articles