GAMW: Generation-Aware Mel-Domain Watermarking Offers New Approach to TTS Audio Provenance

When Synthetic Speech Floods the World, Provenance Becomes a Necessity

TTS systems can already generate speech that is nearly indistinguishable from real human voices, but this brings a series of real-world problems: Was this audio spoken by a real person or synthesized? If synthesized, which model produced it, and in which generation?

Audio watermarking attempts to answer these questions—by embedding information into speech that is imperceptible to human ears but recoverable by machines, functioning as an "identity card." The traditional approach is post-processing: first synthesize the waveform, then add a watermark to the waveform. The problem is that the watermark layer and the post-synthesized audio layer are separate. Once the audio undergoes composite perturbations such as compression, noise addition, or resampling, the post-added watermark is easily destroyed.

A recent study called GAMW (Generation-Aware Mel-Domain Watermarking) proposes a different route: instead of waiting for speech synthesis to complete before adding a watermark, it embeds the watermark directly during the synthesis process. The code is open-sourced on GitHub: https://github.com/ZConJugaTe/GAMW .

How GAMW Works: Embedding in the Mel-Domain Rather Than Tagging the Waveform

The core design philosophy of GAMW can be understood across three dimensions.

Moving the Embedding Point Upstream

Traditional post-processing methods operate on the waveform (time domain), while GAMW chooses to inject the watermark into the intermediate representation in the Mel-domain. Specifically, before waveform synthesis, GAMW embeds a "content-adaptive multi-bit watermark residual" into the representations produced during TTS generation.

This "content-adaptive" aspect is key—the watermark is not a fixed-pattern signal, but a residual that dynamically adjusts based on the current speech content. The goal is to allow the watermark to blend better with the speech content, reducing the impact on speech quality while improving the watermark's survival rate through subsequent processing.

Designing the Decoding Objective

Embedding is only half the story; decoding is equally important. GAMW's decoding objective function combines two mechanisms: focal modulation and dynamic per-bit weighting.

The former borrows the idea of focal loss from object detection, making the model focus more on hard-to-decode samples during training; the latter assigns different weights to different bit positions, preventing certain bits from dragging down overall decoding accuracy. This combination allows a 16-bit watermark payload to maintain a high recovery rate under perturbation conditions.

Focal modulation and focal loss. Focal loss was originally proposed by Kaiming He et al. in object detection (RetinaNet) to address the extreme imbalance between positive and negative samples—by reducing the weight of easy-to-classify samples, the model focuses on hard-to-classify ones. GAMW transfers this idea to watermark decoding: each bit of the watermark is equivalent to a binary classification task, and during training the model pays more attention to bits that are difficult to decode correctly, thereby improving overall decoding success rate.

Dynamic per-bit weighting. During training of a 16-bit watermark, different bit positions have different decoding difficulties. Some bits are more error-prone due to their embedding position or the distribution characteristics of the content-adaptive residual. Dynamic per-bit weighting maintains a learnable weight for each bit position in the loss function, automatically reducing the influence of bits that are already accurately decoded and concentrating gradients on bits that have not yet been learned well.

Cross-Backbone Validation

The research team trained and evaluated GAMW on two different TTS backbones, demonstrating that this method is not tied to a specific TTS architecture and has a degree of generality.

What is the Mel-domain? The Mel scale is a frequency scale that simulates the perceptual characteristics of human hearing—human ears have stronger resolution for low-frequency sounds and coarser resolution for high frequencies. TTS systems typically first convert text into acoustic features (such as Mel spectrograms), and then a vocoder reconstructs these intermediate representations into waveforms. GAMW chooses to embed the watermark at the Mel spectrogram level rather than processing after waveform generation. The reason: the Mel spectrogram is the direct basis for vocoder synthesis, and information injected at this stage is naturally "woven" into the final waveform through the vocoder's synthesis process. Compared to watermark post-processing which only superimposes signals on the waveform surface, this coupling is much deeper.

What is composite perturbation? In real-world pipelines, audio often undergoes multiple processing steps: encoding compression (such as MP3, AAC), packet loss and resampling caused by network transmission, environmental noise addition, etc. A single perturbation involves only one of these operations, while composite perturbation is the serial connection of these operations. Waveform-domain watermarks are weakened with each processing step when facing composite perturbations, and after accumulation become nearly unrecoverable.

Performance Data: 23.5 Percentage Points Improvement Under Composite Perturbation

The core data presented in the paper is quite compelling. With a 16-bit payload:

• Under single perturbation scenarios, GAMW's decoding accuracy exceeds the strongest baseline method by 9.2 percentage points
• Under composite perturbation scenarios, the advantage expands to 23.5 percentage points

The widening gap under composite perturbation is not surprising. Post-processing watermarks can barely cope with single perturbations, but when multiple perturbations are stacked—such as compression followed by resampling and then noise addition—the waveform-domain watermark signal is weakened layer by layer. Because GAMW embeds the watermark in the intermediate representation of the generation process, it has deeper coupling with the speech signal, and its robustness advantage is amplified in composite scenarios.

At the same time, the paper emphasizes that GAMW maintains "competitive speech quality." This means the watermark embedding does not come at the cost of significantly sacrificing audio quality—a core trade-off that requires repeated balancing in the watermarking field.

At higher payloads (beyond 16 bits), GAMW still maintains reliable clean speech decoding capability, indicating that its multi-bit capacity has some room for upward scaling.

How to interpret "23.5 percentage points"? A percentage point is an absolute measure of the difference in accuracy. Suppose the strongest baseline achieves 60% decoding accuracy under composite perturbation; GAMW would achieve 83.5%. If the baseline is 40%, GAMW would be 63.5%. Unlike "a 23.5% improvement," percentage points directly reflect the arithmetic difference between two accuracy rates, unaffected by the base value, and are therefore preferred in academic comparisons.

Significance and Limitations of This Work

TTS watermarking is not just an academic problem. As synthetic speech appears in large quantities across content platforms, customer service systems, audiobooks, and other scenarios, the verifiability of audio provenance is becoming an infrastructure-level requirement. The "generation-aware" approach represented by GAMW—integrating the watermark into the generation process rather than appending it afterward—may be a paradigm shift worth watching in this direction.

However, it should be noted that the paper has only validated on two TTS backbones so far. Whether the method applies equally under different codecs, transmission pipelines, and adversarial attack conditions encountered in real deployment still requires more testing. Additionally, Mel-domain embedding requires the watermarking system to have access to the TTS internal pipeline, which raises the integration barrier for application scenarios that only want to add watermarks at the audio output end.

The release of open-source code (https://github.com/ZConJugaTe/GAMW ) provides a starting point for reproducing and improving such methods. For technologists concerned with AI audio security, deepfake detection, and content provenance, GAMW offers a technical option different from "waveform post-processing."

The fundamental difference between "generation-aware" watermarking and post-processing watermarking. Post-processing watermarking treats the watermark as an additional signal layer applied on top of completed audio—the audio itself is "unaware" of the watermark. Generation-aware watermarking, by contrast, makes the watermark a participant in the generation process, with the watermark and speech content entangled at the representation level. This means that even if an attacker attempts to remove the watermark through recompression or filtering, they would simultaneously damage the speech content itself, thereby raising the cost of attack. This approach has direct reference value for deepfake detection, AI-generated content labeling, and similar scenarios.

One-line Summary

GAMW embeds watermarks into the Mel-domain intermediate representations of TTS generation, achieving 23.5 percentage points higher decoding accuracy than waveform post-processing baselines under composite perturbations.

Paragraph Summary

GAMW proposes a method for embedding watermarks in the Mel-domain during the TTS generation process, replacing the traditional approach of post-processing on waveforms. Its core is injecting a content-adaptive multi-bit watermark residual into TTS intermediate representations, with the decoding end combining focal modulation and dynamic per-bit weighting. With a 16-bit payload, decoding accuracy exceeds the strongest baseline by 9.2 percentage points under single perturbation and 23.5 percentage points under composite perturbation, while maintaining competitive speech quality. The method has been validated for generality on two TTS backbones, and the code is open-sourced. Limitations include that Mel-domain embedding requires access to the TTS internal pipeline, raising the integration barrier compared to waveform post-processing, and it has not yet been tested under more codecs and adversarial attack conditions.