Shanghai Jiao Tong University Open-Sources Deep ASMR: Training Low-Whisper Speech Synthesis with Whisper Data

SJTU open-sources Deep ASMR, decoupling whisper style and timbre for whisper synthesis.
Shanghai Jiao Tong University open-sources Deep ASMR, a speech synthesis method that decouples whisper style from speaker timbre. A language model references others' whispers for reading and pauses, while an acoustic decoder references the speaker's normal recordings for timbre. Trained on 670 hours of Chinese and English data, fine-tuned on an existing model, with data, code, and training scripts released.
Close your eyes and rest well. Behind this whisper-synthesized voice is the Deep ASMR project open-sourced by a team at Shanghai Jiao Tong University, along with a reusable training method.
Technical Core: Separating the Whisper "Style" from the Speaker's "Timbre"
Traditional speech synthesis faces a real problem: if a person only has recordings of their normal speaking voice and no whisper recordings, can they still deliver new lines in a low whisper? Deep ASMR's solution splits the task into two steps, referencing two separate streams of reference audio from different sources.
Specifically, the language model references someone else's whisper, and is responsible for arranging how the new lines are read—where to pause, where to use breathiness, and what the rhythm should be. The acoustic decoder references the person's own normal recording, and is responsible for restoring the timbre. This way, the whisper demonstration and the speaker's identity can come from different voices.
According to an explanation by a Bilibili creator in a video, the team compiled 670 hours of Chinese and English recordings, using different people's whispers to learn this speaking style. The model was not trained from scratch, but fine-tuned on top of an existing speech synthesis model, and the data, code, and training scripts were made public.

The language model and acoustic decoder are a common division of labor in two-stage speech synthesis architectures. The language model is responsible for mapping text to a sequence of discrete speech units (similar to phonemes or subword units), deciding "what to say and how to break sentences"; the acoustic decoder converts these unit sequences into acoustic features (such as mel spectrograms), deciding "who it sounds like." Deep ASMR's innovation lies in having each of the two receive guidance from different reference audio, thereby achieving decoupling of style and timbre.
Data Preprocessing: How a Recording Becomes Training Material
Training data requires alignment across three dimensions: the recording must simultaneously match the text content and the speaker identity. Two encoders work together in the system: one extracts the timbre vector, representing the speaker's voice characteristics; the other compresses the sound into a string of IDs, representing discrete speech units.
Both results are computed only once, then packed into shards together with the original recording, and read directly during training. This packing method reduces redundant computation during training and makes data loading more efficient.
The timbre vector (speaker embedding) is a fixed-dimensional vector extracted from audio, used to characterize the speaker's identity features such as voice quality and pitch. Common methods include deep-network-based d-vector, x-vector, and others. Discrete speech units are representations that compress continuous audio into a finite vocabulary of IDs, typically obtained by clustering with self-supervised learning models (such as HuBERT, Wav2Vec 2.0); each ID corresponds to the abstract speech content of a short audio segment. Combined, they preserve "who is speaking" while also extracting "what is said."
Two-Path Training Strategy and Synthesis Pipeline
SJTU's training script starts from an existing model and learns along two paths. The first path converts whisper recordings into speech IDs, letting the language model look at the text and practice predicting these IDs—essentially learning "how to say it." The second path feeds the IDs and timbre from the recording into the acoustic model, using the real spectrogram of the same recording as the target, to learn to restore the details of the sound—essentially learning "who it sounds like."
At synthesis time, the two paths are connected, and the vocoder outputs the final sound. The cleverest part of this design is the flexibility at the inference stage: suppose you only have recordings of a friend's normal speaking voice and no whisper recording of theirs on hand. The program will find someone with a similar voice from a whisper sample library and have the language model reference how that person speaks softly. But when restoring the timbre, it still uses your friend's recording.

Plugin Practice: What Tuning Only the Acoustic Decoder Can Bring
The plugin developed by the Bilibili creator fixes the two reference paths mentioned above: one provides a Chinese whisper example, and the other provides Xiao Yi's reference voice. One key choice was to fine-tune only the acoustic decoder, freezing the language model and vocoder.
The reason is that the Chinese whisper demonstration can already guide the language model to produce the reading and pauses of the lines. Keeping this reading and changing only the step of voice restoration allows observation of how the texture of breathiness changes. During training, the language model and vocoder are fixed, and only the parameters at the end of the acoustic decoder are updated; at inference time, this set of parameters is loaded onto the original model.
For data splitting, the training set and validation set are divided by program source, ensuring that recordings from the same program do not appear on both sides at once, avoiding data leakage.

Fine-tuning refers to continuing to train some or all parameters on top of a pretrained model using a small amount of target data, to adapt to a specific task. Freezing means fixing the parameters of certain layers without updating them, training only the rest. Fine-tuning only the end parameters of the acoustic decoder can greatly reduce the number of trainable parameters, lowering the risk of overfitting, while preserving the general reading ability already learned by the language model, so that the experiment focuses on changes in breathiness texture.
Data leakage in speech synthesis refers to the training set and validation set containing recordings of the same speaker or the same source, causing the model to appear artificially strong on the validation set and failing to reflect true generalization ability. Splitting the training set and validation set by program source, ensuring recordings from the same program appear on only one side, is a common practice to avoid leakage, and is especially critical when data sources are limited.
User Experience and Limitations
The download package includes fine-tuned weights, training instructions, and inference code. Enter a Chinese sentence in the plugin and you can export a separate vocal track. The current version accepts at most one hundred characters at a time, producing a mono vocal. Long stories need to be generated in segments, and the pauses then joined together.
When speaking softly, some word onsets may be unclear, so you need to listen through after recording. Background music, rain sounds, and left-right ear movement are all left to post-processing. These limitations show that whisper synthesis is suitable as a tool for generating voiceover material, rather than a one-click finished solution.

Significance of Open Sourcing
Deep ASMR making its data, code, and training scripts public means whisper synthesis is no longer a closed-source black box. The 670 hours of Chinese and English recordings covering different people's whispers provide a reproducible baseline for future research. The idea of decoupling whisper style from speaker timbre can also be transferred to other stylized speech synthesis tasks, such as emotional speech and dialect conversion. For developers who want to create Chinese whisper audio, this set of weights and inference code is a starting point they can use directly.
One-line Summary
SJTU open-sources Deep ASMR, decoupling whisper style from timbre to achieve whisper synthesis.
Paragraph Summary
The team at Shanghai Jiao Tong University has open-sourced the Deep ASMR project, proposing a speech synthesis method that decouples the whisper "style" from the speaker's "timbre." The model works in two steps: the language model references someone else's whisper to learn reading, pauses, and breathy rhythm; the acoustic decoder references the person's own normal recording to restore timbre. The team compiled 670 hours of Chinese and English recordings, fine-tuned on an existing speech synthesis model, and made the data, code, and training scripts public. In the plugin practice, only the end parameters of the acoustic decoder were fine-tuned, freezing the language model and vocoder, to observe changes in breathiness texture. The current version processes at most one hundred characters at a time and outputs mono vocals; some word onsets are unclear and require manual checking, while background music and spatial effects are left to post-processing. This open-source work provides a reproducible baseline for whisper synthesis, and its decoupling idea can be transferred to other stylized speech tasks.
Related articles

No Coding Required: He Shipped an App Store App by 'Cutting Features' and Nitpicking with AI
A game designer with zero frontend/backend skills shipped a dual-platform App Store app with AI—not via one-click generation, but by cutting scope and nitpicking like a demanding client.

What Is Harness Engineering? From Concept Origins to Enterprise-Level Agent Implementation
What is Harness Engineering? This article traces the concept's origins and analyzes enterprise-level Agent requirements for tool call fault tolerance, data security, tenant isolation, and memory gaps.

Coze in Practice: A Three-Tier Agent Classification and a Multi-Agent Collaboration Guide
A Bilibili tutorial breaks down the multi-agent collaboration toolchain into three categories: building platforms like Coze Studio and Dify, Agent software like Claude Code, and development frameworks like LangChain—plus a Coze demo of industry templates, three-platform interoperability, local Agent integration, and the trade-offs of credit consumption.