Teaching Speech LLMs to Breathe: New Progress in Non-Verbal Vocalization Modeling

Speech interaction experiences are improving rapidly, thanks to advances in paralinguistic attribute modeling and speech reconstruction paradigms. But those seemingly trivial non-verbal vocalizations in human conversation—breaths, sighs, laughter, pauses—are precisely the key to naturalness. A recent study systematically covers 13 categories of non-verbal vocalizations (NVVs) and proposes methods to enable speech LLMs to "breathe" appropriately during dialogue.

Why Non-Verbal Vocalizations Matter So Much

Human communication follows a complex set of non-verbal norms. The rhythm of breathing hints at turn-taking, a sigh carries emotional weight, and laughter marks moments of empathy. These NVVs span three dimensions—respiration, emotion, and discourse structure—forming the underlying skeleton of conversational naturalness.

The fundamental limitation of current speech interaction models is this: they can say the right words but miss the "human touch" of speaking. The original paper notes that ignoring these events "imposes inherent limitations on the naturalness of current methods." In other words, a model can enunciate perfectly yet struggle to make one feel it is truly "conversing."

13 Categories of Non-Verbal Vocalizations and Data Construction

The study covers 13 major NVV types. This level of granularity means the model needs to learn not vague "tones" but specific vocal events and their positions in dialogue.

To support training, the team built a natural dialogue dataset conforming to human conversational norms through an automated synthesis pipeline. The value of this approach lies in scalability—manually annotating non-verbal vocalizations is extremely costly, while automated synthesis makes large-scale, diverse dialogue scenarios possible while maintaining consistency with conversational norms.

An automated synthesis pipeline typically involves multiple steps: first extracting text and prosodic features from large-scale dialogue corpora, then inserting non-verbal vocalization labels at specific positions using rules or generative models, and finally generating audio with NVVs via speech synthesis technology. The advantage of this approach is the ability to generate annotated data at scale while controlling the type, frequency, and contextual distribution of NVVs, ensuring the data conforms to human conversational norms. Compared to manual recording and annotation, the synthesis pipeline significantly reduces costs and allows flexible adjustment of data distribution to cover diverse dialogue scenarios.

Multi-Stage Fine-Tuning and NVV-Aware GRPO

The methodological core has two parts. The first is multi-stage fine-tuning, which enables the model to gradually acquire the ability to generate contextually appropriate NVV events rather than forcefully injecting them all at once.

The second is NVV-aware GRPO (Group Relative Policy Optimization). GRPO itself is a reinforcement learning optimization strategy based on group-relative comparison. The key modification here is making the optimization objective "aware" of non-verbal vocalizations—that is, the reward signal cares not only about what is said but also about where and how breaths or sighs are inserted. Incorporating paralinguistic events into the optimization loop is a crucial step in transforming naturalness from "decoration" into "capability."

GRPO (Group Relative Policy Optimization) is a reinforcement learning algorithm whose core idea is to estimate the advantage function through relative comparison among a group of sampled outputs, rather than relying on a separate value network. Specifically, for the same input, the model generates multiple candidate responses, and GRPO computes the relative merit of each candidate through intra-group normalization to update the policy. This approach reduces training costs and improves stability. In this paper, the key modification of NVV-aware GRPO lies in the design of the reward function: in addition to traditional text quality rewards, it introduces contextual appropriateness evaluation of non-verbal vocalization events (such as breaths and sighs), enabling the optimization process to guide the model to generate the right paralinguistic events at the right positions.

Empirical Improvements Across Two Model Paradigms

The study validated the approach on different modeling paradigms, significantly enhancing the expressiveness of VocalNet and VITA-Qinyu in empathetic interaction and role-playing dialogue.

The two models represent different architectural routes, meaning the benefits of this method are not tied to a single implementation. In empathetic interaction, appropriate breaths and sighs convey understanding and companionship; in role-playing, non-verbal vocalizations are part of a character's personality. The original paper further discusses the architectural necessity for achieving high-quality natural speech generation—implying that not all architectures benefit equally from NVV modeling, and that architectural choices themselves affect the ceiling of naturalness.

VocalNet and VITA-Qinyu represent two different architectural paradigms for speech LLMs. VocalNet may adopt an end-to-end speech-to-speech generation approach, while VITA-Qinyu may be based on a cascade or fusion architecture from text to speech. Different architectures have different intrinsic capabilities in handling paralinguistic events: for example, end-to-end models may model acoustic details more directly, while cascade models may rely more on intermediate text representations. Therefore, the degree of benefit from NVV modeling methods varies by architecture, and architectural choices themselves determine the upper limit of naturalness a model can achieve.

Implications for Speech Interaction

This work pushes the competitive dimension of speech LLMs from "speaking accurately" to "speaking like a human." When paralinguistic events are systematically modeled, a voice assistant's silence is no longer awkward, and a virtual character's emotions are no longer flat.

More noteworthy is the methodological signal: the combination of data synthesis pipelines, multi-stage fine-tuning, and NVV-aware reinforcement learning constitutes a reusable framework for injecting paralinguistic capabilities. For researchers and product teams focused on the naturalness of speech interaction, this provides a clear technical reference—naturalness is not a post-processing problem but a core issue that needs to enter training objectives and architectural design.

One-line Summary

Research systematically models 13 categories of non-verbal vocalizations, improving dialogue naturalness of speech LLMs through data synthesis and NVV-aware GRPO.

Paragraph Summary

Addressing the issue that speech LLMs ignore non-verbal vocalizations (NVVs), this study systematically covers 13 categories of NVVs (breaths, sighs, laughter, pauses, etc.) and points out that existing models have naturalness limitations due to ignoring these events. The team built a dataset conforming to human conversational norms through an automated synthesis pipeline, adopted multi-stage fine-tuning and NVV-aware GRPO reinforcement learning optimization, enabling models to generate appropriate NVVs in context. Validated on two models with different architectures, VocalNet and VITA-Qinyu, it significantly improved expressiveness in empathetic interaction and role-playing dialogue. The study also discusses the impact of architectural choices on the ceiling of naturalness, providing a reusable technical framework for injecting paralinguistic capabilities.