The 'Unsolicited Speech' Problem in Full-Duplex Voice LLMs: Causal Analysis and Real-Time Mitigation

Full-duplex voice large language models are transforming the way humans interact with machines. Speech-to-Speech LLMs like Moshi and its derivative PersonaPlex can listen and speak simultaneously, enabling truly concurrent generation. But these models have an awkward flaw: during extended periods of user silence, they tend to "fill the void" by spontaneously starting to talk. A study published on arXiv systematically investigates the root causes of this phenomenon and proposes a mitigation approach that requires no retraining and can run in real time.

Why Full-Duplex Voice Models Talk to Themselves

Full-duplex generation is the core selling point of this class of voice LLMs — the model can organize and output its own speech while simultaneously listening to user input, rather than waiting for the user to finish before responding. This more closely mirrors the natural rhythm of human conversation. However, this capability comes with a side effect.

In their experiments, researchers fed the models a "digital-zero" signal (i.e., pure silence) and observed their behavior over five-minute continuous segments. The results showed that Moshi produced unsolicited speech in 12 out of 40 trials, while PersonaPlex did so in 11 out of 40. In other words, roughly one in four times, these models would spontaneously start speaking even when the user was completely silent. This behavior — referred to as "spurious onsets" — significantly undermines the naturalness and reliability of conversational interactions.

Is It a Sampling Problem or a Conditional Dependency Problem?

To find the root cause, researchers proposed and tested two hypotheses.

Hypothesis 1: Repeated Sampling Triggers Accidental Onsets

The first possibility is that even when the model assigns a consistently low probability to speech onset, the continuous stream of sampling over time eventually "hits" a low-probability event, triggering vocalization. Under this mechanism, the problem would essentially be a probability leak accumulated over long-duration generation.

Hypothesis 2: Conditional Feedback from the Model's Own Output

The second possibility is more subtle: the model conditions on its own previously generated "non-speech" output, which in turn causes the onset probability to suddenly spike. In other words, the model's internal state creates a kind of positive feedback loop.

Experimental data clearly supported the second hypothesis. The researchers found that at every observed spurious onset, the speech probability surged by more than nine orders of magnitude within a single 80-millisecond frame. This cliff-like jump indicates the problem is not a gradual accumulation from random sampling, but rather a probability spike driven by the model's conditional dependence on its own historical output.

Precise Mitigation via Causal Counterfactual Reasoning

With the root cause identified, the real challenge became: how to suppress spurious onsets without accidentally suppressing genuine responses? A blunt approach that simply suppresses all speech tendencies might cause the model to go silent even when the user genuinely needs a reply.

The researchers' approach is elegantly conceived — they introduce a causal counterfactual question: Is the model's current vocalization a response to the user's speech, or would its next-token distribution remain nearly unchanged even if the prior user input were silenced?

If "erasing" the user input leaves the model's output distribution nearly unchanged, it indicates the onset is unrelated to the user — a spurious onset that should be suppressed. Conversely, if the distribution changes significantly, the model is genuinely reacting to the user's speech and should be allowed to respond. This intervention-based criterion essentially distinguishes between "user-driven" and "model-driven" speech behavior.

Empirical Results and Real-Time Performance

The method was validated in real microphone noise conditions. With 40 held-out trials per model, the results were impressive:

• Spurious onset suppression: Successfully suppressed all 13/13 spurious onsets for Moshi and all 9/9 for PersonaPlex;
• Genuine response preservation: All 40/40 genuine responses were preserved for both models, with zero false suppressions;
• Real-time capability: The method operates at inference time with no model retraining required, and can run in real time — the 95th-percentile decision latency is under 61 milliseconds, well within the 80-millisecond single-frame budget.

In short, this approach nearly perfectly solves the "unsolicited speech" problem without modifying model weights or introducing significant latency. For voice assistants intended for production deployment, the plug-and-play inference-time nature of this solution is highly attractive.

Implications for Voice Interaction Products

As voice LLMs move toward full-duplex, low-latency natural interaction, "knowing when to speak" and "knowing when to stay silent" become critical details that define the user experience. The value of this research lies not only in providing an engineering solution, but also in using causal analysis to clarify the nature of the problem: spurious onsets stem from the model's conditional dependence on its own output, not from sampling noise.

The counterfactual intervention criterion also offers a generalizable framework — using a hypothesis test of "what would happen if the input didn't exist" to determine whether model behavior is truly driven by external signals. This class of methods may extend to many other scenarios where distinguishing "proactive generation" from "reactive response" is important. The research team has open-sourced the relevant code on GitHub for community reproduction and extension.