FD-VAD: A New Approach to Semantic Endpoint Detection for Full-Duplex Speech

In natural human-machine voice conversations, a deceptively simple yet deeply challenging question persists: how does a machine know whether a user has genuinely finished speaking, or is merely pausing to think? This capability — known as "turn-taking" — is central to the fluency of full-duplex voice interaction. A new study published on arXiv introduces FD-VAD, a method designed specifically to address this pain point.

The Limitations of Traditional Approaches

Current mainstream endpoint detection solutions fall into two broad categories, each with significant drawbacks.

The first is acoustic-based Voice Activity Detection (VAD). These methods detect the presence or absence of speech in an audio signal to identify pauses. They are simple to implement and low-latency, but they completely lack semantic understanding. When a user says "I'd like to book a flight to Beijing…" and then pauses to think, a purely acoustic VAD cannot distinguish between an unfinished sentence and a completed thought — it can only "hear that the sound stopped," but cannot "understand whether the sentence is done."

The second is cascaded ASR-based endpoint detection, which first transcribes speech to text and then uses the text to determine whether the intent is complete. This approach gains semantic information, but at a steep cost: it introduces a hard dependency on transcription accuracy, adds extra processing stages, and accumulates both recognition errors and latency.

VAD technology itself has evolved over many years — early methods relied on signal features like energy thresholds and zero-crossing rates, while modern solutions widely use neural network models (such as Silero VAD or WebRTC VAD) to classify speech versus non-speech frames. But regardless of implementation, VAD's fundamental goal has always been "is there sound?" rather than "is the utterance complete?" In practice, systems typically apply a fixed silence timeout window (e.g., 500ms of silence triggers an end-of-turn decision), which leads to two systematic errors: premature cut-offs during thinking pauses (false interruptions), and artificially extended wait times to avoid those cut-offs, which in turn increases perceived latency. This "acoustic blind spot" is a structural flaw that pure VAD approaches cannot fundamentally resolve.

FD-VAD's Core Idea: Framing Endpoint Detection as Causal Reasoning

FD-VAD's innovation lies in redefining the problem. The researchers model semantic endpoint detection as a causal audio-language reasoning task — directly mapping a bounded causal audio window to a binary "Continue / Stop" decision, with no ASR involved at any stage.

"Causal" here means the model can only use current and past audio information to make its decision, in accordance with the requirements of streaming real-time inference — it cannot "peek" at future speech content. This constraint is critical for low-latency online voice interaction.

Architecturally, FD-VAD uses a lightweight modular design:

• A frozen speech encoder extracts audio features without participating in training, conserving compute;
• A lightweight modality adapter bridges speech representations to the language model's embedding space;
• A parameter-efficiently adapted language model (LM) handles semantic reasoning, with parameter-efficient fine-tuning reducing training cost.

Combined with a last-chunk training objective designed specifically for streaming inference, FD-VAD achieves real-time performance while leveraging the semantic understanding capabilities of a language model.

Parameter-Efficient Fine-Tuning (PEFT) refers to a family of methods that adapt a model to new tasks by updating only a tiny fraction of its parameters. Representative techniques include LoRA (Low-Rank Adaptation) and Prefix Tuning. The motivation is straightforward: large language models have billions of parameters, making full fine-tuning computationally prohibitive and prone to catastrophic forgetting. PEFT methods inject a small number of trainable modules alongside the original weights, achieving performance close to full fine-tuning with less than 1% of parameter updates. In FD-VAD, applying PEFT to the language model serves a dual purpose: it preserves the semantic knowledge acquired during pre-training (the model's "sense of language"), while dramatically reducing the training cost for the endpoint detection task — making the system practically viable for resource-constrained online inference scenarios.

Two Key Technical Design Choices

Beyond the overall architecture, FD-VAD introduces two targeted techniques to improve real-world performance.

Confidence-Gated Endpoint Commitment

In conversation, both premature interruptions and sluggish responses degrade the user experience. FD-VAD employs a confidence-gated endpoint commitment mechanism, using a confidence threshold to balance the trade-off between "interruption risk" and "latency." A stop decision is only committed when the model is sufficiently confident that the user has finished speaking, reducing false interruptions while maintaining responsiveness.

Boundary-Focused Hard-Negative Sampling

The hardest cases to judge are those at ambiguous turn boundaries — such as natural pauses, elongated prosody, or trailing off. FD-VAD uses boundary-focused hard-negative sampling to deliberately strengthen the model's ability to discriminate at these ambiguous boundaries during training, providing more learning signal precisely where the model is most likely to err.

Experimental Results: Leading Performance in Zero-Shot Settings

FD-VAD was evaluated on both in-domain and conversational benchmarks. Results show that FD-VAD outperforms multiple strong streaming and non-streaming semantic turn classifier baselines.

The most compelling result comes from the TurnBench development set: under a zero-shot setting, FD-VAD achieves the highest EOT (End-of-Turn) recall of 0.853 among qualifying systems, subject to a false positive rate constraint of FP ≤ 0.10. Zero-shot means the model was not trained on TurnBench data at all, making this result a strong indicator of generalization.

TurnBench is a benchmark dataset specifically designed to evaluate end-of-turn detection in dialogue, covering speech samples from diverse real-world conversational scenarios — including natural pauses, hesitations, and incomplete sentence structures. The key evaluation metric is "maximize EOT recall subject to a False Positive Rate not exceeding a given threshold" — a setup that directly mirrors real product requirements: the system must control how often it falsely interrupts the user (low FP), while minimizing missed end-of-turn events (high recall). The zero-shot setting means FD-VAD used none of TurnBench's training data, contrasting with many baseline systems that fine-tune on in-domain data. The 0.853 recall therefore reflects genuine generalization rather than in-domain data fitting.

Significance and Outlook

FD-VAD's central message is clear: semantic endpoint detection can be performed directly on streaming audio — without intermediate ASR transcription and without dialogue state tracking. This breaks the assumed dependency that "understanding semantics requires converting to text first."

For full-duplex interaction scenarios such as voice assistants, real-time interpretation, and customer service bots, this ASR-free architecture means shorter response pipelines, lower latency, and more natural conversational rhythm. That said, as a new method, its robustness at larger scales in real-world deployments, its adaptability to multiple languages, and its integration with downstream dialogue systems all warrant further investigation.

Overall, FD-VAD offers a viable path for addressing the turn-taking challenge in full-duplex voice interaction — one that balances semantic understanding with real-time performance, and represents a meaningful step in the evolution of endpoint detection from "hearing sound" to "understanding intent."