ElevenLabs v4 Turbo Hands-On: 150ms Time-to-First-Audio, Voice Agents Say Goodbye to Awkward Pauses

The Overlooked Pain Point of Voice Agents

If you've ever talked to a voice assistant, you've experienced that pause—you finish speaking, wait a few hundred milliseconds or even longer, and only then does it start responding. That delay strips the conversation of any sense of naturalness. The problem isn't the LLM's thinking speed—it's the serial structure of the entire processing pipeline: speech is first transcribed, the LLM generates a complete answer, and only then does the voice model read it out. Three stages executing in sequence, and what the user feels is an unnatural stretch of silence.

According to ElevenLabs' official release info, its newly launched Eleven v4 (quality-first) and v4 Turbo (speed-first) attack this bottleneck from both ends of the pipeline. The Turbo version's official WebSocket benchmark shows a median time-to-first-audio of roughly 150 milliseconds—a number low enough that the pause no longer feels like waiting in a queue, but more like the natural hesitation of a human organizing their thoughts.

Time to First Audio (TTFA) is the core metric for voice agent experience, measuring the time from when the user finishes speaking to when the system starts playing audio. It's different from total response time: users are extremely sensitive to a few hundred milliseconds of "opening delay," but far more forgiving about whether the audio plays smoothly after that. This is exactly the point of streaming TTS—even if full-sentence synthesis still takes seconds, as long as the first audio chunk arrives quickly, the conversation the user perceives is natural. Industry consensus holds that when TTFA drops below 300ms, the conversation approaches human turn-taking rhythm, and at 150ms it's nearly imperceptible as machine latency.

How Do Modern Voice Models Work?

Think of a text-to-speech model as a miniature theater troupe that needs to accomplish three things:

Pronunciation planning—converting letters into phonemes. The same word can be pronounced completely differently depending on context, and context decides everything.

Prosody direction—deciding intonation, rhythm, pauses, and stress. The difference between whispering and shouting the same sentence lies entirely in prosody.

Voice identity—learning a compact "voice embedding" from a short audio sample. Give it 10 seconds of your recording, and it can build your voice fingerprint.

Early voice models were like understudy actors cold-reading a script: line-by-line delivery, flat intonation, and by chapter three of an audiobook the voice started drifting. Eleven v4's new architecture is built around context—it reads through the entire script, understands who's speaking, what just happened, how each line should land, and maintains speaker identity consistency across long texts.

One more detail: it supports inline stage directions: input [softly] I'm not angry, [laughter] I just think it's funny, and the model executes the tags in order—it can even stack multiple directions within a single line. This is the key leap from "text reader" to "voice actor."

A phoneme is the smallest meaning-distinguishing unit in speech, such as the difference between /p/ and /b/ in English. TTS systems convert text into a phoneme sequence and then map it to acoustic features—this intermediate step lets the model handle cases where spelling and pronunciation diverge (like the past and present tense of "read"). A speaker embedding is a fixed-dimension vector extracted from reference audio that captures the speaker's timbre, range, and pronunciation habits, so the model maintains the same identity when synthesizing new text. Together, these two form the basis of modern zero-shot voice cloning: a small amount of reference audio is enough to extract an embedding, with no need to retrain the model for each speaker.

Two Models, Two Positions

ElevenLabs launched two variants based on the same research, targeting completely different scenarios:

	Eleven v4	Eleven v4 Turbo
Use cases	Audiobooks, dubbing, character performance, ads	Voice agents, real-time calls, real-time assistants
Latency	Quality-first	~100ms median inference, ~150ms time-to-first-audio (vendor streaming test data)
Streaming	REST + streaming endpoints	Bidirectional streaming: audio starts before the sentence ends
Languages	90+ (up from 70 in v3)	Same
Cloning	10-second instant voice cloning; professional voice cloning restored after v3 removal	Same

The most noticeable quality improvements are in Japanese, Brazilian Portuguese, Mandarin, and Cantonese—precisely the languages where previous models performed worst. ElevenLabs also claims it took the number one spot on the Artificial Analysis Speech Arena at launch, including the highest pronunciation robustness score in the benchmark's history. It's worth noting this comes from a third-party evaluator rather than independent reproduction, but it remains the most comparable reference data in the industry today.

How Turbo Achieves 150 Milliseconds: Streaming, Not Queuing

A typical voice agent turn's latency budget looks roughly like this: endpointing ~180ms + streaming ASR ~220ms + LLM first token ~650ms + TTS first chunk + network transmission. Turbo's core task is compressing the green bar (the TTS portion), and its trick is architectural rather than faster chips: overlapping speech synthesis with text generation.

Batch TTS models wait for the LLM to finish an entire sentence before synthesizing, so time-to-first-audio grows linearly with sentence length. Turbo, by contrast, outputs audio chunks while receiving tokens, so first audio can emerge within an extremely short time after generation begins—independent of sentence length. In simulation experiments, this gap widens dramatically as sentences get longer: about 3.5x faster at 20 tokens and about 14x faster at 80 tokens. Bidirectional streaming means the model doesn't even need to wait for a sentence boundary—it can start generating audio mid-thought.

Two additional capabilities are critical for production environments.

Long-text context stitching. v4's architecture maintains voice identity consistency across regenerated lines—replace one sentence in chapter 12, and the newly recorded line still sounds like the chapter 12 narrator. This eliminates the biggest pain point in audiobook editing.

Multi-speaker overlap. Dialogue gains the timing and reactive energy of real conversation—speakers interrupting each other, responding spontaneously—rather than the polite A-then-B. This is a hard generative problem that requires modeling turn-taking dynamics, directly targeting AI NPCs and the dubbing market.

Bidirectional streaming means the client and server simultaneously send and receive data over a persistent connection, rather than "send a request, wait for a response." In voice scenarios, this means every token the LLM emits can be immediately consumed by TTS to start synthesizing corresponding audio, and audio chunks can be pushed to the client for playback while still being generated. Endpointing, meanwhile, is responsible for judging whether the user has finished speaking—it needs to distinguish "a brief breath" from "a genuine end," and is an easily overlooked yet hugely impactful part of the latency budget: too conservative detection introduces extra waiting, while too aggressive detection truncates the user's speech.

The Competitive Landscape

Voice models have become a two-axis race—speed and expressiveness—and the number of contenders is surging: Cartesia (Sonic 3.6), OpenAI (GPT-4o mini TTS), Google (Gemini Flash TTS), plus startups like Deepgram, Fish Audio, and Boson. ElevenLabs benchmarked Turbo against Cartesia and OpenAI after subtracting network latency—but any benchmark data should be re-validated under your own text lengths, voice types, and network regions. Trend-wise, Turbo's combination of ~150ms time-to-first-audio with full expressive range has become the threshold every competitor must clear.

ElevenLabs also revealed that enterprise call business now accounts for more than 55% of its overall business, and v4 knows how to handle confrontation, escalation, and waiting scenarios—the voice agent economy has essentially become the voice model economy.

Key Takeaways

The latency budget is the product itself. In voice agents, the difference between 150ms and 700ms time-to-first-audio is the difference between an "intelligent assistant" and a "phone tree menu."

Streaming is architecturally superior to batch. Overlapping synthesis with generation decouples time-to-first-audio from sentence length—this is an architectural move that hardware acceleration can't replace.

Direction tags are the new programming language of voice. [softly], [laughter], stacked sequences, IPA control—prompt engineering has entered the audio domain, and it only works because the model reads context rather than merely recognizing characters.

10 seconds is enough to clone your voice. Instant voice cloning needs only 10 seconds of audio, and this capability line deserves serious treatment—the safety guardrail question (ElevenLabs requires verification for professional voice cloning) is now a load-bearing structure of every deployment plan.

One-line Summary

ElevenLabs launches Eleven v4 and v4 Turbo, using a streaming architecture to push voice agent time-to-first-audio down to about 150ms.

Paragraph Summary

ElevenLabs introduces Eleven v4 (quality-first) and v4 Turbo (speed-first), directly targeting the pause pain point in voice agent conversations caused by serial pipelines. Turbo achieves ~150ms time-to-first-audio through a bidirectional streaming architecture that overlaps speech synthesis with LLM text generation, decoupling time-to-first-audio from sentence length and running about 14x faster than batch mode at 80 tokens. v4 is built around context, supporting inline stage directions, voice identity consistency across long texts, and multi-speaker overlapping dialogue, with significant improvements in Japanese, Mandarin, and other previously weak languages. The article notes that voice model competition has evolved into a two-axis race of speed and expressiveness, and that latency budgets, streaming architectures, and direction-tag programming are becoming the core competitiveness of voice agent products.