ElevenLabs Realtime Launches: A Voice Agent Model with 570ms Time to First Audio

Title

ElevenLabs Realtime Launches: A Voice Agent Model with 570ms Time to First Audio

Meta Description

ElevenLabs launches the Realtime voice agent model series, with v1 Mini and v1 Max achieving 620ms and 570ms time to first audio respectively. It supports 90+ languages, is compatible with the OpenAI Realtime API, and offers end-to-end tuning of speech recognition, turn management, and speech generation.

Article Content

ElevenLabs has launched its new Realtime voice agent model series on Product Hunt, headlined by a 570-millisecond time to first audio. The product tunes speech recognition, conversational turn management, the large language model, and speech synthesis end to end, aiming to make voice AI interactions feel closer to human conversation. For developers building voice assistants, customer service bots, or real-time interactive applications, this is a development worth watching.

End-to-End Tuning: From a Fragmented Pipeline to a Unified Model

Traditional voice agents are usually assembled from multiple independent components: speech recognition (ASR) converts audio to text, an LLM understands and generates a reply, and TTS then synthesizes speech. Each stage introduces latency and error, and is hard to optimize globally. ElevenLabs Realtime adopts a speech-to-speech model architecture, tuning recognition, turn management, the LLM, and speech generation together within a single end-to-end framework.

The core advantage of this approach is reducing losses in the intermediate stages. According to the official description, the model can handle pauses, interruptions, and turn switches in conversation more naturally, rather than relying on external VAD (voice activity detection) or rule engines. For scenarios that require real-time responsiveness—such as voice customer service, AI companions, or real-time translation—this integrated design is expected to significantly improve fluency.

Voice activity detection (VAD) is a key component in traditional voice pipelines, used to determine when a user starts speaking, pauses, or finishes. Typical VAD is based on energy thresholds or lightweight machine learning models, analyzing the audio stream in real time locally or in the cloud. Its accuracy directly affects the turn-switching experience: judging the end too early interrupts the user, while judging too late causes response delays. In cascaded architectures, VAD usually runs as an independent module, loosely coupled with ASR and dialogue management logic, making it hard to adjust dynamically based on context. End-to-end models instead internalize turn detection as part of the model's capability, making joint decisions based on semantic and acoustic cues, thereby reducing reliance on independent VAD rules and performing more robustly in noisy environments or when the user's speaking pace changes.

Time to first audio is a core metric for measuring the responsiveness of a voice agent. It refers to the interval from when the user finishes speaking (or the system determines the turn has ended) to when the agent outputs its first audio packet. It differs from end-to-end latency, which also includes audio transmission over the network and playback buffering time. Time to first audio directly affects users' subjective sense of waiting; the industry generally considers under 800 milliseconds acceptable, and under 500 milliseconds close to the rhythm of natural conversation. Optimizing this metric requires compressing the time consumed by ASR, LLM inference, and TTS. End-to-end models reduce serial processing stages through shared representations and joint inference, which is the main technical path to lowering time to first audio.

Dual-Version Strategy: The Trade-offs Between Mini and Max

ElevenLabs Realtime initially launches two versions: v1 Mini and v1 Max. v1 Mini targets natural conversation scenarios with a time to first audio of 620 milliseconds; v1 Max targets complex tasks, pushing latency down further to 570 milliseconds. Both support more than 90 languages.

Here, "time to first audio" refers to the interval from when the user finishes speaking to when the agent begins to speak. 570 milliseconds is already close to the range of natural pauses in human conversation (typically 200–600 milliseconds), meaning the interaction experience can achieve almost imperceptible waiting. However, the official announcement does not disclose the specific differences between the two versions in model size, inference cost, or task complexity, distinguishing them only as "natural conversation" and "complex tasks." Developers will need to test based on their actual scenarios before choosing.

OpenAI Realtime API Compatibility: Extremely Low Migration Cost

One noteworthy engineering decision is that ElevenLabs Realtime provides an interface compatible with the OpenAI Realtime API. Developers only need to change the endpoint and model name to switch existing applications built on the OpenAI Realtime API over to ElevenLabs' models. This greatly reduces migration costs and also signals that ElevenLabs intends to align with the mainstream ecosystem at the API level, attracting teams that want to try different voice models without rewriting code.

In addition, the model can be used directly on the ElevenAgents platform, offering a managed solution for teams that don't want to deal with the underlying API. This dual-track strategy of "API compatibility + managed platform" covers different needs, from flexible integration to rapid deployment.

The OpenAI Realtime API is an interface specification introduced by OpenAI for real-time voice conversation, supporting both WebSocket and WebRTC transport and allowing developers to send audio and receive voice replies in a streaming manner. Its core design includes session management, an event-driven message protocol, and a built-in turn detection mechanism. Because this API has already seen relatively broad adoption in the developer community, forming a de facto interface standard of sorts, other vendors choosing to be compatible with its endpoint structure and message format can significantly reduce developers' switching costs. ElevenLabs Realtime's adoption of a compatible interface means that the authentication methods, event types, and audio codec flows in existing code can largely be reused—only the service address and model identifier need to be replaced to complete migration. This is especially practical for teams that need multi-model comparison or disaster-recovery switching.

Competition in the Voice Agent Market Heats Up

ElevenLabs is known for high-quality TTS, and the launch of an end-to-end voice agent model marks its extension from a "speech synthesis tool" to "conversational AI infrastructure." The voice agent space is currently highly competitive, with OpenAI's Realtime API, Google's Gemini Live, and a number of startups all vying for developers' attention. ElevenLabs' differentiation lies in its accumulated quality in speech generation and its low-latency engineering capabilities.

However, the vote count (108) and comment count (2) on the Product Hunt page indicate that the product is still in an early exposure phase, with limited community discussion. Whether actual performance matches the advertised latency figures, and the recognition accuracy in multilingual scenarios, still need third-party evaluation and verification.

How Developers Should Evaluate It

For teams building voice interaction products, ElevenLabs Realtime is worth adding to the test list. Three points deserve particular attention: first, end-to-end latency in real network environments, not just the time-to-first-audio metric; second, the naturalness of turn management, especially in scenarios where users interrupt or go silent; third, the actual coverage quality of the 90+ language support, especially for non-English languages. Because it is compatible with the OpenAI Realtime API, A/B testing can be done at low cost to compare completion rates and satisfaction across different models in real user scenarios.

The experience competition in voice agents has only just begun, with latency and naturalness as the two core axes. The launch of ElevenLabs Realtime gives developers a new option, but the ultimate winner will still depend on stability and cost-effectiveness after large-scale deployment.

One-line Summary

ElevenLabs launches the end-to-end voice agent Realtime, with time to first audio as low as 570 milliseconds and compatibility with the OpenAI interface.

Paragraph Summary

ElevenLabs has launched the Realtime end-to-end voice agent model series on Product Hunt, tuning speech recognition, turn management, the LLM, and speech synthesis together to achieve a time to first audio as low as 570 milliseconds. The product is split into v1 Mini (620 milliseconds, for natural conversation) and v1 Max (570 milliseconds, for complex tasks), and supports more than 90 languages. Its interface is compatible with the OpenAI Realtime API, so developers can migrate by simply changing the endpoint and model name, and a managed solution is available directly on the ElevenAgents platform. This marks ElevenLabs' extension from a speech synthesis tool to conversational AI infrastructure, but actual latency and multilingual performance still await third-party verification.