Gemini Real-Time Voice Agent in Practice: Building a Multimodal Agent That Can Listen, See, and Speak

At the AI Engineer conference, Thor from Google DeepMind's Developer Experience team gave a live demonstration on building real-time voice agents with the Gemini API. This session didn't stay at the conceptual level—it used multiple interactive demos to showcase the actual capabilities of the current Gemini model family in multimodal real-time interaction. For developers building voice agents, the architectural approach and API design here are worth examining in detail.

From TTS to Real-Time Conversation: The Layered Capabilities of Gemini Audio Models

Thor opened by playing an expressive voice clip and asked the audience to guess which model produced it. The answer was Gemini TTS—a speech synthesis model that supports streaming output, low latency, and style control through inline audio text. This forms the "output layer" foundation of real-time voice agents.

The real conversational intelligence comes from Gemini 3.1 Flash Live, a native audio model. It differs fundamentally from traditional cascaded voice agent pipelines (ASR to text → LLM reasoning → TTS synthesis): the model doesn't route through text at all. Intelligence is baked directly into the audio model, so it natively understands speech and responds directly in audio, supporting approximately 90 languages during the preview phase.

The direct benefits of this architectural choice are reduced latency and smooth language switching. In the demo, Thor seamlessly switched between English, German, and Chinese within the same conversation, and the model handled all of them naturally. For real-time applications requiring multilingual support, this native multilingual capability is far simpler than stitching together multiple monolingual models.

The Core Difference Between Cascaded Voice Pipelines and Native Audio Models

A traditional cascaded voice agent (ASR → LLM → TTS) requires a model call at each stage, accompanied by a modality conversion between audio and text. This transcription step loses paralinguistic information such as tone, pauses, and emotion, while the serial calls stack up to produce end-to-end latency typically exceeding 1 second. A native audio model (like Gemini 3.1 Flash Live) understands and generates directly in the audio token space, eliminating intermediate transcription, preserving speaker emotion and speech rate characteristics, and significantly reducing latency. The trade-off is that outputs are harder to audit word by word, and observability is weaker than with text intermediates—so cascaded approaches still have a place in scenarios requiring strict compliance or strong controllability.

Live API: Stateful WebSocket-Driven Multimodal Sessions

The core interface for building real-time conversational agents is the Live API. It's a low-latency real-time voice and vision API built on a stateful WebSocket design: after developers create a session, they can continuously send text, audio, and video frames over the WebSocket, and the model streams back audio chunks in real time along with accompanying transcription text.

Key features include: support for 90 languages, barge-in (users can interrupt model output at any time), tool calling capabilities, and audio transcription. Barge-in may seem like a detail, but it's the dividing line for natural voice interaction—without it, an agent feels more like a broadcaster you can't interject into.

Thor specifically noted that real-time media poses significant challenges at scale, which is why the ecosystem includes partners like LiveKit and Pipecat offering quick-start solutions. These teams were also present at the conference expo area.

The Difference Between Stateful WebSocket and Stateless HTTP Calls

Stateless HTTP requests require each call to carry full context, making them suitable for one-shot inference. Real-time voice conversations, however, require the model to continuously remember previous rounds of audio, video frames, and tool call results. With the Live API's stateful WebSocket, the server maintains context during the connection, so developers only need to incrementally send new audio or video frames, and the model streams back audio chunks. This design naturally supports bidirectional streaming and is also a prerequisite for barge-in—the server can receive new user audio during model output and immediately interrupt generation without rebuilding the connection or replaying history.

Live Demo Breakdown: Radio DJ and Story Canvas

The first demo was a "request-a-song radio station": Gemini Live played the DJ, using function calling to trigger the Lyria 3 music generation model to compose on the spot in the user's specified style. When the audience requested a high-energy reggaeton about the AI Engineer conference, the model not only generated complete lyrics and melody but also naturally mixed in Spanish.

The business value of this demo isn't in "requesting songs" itself, but in demonstrating the complete capability combination of a real-time voice agent: multilingual conversation, tool calling, and on-demand triggering of downstream generative models. The same pattern could be swapped to query a CRM for customer information or create a ticket in the background.

The second demo was a "story canvas": Gemini Live generates story text, then calls Nano Banana Lite through the Interactions API to generate illustrations. The Interactions API is Google DeepMind's unified model and agent interaction interface; its stateful nature allows passing the ID of a previous interaction, thereby maintaining consistency of characters and visual style across consecutive generations. The demo generated a story about the origin of bubble tea set in a 1980s Taipei night market, with image generation taking about two to three seconds.

The Relationship Between Function Calling and Search Grounding

Function calling is the mechanism by which the model proactively initiates structured requests during conversation, which external systems execute and return results for—both Lyria and Interactions API triggers rely on it. Google Search grounding is essentially a platform-built-in form of function calling: when the model determines it needs real-time information, it automatically retrieves and cites search results, and developers don't need to implement the retrieval pipeline themselves. Together they form the "action layer" of real-time agents, determining whether an agent can move from pure conversation to actually executing tasks.

Instant Translation and Visual Understanding: Two Easily Overlooked Pieces

Gemini 3.5 Live Translate demonstrated near-real-time speech translation: as the speaker talks, the model automatically detects the language and translates it into the language selected by the listener. On-site attendees could scan a QR code with their phones to listen in German or Chinese. For international conferences and cross-border collaboration scenarios, this capability can be embedded directly into applications.

On the visual understanding side, Thor demonstrated Gemini 3.1 Flash Live in AI Studio with Google Search grounding enabled: the model could see through the camera his black cap and sweatshirt with yellow "Gemini" lettering, and explain that "Zwillinge" means "twins" in German. Search grounding acts as built-in function calling, allowing the model to access real-time information like weather.

How Developers Can Get Started

The most direct entry point is ai.studio/live (or ai.dev/live), where you can converse directly with the model in the browser, switch voices (such as Puck), and enable search grounding. Thor mentioned that the demo itself was vibe-coded with Gemini in Google AI Studio, and developers can fork it and continue building on it.

From an engineering perspective, the division of labor in this tech stack is already fairly clear: the Live API handles real-time multimodal sessions, the Interactions API handles stateful generation chains, and Gemini TTS and Live Translate fill in specialized capabilities for voice input and output. For teams building real-time voice agents, what's worth paying attention to is the trade-off between native audio models and cascaded pipelines—the former wins on latency and naturalness, while the latter is more mature in controllability and observability. Thor's demo shows that the native approach has entered a stage where it can be practically deployed.

One-line Summary

Google DeepMind live-demos the architecture and practical capabilities of building real-time voice agents with the Gemini API.

Paragraph Summary

Based on a live demonstration by Thor from Google DeepMind's Developer Experience team at the AI Engineer conference, this article systematically breaks down the tech stack for building real-time voice agents with the Gemini API. Gemini TTS forms the streaming, low-latency output layer; Gemini 3.1 Flash Live, as a native audio model, skips text intermediation, supports about 90 languages, and achieves low latency with smooth switching. The core Live API uses stateful WebSockets, supporting audio/video frame input, audio chunk streaming, barge-in interruption, and tool calling. Two demos show function calling triggering Lyria 3 for real-time composition, and the Interactions API using interaction IDs to maintain consistency across consecutive generations. The article also covers Live Translate's near-real-time translation and search-grounded visual understanding, and provides developer onboarding paths and ecosystem partners.