Voice Agent Sandwich Architecture in Practice: STT+LLM+TTS Latency Optimization & Pitfall Guide

The Architecture Battle for Voice Agents

Building a truly production-ready enterprise voice agent is far more complex than it sounds. It needs to listen, speak, and possess all the capabilities an Agent should have — iterative loops, intent recognition, tool calling, and context management. What makes it even trickier is that these capabilities often inherently conflict with one another.

Based on real-world demonstrations from technical talks on Bilibili, the underlying architectures for voice agents can currently be divided into three categories: the Sandwich Architecture, the native real-time Speech-to-Speech Architecture, and a Hybrid Architecture that blends both. This article draws on real project experience to dissect the design principles and key pitfalls of the Sandwich Architecture.

The Fundamental Differences Between Three Mainstream Architectures

Sandwich Architecture: STT + LLM + TTS

The Sandwich Architecture is currently the most common and pragmatic approach to building voice agents. Its core idea is simple: place a powerful large language reasoning model (LLM) in the middle, with a Speech-to-Text (STT) model in front for recognition and a Text-to-Speech (TTS) model at the back for audio output.

Like a sandwich, this architecture wraps the Agent's reasoning model between voice input and output layers. Its biggest advantage is that you can freely choose the strongest reasoning model available as the Agent's brain, without being constrained by whether the model natively supports voice.

Native Real-Time Speech-to-Speech Architecture

The second approach is the native real-time Speech-to-Speech architecture. It requires finding a multimodal large model capable of real-time voice interaction — and the keyword here is "real-time."

A truly voice-native multimodal model must support not only voice, images, text, and video as inputs, but also voice as output. This point is frequently misunderstood. Many people assume certain well-known multimodal models meet the criteria, but in reality, most of them support multimodal input while only outputting text.

According to hands-on testing by practitioners, there are virtually no mature and stable real-time voice multimodal models available today — neither domestically nor internationally.

Hybrid Architecture

The third approach combines the Sandwich and native real-time architectures, offering the advantages of both but with significantly higher implementation complexity.

The Core Conflict: Real-Time Performance vs. Agent Capabilities

Why Can't Text-Output Models Meet the Requirements?

There's a critical criterion here: latency. A truly professional voice agent should keep latency under 0.2 seconds, making the delay virtually imperceptible to humans.

If you use a text-output model (such as certain 4B or 24B parameter models), even if you convert the text to speech on the backend, the end-to-end latency will be noticeably perceived by users. This is the fundamental reason why so many voice products on the market feel sluggish.

How to Evaluate Whether a Model Is Suitable for Agent Tasks?

To determine whether a model can serve as an Agent's core reasoning engine, two benchmark scores matter most:

• Agent capability score: Measures the model's accuracy in intent recognition and tool calling
• Coding capability score: Measures the model's logical reasoning and code generation ability

According to shared findings, several top domestic reasoning models are now very close to international top-tier models on both metrics.

The Shortcomings of Omni-Modal Models

What you might not have noticed is that certain omni-modal large models — while supporting both voice input and voice output, making them seem ideal for voice interaction — score far below mainstream reasoning models on Agent benchmarks.

This creates a dilemma:

• Omni-modal models: Strong voice capabilities, weak Agent capabilities
• Top reasoning models: Strong Agent capabilities, no voice support

There's virtually no off-the-shelf solution that simultaneously delivers both real-time performance and strong reasoning ability.

Sandwich Architecture in Practice

Why Choose the Sandwich Approach

Facing the contradictions above, and drawing on recent related research papers, building an improved Sandwich solution is the most pragmatic path forward. The core idea is: put the emphasis on the Agent side.

The Agent's core reasoning model should still be a top-tier model with excellent Agent benchmark scores. These models simply don't support voice — so add STT in front and TTS behind, "sandwiching" the model in between.

The Key Challenge: Latency Optimization

The biggest pain point of the Sandwich Architecture is degraded real-time performance — users will feel noticeable delay. Whether the architecture can be successfully deployed in production hinges on whether the latency problem can be solved.

In live demonstrations, the project achieved approximately 0.3 seconds of response latency through meticulous optimization of both the front-end STT and back-end TTS. After the user finishes speaking, the system begins delivering its response almost immediately, and the real-world experience is remarkably smooth.

Real-World Pitfalls and Lessons Learned

Here are the three most critical pitfalls encountered in practice:

1. Barge-In Recognition During Playback

When the system is reading out a response and the user starts speaking, recognition accuracy drops significantly. A dedicated barge-in mechanism is needed to ensure users can interrupt the system's playback at any time.

2. Message Stacking Can't Be Oversimplified

The demo used a very simple list for message stacking, but in production environments, this approach will cause blocking issues. Context management must use a more rigorous queue or state machine design.

3. Tool Calling Stability

During the demo, the weather query tool call threw an error, highlighting that the robustness of the tool calling pipeline needs significant attention — including timeout handling, retry mechanisms, and fallback strategies.

Summary and Recommendations for Production Deployment

Deploying enterprise-grade voice agents is essentially about finding the right balance between real-time performance and Agent capabilities. For the vast majority of enterprise use cases, the Sandwich Architecture (STT + strong reasoning LLM + TTS) remains the most pragmatic and controllable choice.

Key considerations for production deployment:

• Choose a reasoning model with top Agent benchmark scores
• Focus engineering efforts on latency optimization, targeting sub-0.3-second response times
• Pay serious attention to barge-in mechanisms, context management, and tool calling robustness
• Implement proper monitoring and fallback strategies to ensure production stability

As real-time multimodal large models mature, native Speech-to-Speech or Hybrid architectures may eventually become the mainstream. But until then, solid engineering optimization of the Sandwich Architecture is the most reliable path to production deployment today.