The Sandwich Architecture: The Best Deployment Strategy for Enterprise Voice Agents

Why Are Voice Agents So Hard to Build?

As large model capabilities mature, Voice Agents have become a hot direction for enterprise AI applications. But developers who've actually worked on these projects know that deploying voice agents is far more complex than expected—they need to not only "hear" and "speak," but also possess complete agent capabilities: iterative loops, user intent recognition, tool calling (skill invocation), context management, and more.

The "iterative loops" mentioned here refer to the Agent's approach to complex tasks—rather than giving a one-shot answer, it progressively approaches the goal through multiple rounds of "think-act-observe" cycles (similar to the ReAct framework). "Tool calling" (Tool Use / Function Calling) means the model can identify when to invoke external APIs or services to retrieve information or perform operations, such as querying databases, calling calendar services, or initiating payments. "Context management" involves maintaining session state across multi-turn conversations, remembering what users said previously, and performing information compression and retrieval in long conversations. The combination of these capabilities distinguishes voice agents from simple voice assistants—the latter only handles single-turn Q&A, while the former can complete multi-step complex tasks.

From an engineering perspective, all voice-enabled agents on the market can be categorized into three main architectural patterns. Understanding the differences between these three architectures is a prerequisite for designing a usable voice agent.

Three Mainstream Voice Agent Architectures

The first is the Sandwich Architecture (STT + Agent + TTS). As the name suggests, it "sandwiches" the large language model responsible for reasoning and decision-making (the Agent) between a Speech-to-Text (STT) layer and a Text-to-Speech (TTS) layer. This is the primary focus of this article.

The second is the Native Real-time Speech-to-Speech Architecture. It requires a multimodal large model capable of real-time voice interaction, where both input and output are speech with no intermediate text conversion. The core idea of native Speech-to-Speech models is to convert speech signals into discrete speech tokens via an audio encoder (e.g., audio tokenizer), incorporate them alongside text tokens within a unified sequence modeling framework, and have the model directly output speech tokens that are then decoded back into audio waveforms. Representative works include OpenAI's GPT-4o (real-time voice mode), Google's Gemini 2.0 Flash (native audio output), and others. The advantage of this architecture is that the model can directly learn paralinguistic information like prosody, emotion, and pauses in speech, while eliminating information loss and latency from intermediate text conversion. However, training such models requires massive amounts of speech-to-speech aligned data, and the model scales are typically enormous with high inference costs. In theory, this is the lowest-latency, best-experience solution.

The third is the Hybrid Architecture. It combines characteristics of both the sandwich and native real-time architectures, attempting to strike a balance between the two. The design philosophy is: use the native Speech-to-Speech path for simple, high-frequency interactions (like confirmations, brief answers) to achieve minimum latency; switch to the sandwich architecture for scenarios requiring complex reasoning or tool calling to ensure agent capabilities. This routing strategy requires a front-end intent classifier to determine the complexity of the current conversation turn and dynamically select the processing path. Additionally, hybrid architectures may employ a "speculative execution" strategy: while waiting for the strong reasoning model to return results, a lightweight model generates a quick placeholder response (e.g., "Let me look that up") to reduce perceived user latency.

Real-time Performance: The Make-or-Break Factor for Voice Agents

Why does the native Speech-to-Speech architecture sound optimal yet remain difficult to deploy? The core issue lies in latency and real-time performance.

A voice agent in the true sense should keep latency under 0.2 seconds—the threshold at which humans barely perceive any lag. This threshold originates from classic research in human-computer interaction. Jakob Nielsen noted in his usability heuristic principles that responses under 100 milliseconds feel instantaneous to users, under 200 milliseconds users won't perceive noticeable delay, and beyond 1 second, loading feedback is needed. In voice conversation scenarios, the average turn-taking gap in natural human dialogue is approximately 200-300 milliseconds, so for a voice agent to achieve a natural conversational experience, the end-to-end latency (from when the user finishes speaking to when the system begins responding audibly) must be kept within this range. Beyond 500 milliseconds, users will clearly feel "lag"; beyond 1 second, the conversational experience degrades severely.

But to achieve this, you must find a truly real-time multimodal voice large model.

There's a common misconception here. Many people say: "Isn't model X multimodal? It can handle speech!" But the distinction to make is that the multimodal large model we require must support speech not only as input but also as output. For voice agents, speech output is a hard requirement.

Taking some current visual multimodal models as an example, their inputs do support multimodal data, but their outputs don't support speech. If you force-fit such models, it means the model first outputs text, then converts text to speech—this extra conversion step significantly increases latency, making real-time performance impossible.

Unfortunately, truly mature and stable real-time multimodal voice large models are extremely rare today. This is the biggest obstacle to deploying the native real-time architecture.

Agent Capabilities: Another Unavoidable Evaluation Metric

Even if you manage to find a full-modality large model that supports both speech input and output, you'll encounter a second contradiction—its agent capabilities are often weak.

Key Metrics for Evaluating Model Agent Capabilities

Determining whether a model is "qualified" to serve as the core reasoning engine of an agent primarily involves two benchmark metrics:

• Agent capability score: Reflects the model's accuracy in intent recognition, tool calling, and multi-step reasoning;
• Coding capability score: Reflects the model's logic and code generation abilities.

Mainstream benchmarks for evaluating agent capabilities include: BFCL (Berkeley Function Calling Leaderboard) which evaluates function calling accuracy across dimensions like simple calls, multi-function selection, and parameter filling; AgentBench which tests models' ability to execute multi-step tasks in real environments covering operating systems, databases, web browsing, and other scenarios; and SWE-Bench which tests models' ability to solve real GitHub Issues, reflecting end-to-end code comprehension and modification skills. On the coding side, HumanEval, MBPP, LiveCodeBench, and others are commonly used evaluation sets. These two types of capabilities are highly correlated—tool calling is essentially structured code generation, and complex reasoning tasks also frequently require programming-style logical thinking.

Recent domestic (Chinese) models have shown impressive performance on both metrics. Multiple domestic models have achieved scores very close to top international levels on mainstream testing platforms, demonstrating considerable competitiveness as core reasoning models for agents.

The Agent Weakness of Full-Modality Models

The problem is that those full-modality large models that truly support speech input and output, while able to "hear and speak," score far below mainstream reasoning models on agent benchmark datasets.

This creates an awkward contradiction:

• Models with strong reasoning → Strong agent capabilities, but poor speech I/O support;
• Full-modality speech models → Good speech support, but weak agent capabilities.

Satisfying both requirements simultaneously has virtually no perfect solution today.

The Sandwich Architecture: A Pragmatic Engineering Compromise

Facing the above contradiction, the sandwich architecture offers a pragmatic engineering path. Its core philosophy is: Use the strongest capabilities where they matter most, and fill in the rest with mature components.

Deconstructing the Sandwich Architecture

The core logic of the sandwich architecture is:

1. Middle Layer (Agent + Reasoning Model): Use the model with the highest agent benchmark scores as the core. This layer handles intent recognition, tool calling, context management—the parts that truly embody "intelligence."
2. Front Layer (STT - Speech-to-Text): Connect a speech-to-text model before the reasoning model to solve the speech input problem.
3. Back Layer (TTS - Text-to-Speech): Connect a text-to-speech model after the reasoning model to solve the speech output problem.

STT technology, through rapid development in recent years, has achieved remarkably high accuracy. Open-source models represented by OpenAI's Whisper have reduced Word Error Rate (WER) to below 5% in multilingual scenarios. Commercial solutions like Google Speech-to-Text and Azure Speech have been extensively optimized for enterprise scenarios, supporting domain adaptation and hot-word customization. The TTS field has likewise made breakthrough advances—neural network-based TTS models (such as VITS, Bark, ChatTTS, Fish Speech, etc.) can now generate natural speech approaching human quality, supporting emotion control, speed adjustment, and multi-speaker switching. More critically, the maturation of streaming TTS technology allows models to synthesize speech while generating text—as soon as the reasoning model outputs a sentence fragment, TTS immediately begins synthesis, dramatically reducing the user-perceived end-to-end latency.

By "sandwiching" a strong reasoning model between STT and TTS, you ensure both the agent's core capabilities and complete speech I/O coverage. This is the origin of the sandwich architecture's name.

Advantages and Costs of the Sandwich Architecture

The sandwich architecture's greatest advantage is flexibility and capability ceiling: you can freely choose the strongest reasoning model available without being constrained by speech capabilities; STT and TTS can each be selected as optimal standalone components. When a better model emerges for any single layer, it can be independently replaced and upgraded without affecting other layers. This modular design also facilitates team division of labor—speech engineers handle STT/TTS optimization while AI engineers iterate on Agent logic.

The cost is latency. Compared to native Speech-to-Speech, there are two additional conversion steps, resulting in theoretically higher latency. Specifically, the STT step typically adds 100-300 milliseconds (depending on whether streaming recognition is used), the TTS step adds 100-200 milliseconds (significantly compressible with streaming synthesis), and combined with the reasoning model's own Time-to-First-Token (TTFT), the total end-to-end latency typically falls between 500 milliseconds and 1.5 seconds. But given the current reality of lacking mature real-time full-modality models, trading controllable latency for powerful agent capabilities is a more reasonable tradeoff for the vast majority of enterprise scenarios.

Architecture Selection Is a Game of Tradeoffs

Voice agent architecture selection is fundamentally a tradeoff between real-time performance and agent capabilities. The native real-time architecture offers the best experience but is constrained by an immature model ecosystem; the sandwich architecture achieves a balance between capability ceiling and engineering feasibility, making it the mainstream choice for current enterprise deployments.

For teams looking to quickly build usable voice agents, the most prudent path is to first run through the business loop with the sandwich architecture, then gradually evolve toward native architecture as real-time multimodal models mature. There are no absolutely superior or inferior technical architectures—only whether they match your current resources and needs.

Key Takeaways