Voice Agent Sandwich Architecture in Practice: Building a Real-Time AI Voice Agent in 2,000 Lines of Code

A recent tech talk on Bilibili fully broke down an AI voice agent development approach based on Voice Agent, implementing a real-time conversational agent project in under 2,000 lines of code. Instead of choosing the popular Speech-to-Speech end-to-end route, the solution returned to the classic "sandwich architecture" and made targeted optimizations for latency control.

Comparison of Three Mainstream Voice Agent Architectures

From an engineering implementation perspective, the underlying architecture of all voice agents can be reduced to three routes.

The sandwich architecture consists of STT (Speech-to-Text) + a large language reasoning model + TTS (Text-to-Speech) chained in series. The reasoning model is sandwiched between the two voice modules like a filling, with a clear structure. This is precisely the route adopted by the presenter's project.

The native real-time Speech-to-Speech architecture requires the model to simultaneously possess real-time multimodal input and output capabilities—input supports speech, images, text, and video, and output must also be multimodal including speech. Currently among mainstream large models, "almost none" meet this requirement.

The hybrid cascaded architecture attempts to combine the characteristics of the former two, integrating the sandwich structure with real-time reasoning.

STT, TTS, and end-to-end latency are the prerequisite concepts for understanding these three architectures. STT (Speech-to-Text) is responsible for transcribing user speech into text, with mainstream solutions including the Whisper series, FunASR, etc.; TTS (Text-to-Speech) is responsible for synthesizing the text generated by the model into speech, and in recent years solutions based on diffusion models and neural codecs (such as CosyVoice, Fish Speech) have achieved near-human naturalness. The so-called "end-to-end latency" refers to the time from when the user finishes their last word to when the agent begins playing its first segment of speech, rather than the total duration of the entire reply. It consists of four chained stages: STT recognition time, LLM first-token generation time, TTS first-packet synthesis time, and the network and queuing overhead between each stage. The reason the sandwich architecture faces high latency pressure is precisely because these three stages are additive—jitter in any single stage directly raises overall latency. Therefore, pipeline parallelism, streaming output, and first-packet priority strategies become the core of engineering optimization.

The Real-World Dilemma of Real-Time Multimodality

An easily overlooked technical detail: many multimodal large models on the market (such as the Zhipu series, Qwen series, GPT series) support multimodal input, but their output does not support multimodality. According to the presenter's analysis, the two most critical metrics for voice agents are real-time performance and latency, and a qualified product should have end-to-end latency within 0.2 seconds, imperceptible to humans.

When adopting the Speech-to-Speech architecture, users can clearly perceive the latency—because the model outputs text, which still requires an additional cascade conversion to become speech. This explains why even if you find a model supporting multimodal input and output, using it directly for a voice agent is still not ideal.

Input multimodality and output multimodality are two capability dimensions that are often confused. Input multimodality refers to the model's ability to receive prompts in various forms such as text, images, audio, and video; output multimodality requires the model to directly generate audio waveforms or image tokens, not just text. The vast majority of so-called "multimodal large models" only achieve the former—they first convert speech to text before understanding it, and still output responses in text form. For voice agents, this means an additional TTS module must be inserted into the pipeline, adding a conversion stage and naturally increasing latency. Models that truly natively support voice output (such as GPT-4o's real-time voice mode) adopt a method of directly modeling speech tokens, skipping the intermediate text state. However, such models are typically closed-source, difficult to deploy privately, and perform worse than pure text reasoning models on complex tasks like tool calling—this is precisely the dilemma in current engineering selection.

Agent Capability Is the True Core Competitiveness

Voice is merely the shell; the real difficulty lies in the agent's own capabilities: loop iteration, user intent recognition, tool calling, Skill invocation, context management. And current native real-time multimodal models generally score low on agent benchmarks.

The presenter particularly emphasized that when evaluating large models, two metrics deserve the most attention:

• Agent capability score: reflects the model's accuracy on tasks such as tool calling and multi-step reasoning
• Coding capability score: reflects the level of code generation and understanding

According to the presenter, the Agent and Coding scores of recent DeepSeek^Ad official versions, Kimi K3, and Zhipu's latest models on well-known foreign testing platforms are already very close to the levels of OpenAI GPT-5.5/5.6 series and Anthropic Opus 4.8 and other models. This means domestic developers now have more high-quality reasoning models to choose from.

Agent capability and Coding capability are singled out as evaluation metrics because they are highly correlated with a model's "execution power." Agent capability is typically measured through dimensions such as tool calling accuracy, multi-step task completion rate, and instruction-following degree, with typical benchmarks including BFCL, τ-bench, GAIA, etc.; Coding capability is measured through code generation and repair benchmarks (such as SWE-bench, LiveCodeBench). Together they reflect whether a model can "correctly call external interfaces, maintain multi-turn state, and generate executable results" in real production tasks. For voice agents, even if the voice pipeline is perfectly smooth, if the model cannot invoke tools or remember context, the interaction experience will still collapse. Therefore, when selecting models, these two scores should be prioritized over general Q&A or translation leaderboards—the latter's correlation with actual agent performance is far lower than intuition suggests.

Engineering Breakthrough for the Sandwich Architecture

The contradiction is clear: models with strong Agent capabilities have poor support for voice input and output; while fully multimodal voice models score very low on Agent benchmarks.

The presenter's solution is: add STT in front of a strong reasoning model and TTS behind it, forming a sandwich structure. The core reasoning model uses high-Agent-score models such as DeepSeek official version, Opus 4.8, GPT-5.5/5.6, or Zhipu's latest model, thereby ensuring agent capability while also supporting voice interaction.

The cost of this architecture is the loss of real-time performance—users can perceive the latency. The project's key breakthrough lies in solving this latency problem—this is precisely the technical highlight of this practical project.

Streaming processing and Voice Activity Detection (VAD) are two key engineering methods for reducing latency in the sandwich architecture. Streaming processing requires all three stages—STT, LLM, and TTS—to work incrementally: STT produces intermediate results while listening, LLM emits tokens while receiving text, and TTS synthesizes the first segment of audio and immediately delivers it without waiting for the entire sentence to complete. VAD (Voice Activity Detection) is used to determine whether the user has finished speaking, directly deciding "when to cut off recording and when to trigger reasoning," and its sensitivity setting is a trade-off between latency and false interruptions. Additionally, an interruption mechanism must be introduced: when the user interjects mid-conversation, the system must immediately stop the current TTS playback and clear the subsequent reasoning queue, otherwise it produces the terrible experience of "talking to itself." Only by combining these mechanisms can the originally additive multi-stage latency be compressed into a perceptible threshold approaching 0.2 seconds—this is also the most engineering-experience-intensive part of the project's approximately 2,000 lines of code.

Applicable Scenarios and Selection Recommendations

For developers hoping to quickly build AI voice agents, the advantages of the sandwich architecture are: module decoupling, with STT, LLM, and TTS independently replaceable; the agent capability ceiling determined by the chosen reasoning model; and simple engineering implementation with controllable code volume.

It should be noted that this route has high requirements for latency engineering optimization and is suitable for teams with some concurrency and streaming processing experience. If pursuing extreme low latency and not requiring high agent complexity, native real-time multimodal solutions can be evaluated; if complex tool calling and multi-step reasoning are needed, the sandwich architecture paired with strong reasoning models remains the more pragmatic choice currently.