Why Voice AI Is Harder Than You Think: PolyAI's Shawn Wen on the Science of Conversational Timing

When ChatGPT burst onto the scene, many assumed voice AI was a solved problem — just chain together ASR, an LLM, and TTS and you're done. But PolyAI's Shawn Wen offers a counterintuitive take in this interview: voice conversation is far harder than most people imagine, and the real difficulty isn't reasoning — it's adaptation.

Voice Is the First Step Into the Physical World

Wen puts forward a compelling framework: virtually all AI innovation today is concentrated in the text and digital world. Solving a PhD-level physics problem is no longer a stretch, because there's a massive corpus of documents and papers for models to learn from. But voice introduces an entirely new dimension — time.

"Time is a critically important factor, and we haven't yet trained models that truly understand it," he says. He draws an analogy to robotics: for humans, collecting data in the physical world is natural — we have vision, we have limbs, we can reach out and touch things. Computers have none of that. All they know is how to use a computer and how to process text.

He invokes the famous Wozniak coffee cup challenge (get a robot to make me a cup of coffee) and a friend's robotics startup whose first goal was "getting the robot to pour me a beer" — both illustrations of how hard physical-world tasks really are. Voice is hard for the same reason: "machines have never truly observed the vast volume of real conversations between humans." This echoes Rich Sutton's "Bitter Lesson": the combination of failure modes in real conversation is exponential, handwritten rules can never cover them all, and only end-to-end trained neural networks can achieve the necessary coverage.

Rich Sutton's "Bitter Lesson" (2019) is a landmark reflection in machine learning. Sutton reviewed 70 years of AI research and concluded that whenever researchers tried to encode human prior knowledge (hand-crafted rules) into systems, the approach worked short-term but was consistently overtaken in the long run by brute-force computation and general learning methods — in chess, Go, speech recognition, and computer vision alike. This lesson hits especially hard for voice conversation: the combination of failure modes in real dialogue (user accents, background noise, interrupted speech, semantic ambiguity) is exponential. Any engineering effort trying to hand-write rules covering "all the edge cases" is doomed to fail. Sutton's conclusion: search and learning are two general-purpose methods, and given enough compute, they will outperform any accumulation of domain-expert knowledge. This is the theoretical foundation behind PolyAI's bet on end-to-end training over modular rule-based systems.

Enterprises Want a Contradiction

When it comes to what enterprise customers actually want, Wen identifies a core paradox:

"They want the agent to be as capable as a human, but as controllable as a machine. These are genuinely contradictory demands."

On top of that, enterprises require branded voices — agents that can correctly pronounce brand names and express the company's identity the way the brand expects. This tension between "human-level capability" and "machine-level controllability" is the fundamental challenge of deploying AI in enterprise settings. Users expect the AI to be "at least as good as the best person on the team," which makes the adoption curve steep. Only when companies find clear productivity gains do they gradually come around and discover "the agent is actually better than they expected."

Wen draws a sharp line between consumer and enterprise: "In consumer settings, voice agents are already buildable today. But enterprise has additional governance and compliance requirements that have to be met."

Why Build an End-to-End Model

About a year ago, PolyAI made a call: cascaded architectures would eventually become obsolete, and end-to-end models were the future. They stopped investing in their own standalone ASR model as a result.

The latest speech-to-speech models on the market produce stunning demos, but fall apart in all sorts of ways the moment they're deployed in a contact center. Wen gives a vivid example: an elderly woman calling a bot for the first time, repeatedly confused and hesitant, while the agent keeps interrupting and talking over her — a frustrating experience that is simply fatal for adoption.

"The frontier labs aren't optimizing for what we care about — which is real turn-taking in B2B conversational settings."

A traditional cascaded system is made up of three parts: an ASR module, an LLM, and a separate turn-detection module. These three components don't play well together and require constant manual tuning. By "collapsing" them into a single end-to-end model, the LLM can directly perceive audio — whether that's a slower speaking pace, background noise, or crosstalk — and the model learns to adapt from data.

Cascaded architecture is the dominant implementation path for voice AI products today, typically consisting of three independent modules in series: ASR converts audio to text, an LLM understands the text and generates a response, and TTS reads the response aloud. Each module has its own latency, error rate, and tuning surface, and the "seams" between modules are where failures cluster. For example, if ASR mishears "I want to cancel my order" as "I want to cancels my order," the LLM generates a response on corrupted input, and TTS reads it back — errors amplify at every stage. Turn-taking is especially painful in cascaded architectures, typically handled by a Voice Activity Detection (VAD) module that only senses silence duration and cannot understand semantic signals like "I paused because I'm thinking, not because I'm done." End-to-end models collapse all of these modules into a single unified neural network, allowing the model to jointly reason about turn state from both audio signals and semantic context — solving the inter-module coordination problem at its root.

The Clever IO Contract: Audio-Native Model Architecture

Wen walks through their high-level architecture in detail. At its core is an audio-native LLM — but one that outputs text rather than speech. The reason is practical: text is much easier to apply guardrails to, while speech is too difficult to control — a direct reflection of enterprise demands for controllability.

The model's workflow is elegantly designed:

• The first token predicts the turn signal: is the user just starting to speak, still speaking, or done? This token is extremely cheap to generate.
• Audio preprocessing is front-loaded: by the time the first token is predicted, the audio has already been encoded into the embedding space, so the model doesn't need to re-encode the full utterance from scratch.
• Streaming output of responses or tool calls: returned in a streaming fashion, like ChatGPT.
• Generating citations: marking which facts in the knowledge base the answer is grounded in, for enterprise auditing.
• Transcription output comes last: the order is deliberately inverted, because transcription is only needed for auditing and debugging — it doesn't need to complete before the system can respond to the user.

Wen repeatedly emphasizes: "Our innovation isn't in the model itself — it's in the IO contract." That is, how to prepare data and how to design the input/output mechanism. Since open-source models update monthly or even weekly, PolyAI maintains its own data moat and training scripts — the moment a new model drops, they can train on it and pick the best one. "The underlying model doesn't actually matter that much."

Guardrails in AI engineering refer to a set of mechanisms that constrain model outputs to prevent harmful, inaccurate, or policy-violating content. Typical measures include keyword filtering, output classifiers (detecting hate speech or PII leakage), factual consistency checks (against a knowledge base), and format enforcement. Applying guardrails to speech output is far harder than to text — text is a discrete, parseable character sequence, while a speech waveform is a continuous signal, and modifying a single word means re-synthesizing an entire audio segment at significant latency cost. PolyAI's hybrid "audio in, text out" architecture is essentially a way to retain the information richness of audio at the perception layer (speaking pace, emotion, pauses) while retreating to the controllability of text at the generation layer, with downstream TTS synthesizing the final speech — striking an engineering balance between enterprise compliance requirements and model capability. The citation mechanism further strengthens auditability, with each response tagged with its "source," allowing enterprise legal and compliance teams to trace the agent's reasoning.

Data, Noise, and Latency Engineering

Voice data is far harder to collect than text data. PolyAI's advantage is that its platform already runs inside enterprise contact centers — banks, utilities, logistics, restaurants, hotels — naturally generating large volumes of real conversational data. By redacting all PII and generating synthetic PII data in its place, the model learns the mechanics of conversation rather than real user data.

For noise handling, they took a counterintuitive approach: deliberately injecting various types of noise. They even used generative models to overlay background noise on training data (for example, adding a crying baby in the background while the bot reads out a name). Interestingly, the noise-reduction and audio-cleaning pipelines from the previous generation of technology actually made the new model perform worse — "because the environment was too clean, something it had never seen before."

On latency engineering, Wen notes that historically the biggest bottleneck in cascaded systems was the crude parameter for "deciding when the user has finished speaking": tuned too aggressively and the system keeps interrupting; tuned too conservatively and users run out of patience. Because the end-to-end model can simultaneously sense both speaking pace and semantics, "if I pause mid-sentence and the meaning isn't complete, the agent knows to keep waiting."

They're also exploring adaptive budget reasoning: training the model to judge for itself "what's the probability I can give a good answer if I stop thinking right now" — completing its reasoning within a latency budget, like a countdown race. Combined with product-layer techniques such as filler words, typing sounds, and hold music, the goal is to maintain user trust — because "users have been burned by automated systems too many times, and three to five seconds of silence on a phone call will trigger panic."

Adaptive Budget Reasoning (also called "auto reasoning" in this context) is an emerging research direction in the reasoning model space. The core idea is to let the model dynamically decide "how much computation (i.e., reasoning steps) is worth spending on this particular problem." Traditional reasoning models (like OpenAI o1/o3) generate a fixed-length chain-of-thought before producing an answer, which is infeasible in latency-sensitive settings — five seconds of waiting on a phone call is already the limit. Adaptive budget reasoning lets the model evaluate the "marginal return of continuing to think" after each reasoning step, stopping early and outputting an answer once confidence exceeds a threshold. This mirrors how human experts make decisions: simple questions get immediate answers, complex ones get deliberation. Combined with filler words and typing sound effects, the goal is to maintain perceived fluency for the user while the model "thinks," masking necessary compute latency and preventing the trust collapse that silence causes.

The Voice Personality Paradox and the Benchmarking Problem

On whether voice should have personality, Wen's observations are pragmatic: most enterprises lean conservative, reluctant to attach too much persona to an agent — their goal is "just solve the problem first." But generic voices actually perform worse — consumers immediately recognize them as robotic and lose trust.

The most successful voices often carry regional characteristics: UK contact centers favor a Newcastle accent, because users hear it and assume they're talking to a real person. One amusing finding: early English voices weren't natural enough, and the most natural-sounding one turned out to be "English with a slight French accent" — it was precisely that imperfection that made the voice feel real.

Benchmarking is another major challenge. Wen is critical of existing public benchmarks for being unable to measure voice agents: they're either modular (testing only ASR or the LLM in isolation), or they rely heavily on synthetic data because of voice privacy concerns. More critically, "a voice agent benchmark that ignores latency isn't a real benchmark — users can't wait 60 seconds on a phone call." PolyAI has built internal benchmarks based on real conversational data, comprehensively evaluating response time, comprehension accuracy, tool call quality, and citation quality — and plans to open-source them within the next month or two to advance the industry.

Cognitive Debt and the Next Decade of Voice AI

The deepest part of the conversation concerns "cognitive debt": the bottleneck has shifted from producing content to verifying and auditing content. Wen admits his own team struggles with it — people have started sending each other long AI-generated texts (AI slop), forcing the recipient to use AI to summarize it in turn. "Human communication now requires AI in the middle to filter the useful information for us — it's a bit like going in circles."

He offers a sharp definition of slop — "generation without the ability to back it up." Whether something counts as slop is inherently subjective: under his own context engineering it doesn't qualify, but in a colleague's eyes who lacks the background, it might. Enterprise customers are still uncomfortable with auto-merged PRs and automated code reviews, insisting on two-week review cycles and full auditability. PolyAI has therefore built extensive visualization components into its platform, letting enterprises trace every step of an agent's workflow, tool set boundaries, and citation chains.

Looking ahead ten years, Wen's predictions are clear:

• Turn-taking must be end-to-end, collapsed into the entire pipeline — this is the hardest and most frustrating part of the engineering;
• Voice modalities will diverge: consumer and entertainment (Alexa- and Google Assistant-style) will prioritize delight, while enterprise customer service will prioritize professionalism, compliance, and auditability;
• A second wave of IoT hardware: voice is the natural interaction modality for IoT devices.

He also honestly identifies a deep unsolved problem: when trying to get an agent to absorb a company's cultural values, it often "hyperfocuses on one specific point," because the reasons behind those values were never written into the context. Reaching the ideal state where the technology is invisible, where you can teach the system like you'd teach a child and have it naturally integrate into the organization, will require enormous work on shared context and shared organizational identity.

As Wen puts it, the true intelligence in voice AI "isn't about reasoning — it's about adaptation." That may be the most important key to understanding the next generation of conversational AI.