TTS
Text-to-Speech (TTS) is a speech synthesis technology that automatically converts written text into spoken audio output. Core capabilities include linguistic analysis and text processing, generation of natural speech waveforms via speech synthesis engines, and support for multiple languages and voice types. TTS is broadly applied in accessibility assistance, intelligent assistants, navigation systems, and educational software, and constitutes a foundational technology for human-computer interaction.
Core Facts
GPT-4o's end-to-end voice architecture incorporates audio as discrete tokens into a unified Transformer processing framework, handling the full listen-think-speak pipeline within a single model.
There are approximately 7,000 languages in the world, and the vast majority are low-resource languages.
The new generation of voice interaction has shifted to an end-to-end multimodal model architecture that understands and generates directly at the speech signal level, compressing latency and preserving paralinguistic information.
Timeline (last 90 days)
A response delay exceeding 2 seconds significantly increases the hang-up rate.
In the third project, 40% of people abandoned on the "processing" page because the wait was too long; after switching to a faster and cheaper TTS model, the completion rate jumped from 60% to 82%.
The third project (converting technical blog posts into audio summaries) took the "path of least pain": paste a URL and get a 3-minute audio file, with no account, history, or download feature; text was extracted with readability-lxml, audio with a TTS API; it was delivered in one weekend, with $12.80 in API costs, 14 hours invested, and 31 users.
Emotion control in traditional TTS tools relies mainly on hard switching of tone tags, and changing emotion within a single sentence requires splitting it into multiple segments to generate and then splice together.
When the TTS engine runs alone, generating 1 second of audio takes about 0.6 seconds (RTF about 0.6), but after sharing the GPU with the LLM, first-sentence synthesis time balloons to about 1.5 times its own length, whereas when the LLM is idle it takes only about 1.0 times.
When running LLM and TTS simultaneously on a consumer-grade GPU, parallel scheduling causes the TTS's first-sentence synthesis to fall severely behind, thereby exhausting the playback buffer and causing stuttering in voice playback.
On an RTX 4060 Laptop (8GB VRAM), having the LLM actively pause and wait for TTS to synthesize sentence by sentence can eliminate audio playback stutter, at the cost of greatly extending the LLM's overall generation time (in testing, from about 12 seconds to about 54 seconds).
In parallel mode, the first audio arrived only 2.3 to 2.5 seconds after the request (rather than 1.0 to 1.2 seconds), and in the three runs all stutters were concentrated in the first 13 seconds (while the LLM was still writing or had just finished), with no stutters afterward.
In controlled tests that changed only the scheduling (Japanese prompts, simulated 0.5-second buffer, identical generated text, 6 runs each), all 6 runs stuttered in parallel mode, while 0 stuttered in alternating mode.
In the controlled tests with step-by-step improvements, with the configuration of "both run on the GPU alternately", the mean first audio was 2.3 seconds (range 1.9-2.7), and the mean number of stutters per answer was 0.09 (range 0-1), better than "both run on the GPU simultaneously" at 3.9 seconds (3.8-4.1) and 2.84 times (9-17 times).
40 more timeline events
All Facts (20)
GPT-4o's end-to-end voice architecture incorporates audio as discrete tokens into a unified Transformer processing framework, handling the full listen-think-speak pipeline within a single model.
90%VerifiedThere are approximately 7,000 languages in the world, and the vast majority are low-resource languages.
90%VerifiedThe new generation of voice interaction has shifted to an end-to-end multimodal model architecture that understands and generates directly at the speech signal level, compressing latency and preserving paralinguistic information.
80%VerifiedTraditional voice AI systems adopt a cascaded architecture that splits voice processing into independent modules such as ASR, NLU, dialogue management, and TTS, with information loss and latency accumulation between each module.
80%VerifiedThe three-stage architecture of traditional voice interaction (ASR+LLM+TTS) typically has a total latency of 1–3 seconds.
80%VerifiedModern voice cloning solutions typically use a speaker encoder to map a voice into a high-dimensional embedding vector, which is then injected into the acoustic model during the TTS decoding stage to control the output timbre.
80%VerifiedThe training data of most speech models is primarily monolingual, and mixed-language corpora are relatively scarce.
75%VerifiedTime to First Audio (TTFA) refers to the time elapsed from text input to the output of the first frame of playable audio, and is a core metric for measuring the response speed of streaming TTS systems.
75%VerifiedA complete voice AI interaction involves three serial processing stages: STT, LLM inference, and TTS.
75%VerifiedTraditional voice dialogue systems consist of three independent modules connected in series—ASR (Automatic Speech Recognition), LLM (Large Language Model), and TTS (Text-to-Speech)—each trained separately and executed sequentially.
75%Verified传统语音AI普遍采用级联式架构，先用ASR将语音转文字，再交给语言模型处理，最后用TTS合成输出
75%VerifiedTraditional speech-to-speech translation uses a three-module cascaded architecture of ASR → MT → TTS, which suffers from error accumulation and compounded latency.
75%VerifiedTraditional AI voice interaction systems use a pipeline architecture of four separate modules—ASR, NLU, NLG, and TTS—connected in series, with a combined total latency typically between 1 and 3 seconds.
75%VerifiedEnd-to-end native architectures directly process audio input and output audio without relying on intermediate conversion steps, preserving richer acoustic features.
70%VerifiedAI real-time voice translation systems simulate the simultaneous interpretation process through a pipeline collaboration of three modules: Automatic Speech Recognition (ASR), Neural Machine Translation (NMT), and Text-to-Speech (TTS).
70%VerifiedODS integrates voice interaction capabilities, supporting speech-to-text (STT) and text-to-speech (TTS)
65%VerifiedTraditional cascaded architectures generate video before adding audio, which leads to problems such as audio-video desynchronization, lip-sync mismatches, and ambient sound being out of step with on-screen actions.
65%VerifiedAchieving an interruptible real-time conversational experience involves a technical pipeline covering voice activity detection (VAD), automatic speech recognition (ASR), intent understanding and context management, retrieval and generation, TTS synthesis, and seamless continuity.
65%Verified800毫秒是语音代理端到端延迟的业界公认阈值，人类面对面对话回应延迟通常在200-500毫秒之间
65%VerifiedThe standard pipeline for AI short film production is to first create asset images, then the shot breakdown, and finally the audio
65%