End-to-End Latency
A core performance metric for voice agents, referring to the time between when the user finishes speaking their last word and when the agent begins playing its first segment of speech. It consists of four cascaded components: STT, the LLM's first token, the TTS first packet, and network overhead.
Timeline (last 90 days)
Time to first byte (TTFB) refers to the time from sending text to receiving the first audio byte; end-to-end latency includes the entire time consumed by audio capture, ASR transcription, intent processing, TTS synthesis and playback buffering.
Assume that within one round there are N mutually independent tool calls, each taking time t₁…t_N, and the total time for serial execution is approximately Σtᵢ.
When executed concurrently (concurrency ≥ N), the total time for N independent tool calls is approximately max(tᵢ).
When a single tool call itself is the bottleneck (max(tᵢ) is large), continuing to increase concurrency yields almost no improvement in total latency and only increases resource pressure.
Appointment conversation scenarios expect the end-to-end latency from when the user finishes speaking to when the agent begins responding to be controlled within hundreds of milliseconds.
All Facts (5)
Time to first byte (TTFB) refers to the time from sending text to receiving the first audio byte; end-to-end latency includes the entire time consumed by audio capture, ASR transcription, intent processing, TTS synthesis and playback buffering.
50%UnverifiedAssume that within one round there are N mutually independent tool calls, each taking time t₁…t_N, and the total time for serial execution is approximately Σtᵢ.
50%UnverifiedWhen executed concurrently (concurrency ≥ N), the total time for N independent tool calls is approximately max(tᵢ).
50%UnverifiedWhen a single tool call itself is the bottleneck (max(tᵢ) is large), continuing to increase concurrency yields almost no improvement in total latency and only increases resource pressure.
50%UnverifiedAppointment conversation scenarios expect the end-to-end latency from when the user finishes speaking to when the agent begins responding to be controlled within hundreds of milliseconds.
50%