Time To First Token (TTFT): The Key Performance Metric That Defines AI Product Experience

We've Been Measuring Latency Wrong

When evaluating the performance of an AI model or application, most people instinctively focus on a single number: how long a response takes from start to finish. It's the most intuitive metric — and it's almost entirely irrelevant to the interactive user experience.

A Reddit developer shared a misconception he'd held for a long time, one that reveals a blind spot common to many engineers: what actually determines how users feel is Time To First Token (TTFT) — the gap between pressing Enter and seeing the first character appear on screen.

This insight sounds simple, but it cuts to the heart of AI product experience design. Once tokens start streaming, your brain concludes that "the system is working" and begins reading the output. A response that starts within 300ms but takes 20 seconds to complete feels fast; one that takes 3 seconds to start but finishes in 12 seconds feels slow — even though the latter is objectively quicker.

You measure total duration. You experience the opening moment. The gap between these two things is the hidden root cause behind countless poor AI product experiences.

reddit source

Why TTFT Is the "Only Number That Matters"

This idea deserves a closer look, because it surfaces a chain of engineering consequences that are easy to overlook.

The Hidden Cost of Buffering

Buffering anywhere in the pipeline is far more expensive than it looks on a chart. Any intermediate layer that collects a complete response before passing it along turns an interaction that feels fast into one that feels slow — with virtually no impact on total duration.

In other words, if you're evaluating any component sitting between you and the model provider (an agent layer, gateway, or routing service), TTFT is the critical property to test — and it will never appear in any benchmark that measures completion time.

The Early Warning Sign of Degradation

TTFT is also an early indicator of provider performance degradation. Before a status page turns red, TTFT starts to drift: queuing increases, scheduling delays grow longer — but requests still complete successfully, so nothing gets logged as an "incident."

If you're only watching total duration and error rates, you'll completely miss a bad stretch — while your users' experience quietly deteriorates. This places a clear requirement on operational monitoring: make TTFT a core observability metric.

At a technical level, TTFT refers to the time elapsed from when a client sends a complete request to when the server returns the first token in a streaming response. It's essentially composed of two parts: network round-trip latency (the time for the request to reach the server) and server-side queuing + prefill time (the time for the model to process the input prompt, complete KV Cache computation, and generate the first output token). The complementary metric is TPOT (Time Per Output Token) — the average time to generate each subsequent token, which determines the "scrolling speed" of streaming output. Total latency is roughly equal to TTFT + TPOT × number of output tokens. For long outputs, TPOT contributes far more to total duration than TTFT — which is precisely why looking at total latency alone systematically misleads engineers. It conflates a tail cost that's nearly irrelevant to user perception with the upfront cost that defines the "first impression."

How Reasoning Models Upend the Entire Framework

Here's an important wrinkle: today's popular reasoning models fundamentally change this logic.

If a model needs to "think" before producing output, TTFT as classically defined can stretch to several seconds even when everything is working perfectly. The critical question shifts: whether that wait reads to the user as "stuck" or "working" depends almost entirely on whether the interface shows the user something during that wait.

The same delay, different experiences — decided by the UI. This explains why products like ChatGPT and Claude display "thinking" animations or stream their reasoning process. It's not decorative — it's a necessary countermeasure against the perception of high TTFT.

The core mechanism of reasoning models (such as OpenAI's o-series and DeepSeek-R1) is generating an internal Chain-of-Thought before producing a final answer. This chain of thought can span hundreds or even thousands of tokens, all completed internally before any user-visible output begins. From a TTFT perspective, this means that even when the model is running normally, users may wait 5–30 seconds before seeing the first meaningful output. Mainstream products take two general approaches: streaming the chain of thought (like Claude's "Extended Thinking" mode), letting users watch the reasoning in real time — transforming high TTFT into the perception of "thinking carefully"; or displaying placeholder animations to fill the visual void. Both strategies are fundamentally about using information design at the UI layer to reframe objectively high latency as a sense of "deep engagement" — a textbook application of perception engineering in AI products.

Averages Lie. p95 Tells the Truth.

The author makes a sharp critique: averages hide exactly what you care about.

TTFT distributions are highly skewed — a beautiful median paired with an ugly long tail makes a system feel unreliable; a slightly worse median with a tighter tail makes it feel stable. The reason is simple: people remember the bad interactions.

Therefore:

p95 is the honest number — it reflects the true state of the worst 5% of experiences
The mean is the flattering number — which is probably exactly why it gets published so widely

For any team that takes user experience seriously, this is a principle worth writing into your performance reporting standards: report percentiles, not averages.

p95 (the 95th percentile) is a statistical measure indicating that 95% of all measured requests have a latency below that value. In other words, p95 represents the upper bound of latency experienced by the "worst 5% of users." Other commonly used percentiles in latency analysis include p50 (median), p99, and p99.9. TTFT distributions typically exhibit a right-skewed long tail: the vast majority of requests cluster at lower latency values, but a small number — affected by queuing, GC pauses, network jitter, and similar factors — produce extremely high latency, pulling the mean significantly upward. Because the mean is highly sensitive to outliers, a single 10-second timeout can pull the average of a thousand 200ms requests up to 210ms, while p95 remains largely unaffected. Google's SRE practices and Amazon's internal standards both explicitly require high percentiles (p99 or p99.9) rather than means as the basis for SLA definitions — for exactly this reason.

An Unsolved Problem

The author candidly admits he runs a project (routera.one) that adds an extra "hop" to the path — and a hop is precisely the kind of thing that, if built carelessly, quietly destroys TTFT without affecting total duration. That's why he's especially sensitive to this metric: anything in the path has to prove it deserves to be there on this dimension.

He also honestly defines the limits of applicability: for batch processing or background tasks, none of the above applies. When no one is watching a screen, optimizing TTFT is wasted effort — throughput and cost are the right metrics. TTFT's dominance is specific to interactive scenarios where the opening moment defines everything.

Finally, he poses an open product question: how do you communicate "the provider is slow" to users without training them to ignore you?

Pop a warning every time p95 spikes, and within a week it becomes wallpaper nobody reads
Show nothing, and users blame your product for a bad afternoon that someone else caused

There's no clean answer — and it's a real dilemma every team building AI infrastructure will eventually face.

Final Thoughts

The value of this discussion is that it brings a systematically overlooked metric to the forefront. As large model capabilities increasingly converge, the perceived quality of interaction latency is becoming a key battleground for product differentiation.

Keep these principles in mind: measure TTFT rather than total duration; report in percentiles rather than averages; treat TTFT as an early sentinel for service degradation; and use UI design to compensate for the inherent latency of reasoning models. These seemingly small trade-offs ultimately determine whether users experience your product as "fast" or "slow."