Time to First Token

The latency from issuing a request to the model returning the first output token, a key metric for the responsiveness of interactive applications. In autoregressive generation the model must first complete a prefill forward computation over the entire input context, so longer contexts result in longer time to first token.