Latency Budget
The practice of breaking down a user experience goal into measurable time slices for each stage, ensuring that the latency of every step stays within an acceptable range so that overall response time is maintained.
Timeline (last 90 days)
For a chatbot, 40 seconds is slow, but for a letter one has been procrastinating on because one does not want to read it, it is fast enough; the right question is not "is it fast" but "is it fast enough for this task".
Teams concerned about latency or cost budgets need a separate test harness, because a pass-rate floor says nothing about tail behavior.
In the case study in the article, after deducting all required components, only 50 milliseconds of latency budget remained for the guardrail layer.
All Facts (4)
For a chatbot, 40 seconds is slow, but for a letter one has been procrastinating on because one does not want to read it, it is fast enough; the right question is not "is it fast" but "is it fast enough for this task".
50%UnverifiedTeams concerned about latency or cost budgets need a separate test harness, because a pass-rate floor says nothing about tail behavior.
50%UnverifiedIn the case study in the article, after deducting all required components, only 50 milliseconds of latency budget remained for the guardrail layer.
50%Unverified实时语音对话中的检索增强生成通常要求端到端延迟控制在500毫秒以内以维持自然对话节奏
50%