Unverified50% confidenceBenchmarkExact time
In the comparison of independent execution for a burst of 8 requests (each with a 532-token prompt and 128-token output), the single-flight plus release-on-request-completion strategy recovered 528 tokens per follower, reduced average TTFT by 47.5%, and reduced the slowest request completion time by 25.8%; the single-flight plus release-after-prefill-publish strategy likewise recovered 528 tokens, reduced average TTFT by 82.9%, and reduced the slowest request completion time by 32.4%.
1
Sources
50%
Confidence
Medium-term (~90 days)
Relevance
10/16/2026
First Seen
Valid until: 1/14/2027
Sources
Related Entities
Related Claims
UnverifiedAccording to an analysis based on real API traffic, out of about 411,000 requests over 7 days, 99.3% of processed tokens were input tokens; the median request had 3,782 input tokens and returned only 147 tokens.69% similarUnverifiedThe prompt caching mechanism allows already-processed input prefixes to be cached server-side, and if the next request starts with the same prefix, it is read at a price far below normal input; common cache TTLs are 5 minutes and 1 hour, and writing to the cache costs extra.67% similarUnverifiedPrompt Caching is triggered when the prompt prefix is completely identical and exceeds a threshold of approximately 1,024 tokens, and cached tokens are typically billed at 25% or even less of the original price.66% similarUnverifiedA calculation example: asking 20 questions about a 50,000-token document requires about 1 million input tokens without prompt caching, and about 157,000 with caching, reducing input cost by about 84%, with faster time to first token as well.65% similarUnverifiedPrompt caching means the identical prefix shared by multiple requests is fully billed and processed only on the first request, and subsequent requests hitting the cache can lower the input token price and reduce latency; but caching cannot shorten the total context length, and long histories still occupy the context window and raise latency.65% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1048422API
curl https://kongchang.com/api/v1/knowledge/claims/1048422MCP
get_claim(id=1048422)