Unverified50% confidenceFactExact time
At small batch sizes the bottleneck is memory bandwidth, while at large batch sizes the rising compute density makes compute the bottleneck, which explains the performance crossover between the 3090 and the L40S.
1
Sources
50%
Confidence
Long-term
Relevance
10/18/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe speedup ratio of speculative decoding is less ideal than for small and medium models when bandwidth has become a hard bottleneck or when the model exceeds 200B.76% similarUnverifiedWhen an operator's arithmetic intensity is below the hardware's ridge point (peak compute divided by peak bandwidth), it is bandwidth-bound; otherwise it is compute-bound.75% similarUnverified自回归模型随序列长度增加，KV Cache占用线性增长，显存带宽成为制约速度的硬件瓶颈75% similarUnverifiedIn ultra-long context inference scenarios, the performance bottleneck is often not compute but the KV cache itself.73% similarUnverifiedBrowser-side LLM inference faces limitations including a performance ceiling, memory constraints, and initial loading overhead, making inference speeds difficult to match native GPU acceleration73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1061230API
curl https://kongchang.com/api/v1/knowledge/claims/1061230MCP
get_claim(id=1061230)