Unverified50% confidenceBenchmarkExact time
Llama 2 13B on an H100 at batch size 1 has a latency of approximately 8 milliseconds per token and a throughput of approximately 124 tokens/second, while at batch size 256 it requires approximately 240 GB of memory, which does not fit on a single H100.
1
Sources
50%
Confidence
Long-term
Relevance
9/23/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThe M5 Max's token generation speed is approximately in the range of 10–30 tokens/s (depending on model size), while the NVIDIA H100's HBM3 memory bandwidth reaches up to 3.35 TB/s, enabling inference speeds tens of times faster than local devices.70% similarUnverifiedLlama 3.1 7B量化后仅需约4-5GB显存即可运行67% similarUnverifiedThe A100's shared memory capacity per SM is 164KB (configurable), and the H100 increases this to 228KB.67% similarUnverified在配备96GB显存的M2 Max上以1-bit精度运行时，简单任务生成速度稳定在约28-30 tokens/秒66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/942436API
curl https://kongchang.com/api/v1/knowledge/claims/942436MCP
get_claim(id=942436)