Unverified50% confidenceFactExact time
H100的内存带宽约3.35TB/s，GPT-4规模模型单次生成一个Token需读取约1TB参数数据，理论上每秒最多生成约3个Token
1
Sources
50%
Confidence
Long-term
Relevance
7/19/2026
First Seen
Sources
Related Claims
UnverifiedThe KV cache for a 128K context window of a GPT-4-level model can occupy tens of gigabytes of VRAM79% similarUnverifiedUnder a 45nm process, a single 32-bit floating-point multiplication consumes approximately 3.7pJ of energy, reading 32-bit data from off-chip DRAM consumes approximately 640pJ, and on-chip SRAM access requires approximately 5pJ.75% similarUnverifiedIf the quantized model weights are 150 GB, reading the weights once at a bandwidth of 273 GB/s takes approximately 0.55 seconds, giving a theoretical single-token generation speed of approximately 1.8 tokens/s (excluding computation latency and cross-node communication overhead).73% similarUnverifiedTaking the LLaMA-2 70B model as an example, a sequence of 2048 tokens requires approximately 4 GB of VRAM for the KV cache alone.71% similarUnverifiedA single inference at the million-token level requires tens to hundreds of gigabytes of high-end GPU memory (A100/H100 class).70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/563265API
curl https://kongchang.com/api/v1/knowledge/claims/563265MCP
get_claim(id=563265)