H100
Core Facts
The NVIDIA A100's HBM2e capacity is 80GB, the H100 has 80GB HBM3, and the H200 has 141GB.
The HBM3 memory used by NVIDIA's H100 has a bandwidth of up to 3.35 TB/s.
xAI built a supercomputing cluster called Colossus, reportedly equipped with more than 100,000 NVIDIA H100 GPUs.
Timeline (last 90 days)
In GPU Index's price data snapshot, the lowest price for H100 is $1.37/hr (Vast.ai, spot instance), for H200 is $2.62/hr (CoreWeave, spot), for A100 is $1.15/hr (Paperspace, three-year contract), for L40S is $0.79/hr (RunPod, community cloud), and for B200 is $4.26/hr (CoreWeave, spot).
An NPU equipped with VKORE reaches 170% of a single H100's per-device pretraining throughput.
The figure "pretraining throughput reaching 170% of a single H100" is jointly affected by batch size, sequence length, model scale, precision format, and interconnect bandwidth, and is not a general conclusion; the ratio may change with a different model or hyperparameters.
The H100 uses the Hopper architecture with roughly 3.35 TB/s of HBM3 bandwidth, representing a generational performance gap compared with the A100
Under a single-request serial load, each request decodes at 648–649 tokens/s, and the whole server produces 589–593 tokens/s.
On the same H100, a scale-to-zero cold wake-up took about 75 of 77.3 billed seconds before the server was capable of responding.
A developer deployed a 35B-parameter model on an H100 GPU with 80GB of VRAM, with a hardware cost of about 5 million yen (about 35,000 USD).
A dense 35B model requires more than about 7 billion multiply-add operations per token, and latency on a single card may be as high as hundreds of milliseconds or even seconds.
When running a 35B-parameter model on an 80GB H100 for conversation, each generated token actually uses only about 3B equivalent of weights.
NVIDIA 有意在 RTX 4090 上取消了 NVLink，业界普遍认为这与数据中心 A100/H100 的产品分层策略有关。
40 more timeline events
All Facts (20)
The NVIDIA A100's HBM2e capacity is 80GB, the H100 has 80GB HBM3, and the H200 has 141GB.
90%VerifiedThe HBM3 memory used by NVIDIA's H100 has a bandwidth of up to 3.35 TB/s.
90%VerifiedxAI built a supercomputing cluster called Colossus, reportedly equipped with more than 100,000 NVIDIA H100 GPUs.
90%VerifiedThe pre-training process of large language models requires thousands of high-end GPUs running in parallel for months, with costs typically ranging from tens of millions to hundreds of millions of dollars.
90%Verified训练一次GPT-4级别的模型需要数万块A100/H100 GPU协同工作数月
90%VerifiedThe retail price of a single NVIDIA H100/H200 GPU is as high as $30,000–$40,000.
90%VerifiedNVIDIA's H100/H200 series GPUs hold more than 80% of the AI training and inference market.
90%Verified美国自2022年起对中国实施先进半导体出口管制，限制高端AI训练芯片如NVIDIA A100/H100及其后续产品的出口
85%VerifiedCerebras' latest-generation WSE-3 chip has an area of approximately 46,225 square millimeters, integrates 4 trillion transistors and 900,000 AI-optimized cores, and is more than 50 times the area of NVIDIA's H100 GPU.
80%VerifiedNVIDIA's MIG technology enables hardware-level resource isolation on high-end GPUs such as the A100/H100.
80%VerifiedCore innovations of the Blackwell architecture include the second-generation Transformer Engine (supporting FP4 precision), the NVLink Switch system (supporting interconnection of 576 GPUs), 20 petaFLOPS of FP4 compute, and 192 GB HBM3e memory (with bandwidth reaching 8 TB/s).
80%VerifiedBlackwell's second-generation Transformer Engine adds FP4 precision support, which in theory can improve Transformer model training throughput by 2–4x compared to the H100.
75%VerifiedA 35B model requires approximately 70GB of VRAM at FP16 precision, far exceeding the RTX 5090's 32GB capacity, so single-GPU deployment requires quantization.
75%VerifiedThe NVIDIA H100 SXM5 has a peak FP16 compute performance of approximately 989 TFLOPS, with a single-card price exceeding $30,000 and a cloud rental cost of approximately $2–4 per hour.
75%VerifiedTop-tier AI chips such as the H100 are priced at approximately $30,000–$40,000 per unit.
75%VerifiedThe United States has restricted exports of high-end GPUs to China, including NVIDIA's A100 and H100
75%VerifiedHBM is a core component of Nvidia's AI training chips including the A100, H100, and H200.
75%VerifiedEven after quantization, 70B-level models often require more than 40 GB of VRAM or a multi-GPU configuration.
75%VerifiedMainstream NVMe SSDs can achieve sequential read speeds of 3500–7000 MB/s.
75%VerifiedThe KV Cache of a 70B-parameter model with 128K context support can require hundreds of gigabytes of VRAM at full load, exceeding the 80GB VRAM capacity of a single H100.
70%