HBM
High Bandwidth Memory (HBM) is a high-performance memory standard based on three-dimensional stacking technology. It achieves extremely high data transfer bandwidth and energy efficiency by vertically stacking multiple layers of DRAM dies interconnected via through-silicon vias (TSVs). HBM is used primarily in scenarios with demanding memory bandwidth requirements, including AI training, high-performance computing, and graphics processing, and is an important component of modern GPUs and AI accelerator chips.
Core Facts
The HBM3 memory used by NVIDIA's H100 has a bandwidth of up to 3.35 TB/s.
HBM is a packaging technology that vertically stacks multiple layers of DRAM chips through Through-Silicon Via (TSV) technology and integrates them with the GPU via a silicon interposer.
Compared to the previous-generation V4 Flash, V4.1 Flash reduces the demand for high-bandwidth memory (HBM) to one-quarter and the demand for SSD persistent storage to one-eighth.
Timeline (last 90 days)
Swapping KV cache in and out depends on the interconnect bandwidth between host memory and GPU VRAM; taking PCIe 4.0 x16 as an example, the one-way effective bandwidth is on the order of about 25 GB/s, while the HBM bandwidth of a single data center GPU can reach 2–3 TB/s, a difference of about two orders of magnitude.
Compared with HBM, LPDDR5X compromises somewhat on bandwidth but has advantages in capacity expansion cost and power consumption control.
HBM的容量决定单卡能承载的模型参数量与KV Cache长度，其堆叠层数和封装工艺往往比算力峰值更能左右一张AI卡的实际商业价值。
HBM has an extremely wide physical bus width (up to 1024 bits or higher), enabling it to deliver far greater memory bandwidth than GDDR at lower frequencies
NVIDIA H100/H200 relies on CoWoS to stack-package GPU dies together with HBM memory.
The compute bottleneck has shifted from chip production capacity to advanced packaging (CoWoS) and high-bandwidth memory (HBM).
As the scale of large language model training clusters expands, HBM demand is growing at an approximate rate of doubling per year.
Modern HBM memory incorporates a built-in Post-Package Repair (PPR) mechanism, allowing a small number of faulty rows to be repaired via electrical signals after packaging.
AI servers consume far more HBM per unit than traditional servers.
Currently, there are only three major global suppliers of HBM: Samsung, SK Hynix, and Micron
40 more timeline events
All Facts (20)
The HBM3 memory used by NVIDIA's H100 has a bandwidth of up to 3.35 TB/s.
90%VerifiedHBM is a packaging technology that vertically stacks multiple layers of DRAM chips through Through-Silicon Via (TSV) technology and integrates them with the GPU via a silicon interposer.
90%VerifiedCompared to the previous-generation V4 Flash, V4.1 Flash reduces the demand for high-bandwidth memory (HBM) to one-quarter and the demand for SSD persistent storage to one-eighth.
75%VerifiedHBM achieves high-density interlayer interconnection by vertically stacking multiple layers of DRAM chips using through-silicon via (TSV) technology, delivering memory bandwidth up to 3–5 times higher than conventional GDDR6 memory, while consuming less power and occupying a smaller package footprint.
75%VerifiedThe GB300 integrates HBM3e high-bandwidth memory, with a single-card capacity of up to 288GB and memory bandwidth exceeding 10TB/s
75%VerifiedHBM is a core component of Nvidia's AI training chips including the A100, H100, and H200.
75%VerifiedMainstream NVMe SSDs can achieve sequential read speeds of 3500–7000 MB/s.
75%VerifiedWhen running large models locally, GPU VRAM is often more critical than compute power; hardware selection should prioritize VRAM capacity over performance benchmarks.
65%VerifiedThe Prefill stage of LLM inference is compute-intensive with GPU utilization close to 100%, while the Decode stage is memory-bandwidth-intensive with the bottleneck being HBM VRAM bandwidth.
65%UnverifiedTraditional GPU memory relies primarily on HBM, which achieves high bandwidth through 3D stacking and ultra-wide buses but has limited capacity, high expense, and significant power consumption
90%UnverifiedStorage-class memory exhibits notable disadvantages in access latency and bandwidth compared to HBM's multi-terabyte-per-second bandwidth
85%Unverified内存行业的晶圆产能需要在DDR、LPDDR和HBM三种主要内存类型之间分配
85%Unverified在AI浪潮到来之前，HBM仅占晶圆分配的2%
85%Unverified每生产1GB的HBM所消耗的晶圆产能是生产1GB DDR或LPDDR的三倍以上
80%UnverifiedBoth the NVIDIA H100/H200 and the AMD MI300X integrate HBM.
70%UnverifiedFlashAttention reduces the number of reads and writes to HBM by retaining intermediate results in SRAM as much as possible, thereby breaking through the memory bandwidth bottleneck to achieve speedup.
60%UnverifiedA tiered memory architecture organizes HBM, DRAM, and CXL extended memory pools into a hierarchical structure, with HBM serving as the high-speed cache layer, DRAM as the main memory layer, and CXL providing large-capacity shareable memory resources.
60%UnverifiedHBM uses TSV technology to drill thousands of micron-scale through-holes in extremely thin DRAM wafers, requiring each DRAM die layer to be ground down to approximately 30–40 micrometers in thickness.
60%UnverifiedSRAM access latency is approximately 1 ns, HBM approximately 100 ns, making SRAM latency about 10–50 times lower than HBM.
60%UnverifiedThe global HBM supply is almost entirely monopolized by three companies: SK Hynix, Samsung, and Micron.
60%