Storage-Class Memory Revolution: GPU Memory May Leap to Multi-Terabyte Capacity

The Memory Bottleneck: The Hidden Ceiling on AI Compute Growth

As demand for large model training and inference explodes, GPU compute power often dominates the headlines, but what truly constrains real-world performance is frequently memory capacity. Current flagship AI accelerators typically offer only tens to low hundreds of gigabytes of memory—for example, the NVIDIA H100 comes with 80GB of HBM3, and even the latest high-end models struggle to break the 200GB barrier.

However, a model with hundreds of billions or even trillions of parameters, when accounting for weights, intermediate activations during training, and optimizer states, can easily require hundreds of gigabytes or even several terabytes of memory. This forces engineers to employ complex techniques like model parallelism and tensor sharding, splitting models across dozens or hundreds of GPUs working in concert. This approach not only drives up hardware costs but also introduces massive cross-device communication overhead.

It is against this backdrop that a new memory solution inspired by storage technology has captured industry attention. According to discussions in the tech community, this technology could enable a single GPU's usable memory capacity to "explosively" grow to the multi-terabyte level.

Borrowing from Storage Architecture: A New Approach to Memory Tiering

From HBM to Storage-Class Memory

Traditional GPU memory relies primarily on HBM (High Bandwidth Memory), which achieves extremely high bandwidth through 3D stacking and ultra-wide buses—but at the cost of limited capacity, high expense, and significant power consumption. Simply stacking more HBM to reach terabyte-level capacity is impractical both physically and economically.

The core idea behind the new approach is to "learn from storage technology" and build a tiered memory hierarchy. The inspiration comes from the enormous density advantages that modern storage technologies like SSDs and flash memory offer—flash chips provide far greater storage density per unit area than DRAM, at a much lower cost. By introducing these storage-oriented media into the GPU memory hierarchy, it becomes theoretically possible to expand GPU-addressable memory space to multiple terabytes at an acceptable cost.

Tiered Caching Strategies and Performance Trade-offs

This "storage-class memory" is not without trade-offs. Compared to HBM's multi-terabyte-per-second bandwidth, storage-oriented media exhibit notable disadvantages in access latency and bandwidth. Therefore, these technologies typically employ tiered caching strategies: the hottest, most frequently accessed data resides in high-speed HBM, while less frequently accessed bulk data (such as certain model weights, historical KV caches, etc.) is placed in high-capacity storage-class memory.

For large model inference scenarios, this tiered architecture is particularly compelling. Many inference workloads are bottlenecked by capacity rather than pure bandwidth—as long as an entire model can fit on a single card, avoiding expensive cross-card communication, overall efficiency may improve dramatically even if some data accesses are slightly slower.

Potential Impact on AI Infrastructure

Lowering the Barrier to Large Model Deployment

If single-card memory can be expanded to multiple terabytes, the most direct beneficiaries will be large model deployment and inference. Running an ultra-large-scale model today often requires an entire server equipped with 8 GPUs, but in the future, perhaps just a few accelerator cards with massive memory could accomplish the same task. This would significantly reduce the hardware threshold and operational costs for inference services.

For small and medium-sized enterprises and research institutions, this means the opportunity to run previously inaccessible large models locally without building enormous GPU clusters, further democratizing AI applications.

Redefining the Balance Between Compute and Memory

For a long time, GPU design has sought a balance between compute power (FLOPS) and memory capacity and bandwidth. The introduction of storage-class memory technology could break existing proportional relationships, making "large memory, moderate bandwidth" a new product form factor specifically serving capacity-sensitive AI workloads.

You may not have noticed, but these technologies are currently more at the proof-of-concept and early exploration stage. While related discussions in the tech community have garnered limited attention, they nonetheless reflect the industry's sustained interest in memory expansion directions. Any new memory technology must overcome multiple hurdles—yield rates, cost, software stack adaptation—before it can truly be commercialized.

A Balanced Perspective: Opportunities and Challenges Coexist

From a technology evolution standpoint, further blurring the boundary between storage and memory is a natural trend in computing architecture development. Emerging interconnect standards like CXL (Compute Express Link) are already driving memory pooling and tiered memory into reality, while introducing storage-class memory into GPUs represents an extension of this philosophy into AI hardware.

However, we must also maintain a cautious outlook. Bandwidth remains a critical performance metric for many AI training tasks, and whether storage-class memory can play a role in training scenarios depends on whether its latency and throughput can meet practical demands. Moreover, software ecosystem adaptation is equally crucial—deep learning frameworks need to intelligently manage data scheduling across different memory tiers to truly unlock the hardware's potential.

Overall, storage-class memory technology paints an enticing picture: freeing GPUs from the shackles of memory capacity limitations and providing ample space for ever-larger AI models. The extent to which it can deliver on the promise of "multi-terabyte memory" remains to be proven through time and actual products. But one thing is certain: memory architecture innovation is becoming a critical battleground in the next round of AI hardware competition.