Reading Data Directly into GPU Memory via GPUDirect Storage in Python

AI training and inference datasets continue to grow in scale, while GPU compute capacity grows much faster than the throughput of traditional I/O paths. NVIDIA GPUDirect Storage (GDS) provides a direct path that bypasses CPU memory, allowing data from storage devices to be written directly into GPU memory. Based on technical notes from NVIDIA's developer blog, this article outlines how GDS works and offers practical guidance for reading data into GPU memory in Python.

Why Bypass CPU Memory

The traditional data loading path is: storage device → system memory (page cache) → CPU copy → GPU memory. Along this path, data must be moved twice, making both the CPU and the PCIe bus bottlenecks, while also consuming valuable host memory bandwidth. When datasets reach the terabyte scale and the number of GPUs increases, I/O often becomes the limiting factor before compute does.

The core idea of GDS is to establish a direct DMA channel between storage and GPU memory. Data flows from NVMe devices through the PCIe switch directly into GPU memory, with the CPU only initiating and coordinating—no longer participating in data movement. According to NVIDIA's developer blog, this path reduces the number of data copies, lowers CPU load, and allows the GPU to start computing sooner.

DMA (Direct Memory Access) is a mechanism that allows hardware devices to transfer data directly to or from system memory or device memory without CPU intervention. In the traditional I/O path, the CPU must personally participate in data movement (e.g., copying from the kernel buffer to the user buffer), which not only consumes CPU cycles but also reduces overall efficiency through context switches and cache pollution. GDS uses DMA engines to let the NVMe controller write data directly into GPU memory, with the CPU only responsible for issuing transfer commands and coordinating completion notifications. In this mode, the CPU is freed from heavy data movement and can focus on task scheduling or other work, while redundant traffic on the PCIe bus is also reduced. Understanding DMA is the foundation for understanding GDS's performance advantages: it explains why "bypassing CPU memory" can simultaneously reduce latency and CPU utilization.

How GPUDirect Storage Works

Storage-Side and Filesystem Support

GDS does not work transparently with any storage. It relies on local NVMe devices that support direct I/O, or certified networked storage solutions (such as RDMA-based parallel filesystems). The filesystem must provide a compatible layer, and the cuFile driver is responsible for establishing mappings between storage and CUDA devices in user space.

The Role of the cuFile API

NVIDIA provides cuFile, a set of C APIs that wrap operations such as opening files, registering buffers, and performing reads and writes. The typical Python path for using GDS is built on top of cuFile, with upper-layer libraries (such as cuCIM and KvikIO) wrapping it into Python interfaces. Understanding cuFile semantics helps troubleshoot performance issues—for example, buffers must be aligned, and reads and writes require explicit synchronization.

cuFile is NVIDIA's official user-space API set for GPUDirect Storage, part of the CUDA toolkit. It provides a set of functions similar to POSIX file operations, such as cuFileOpen, cuFileRead, cuFileWrite, and cuFileClose, but specifically designed to establish direct data transfers between storage devices and GPU memory. Internally, cuFile interacts with the filesystem through a kernel driver (nvidia-fs) and uses DMA engines to perform the actual I/O. Developers should note that cuFile reads and writes are asynchronous and must be synchronized with CUDA streams using functions like cuFileStreamRegister and cuFileStreamDeregister to ensure data readiness. In addition, cuFile requires file descriptors to be opened with O_DIRECT, and buffers must be registered (cuFileBufRegister) to enable the direct path. These semantics differ significantly from ordinary file I/O and are a common source of errors when using GDS.

Paths for Using GDS in Python

Leveraging Wrapper Libraries Such as KvikIO

Currently, Python developers rarely call cuFile directly; the more common approach is to use wrapper libraries maintained by NVIDIA. Take KvikIO as an example: it provides file-oriented read/write interfaces and internally selects GDS or the traditional POSIX path automatically based on the environment. Developers can load data from files into CuPy arrays or PyTorch tensors with minimal code, without writing C extensions by hand.

Buffer and Alignment Considerations

When using GDS, the target buffer must reside in GPU memory and meet alignment requirements. On the Python side, GPU memory tensors are typically allocated via CuPy or PyTorch, and their pointers are passed to the underlying interface. If the buffer comes from host memory or is unaligned, GDS falls back to a compatible path and the performance gains disappear. Therefore, code should explicitly confirm that data lands in GPU memory and avoid triggering implicit copies during loading.

Memory alignment is critical in GDS because direct DMA transfers require both source and destination addresses to meet device-specific alignment boundaries (typically 4KB or larger). For GPU memory buffers, tensors allocated by CuPy and PyTorch are usually already aligned to CUDA requirements, but obtaining a sub-region via slicing or offset operations may break alignment. In addition, file offsets and transfer lengths must also be aligned; otherwise, cuFile will reject the direct transfer and fall back to a compatible path. Developers can use cuFileGetAlignment to query specific alignment requirements and ensure buffer sizes are integer multiples of the alignment granularity. In Python, you can verify by checking whether the tensor's data_ptr() modulo the alignment value equals zero, or simply use the aligned allocation utilities provided by wrapper libraries. Ignoring alignment not only degrades performance but can also cause silent fallbacks that are difficult to diagnose.

Error Fallback and Portability

GDS depends on specific hardware and driver combinations and is not supported in every environment. Mature wrapper libraries probe capabilities at initialization and automatically fall back to traditional read methods when unsupported. For applications, this means the same code can move between development machines and deployment environments, but performance will differ. It is recommended to log the actual path used to help locate performance regressions.

Applicable Scenarios and Trade-offs

The benefits of GDS are most obvious in the following scenarios: large individual read chunks, datasets far exceeding host memory capacity, CPUs simultaneously burdened with other intensive tasks, and multiple GPUs loading in parallel from the same storage pool. For frequent random reads of small files, or when data already resides in the page cache, the extra coordination overhead of the direct path may offset the gains.

From an engineering perspective, GDS is not a universal accelerator but a targeted solution to I/O bottlenecks. Before adopting it, use a profiler to confirm that the bottleneck is indeed on the data loading side, then evaluate whether the storage device and filesystem meet GDS prerequisites. The original NVIDIA developer blog also emphasizes that understanding the overhead of each stage in the data path is more important than blindly enabling new features.

Page cache is a memory region used by the operating system kernel to cache file data, intended to accelerate repeated reads. In the traditional I/O path, file data is first read into the page cache and then copied from the page cache to the user buffer. The GDS direct path typically bypasses the page cache (opening files with O_DIRECT), which means each read accesses the storage device directly and no copy is retained in memory. As a result, for workloads that frequently reread the same small files, GDS may perform worse than the traditional path due to the lack of caching. Conversely, for scenarios involving one-time reads of large-scale datasets or data volumes far exceeding memory capacity, bypassing the page cache avoids cache pollution and memory waste. When evaluating whether to enable GDS, weigh the data access pattern: for sequential large-block reads with no reuse, GDS offers clear advantages; for random small-file reads with hot data, the traditional cached path may be more appropriate.

Summary

GPUDirect Storage compresses the data loading path from two copies into a single direct transfer, providing a more efficient I/O option for large-scale AI workloads. In the Python ecosystem, wrapper libraries such as KvikIO make it possible to adopt this capability at low cost. In practice, hardware compatibility, buffer alignment, and fallback behavior are the three areas that require particular attention.

One-line Summary

GDS bypasses CPU memory via a direct DMA channel, reducing copies from storage to GPU memory from two to one; in Python, it can be accessed through libraries such as KvikIO.

Paragraph Summary

This article introduces the working principles of NVIDIA GPUDirect Storage (GDS) and its practical path in Python. Traditional data loading goes storage → system memory → CPU copy → GPU memory, and the two moves make the CPU and PCIe bottlenecks. GDS establishes a direct DMA channel between storage and GPU memory, with the CPU only coordinating rather than moving data, thereby reducing copies and CPU load. The article details the role of the cuFile API and storage/filesystem support, and notes that Python developers typically use wrapper libraries such as KvikIO. Buffers must reside in GPU memory and meet alignment requirements; otherwise, automatic fallback occurs. Finally, it outlines applicable scenarios: large-block reads, datasets exceeding host memory, and multi-GPU shared storage pools benefit clearly, while random small-file reads may not be worthwhile. Use a profiler to confirm the bottleneck before enabling GDS.