Memory Offloading
A large-model inference optimization technique used when GPU VRAM is insufficient to hold a complete model. Selected model layers are offloaded to system RAM or disk-based virtual memory and loaded back into VRAM on demand during inference, enabling large-parameter models to run with limited GPU memory.
Timeline (last 90 days)
Combining quantization with memory offloading theoretically allows loading models far exceeding capacity in extremely low-VRAM environments, but the data transfer introduces additional latency, and actual inference speed is often only a fraction of that achieved with pure VRAM operation.
Memory-bound typically refers to insufficient GPU memory capacity, which requires offloading or model parallelism to address.
All Facts (2)
Combining quantization with memory offloading theoretically allows loading models far exceeding capacity in extremely low-VRAM environments, but the data transfer introduces additional latency, and actual inference speed is often only a fraction of that achieved with pure VRAM operation.
50%UnverifiedMemory-bound typically refers to insufficient GPU memory capacity, which requires offloading or model parallelism to address.
50%