OOM
A phenomenon in large model inference where GPU VRAM is exhausted, causing the process to crash. Because GPU memory is physically isolated and cannot elastically expand, the primary trigger in high-concurrency or long-context scenarios is the depletion of KV cache capacity.
Timeline (last 90 days)
A severely overspecified model parameter count will cause an OOM (out-of-memory) error and prevent startup, while a slightly overspecified count will borrow regular memory and cause a significant drop in speed
OOM, or Out of Memory, refers to a situation where the memory requested by a program exceeds the available system limit, causing the operating system to forcibly terminate the process.
For devices with tight memory, blindly using a larger quantization model may result in frequent OOM errors.
OOM (out-of-memory / GPU memory overflow) is a stability threat unique to large model inference services, because GPU VRAM is a physically isolated fixed resource that cannot be temporarily expanded using virtual memory or swap the way CPU memory can.
After loading a 70B-parameter model onto an 80GB A100, the space left for the KV cache is extremely limited, and when concurrency increases or context length grows, the KV cache can rapidly exhaust VRAM, triggering OOM and crashing the inference process.
VRAM fragmentation can cause large tensor allocation to fail with an OOM error even when the total remaining VRAM is sufficient, because no contiguous block of adequate size can be found.
One of the core bottlenecks for running large language models locally is memory capacity, as model parameters need to be loaded into RAM or VRAM.
When memory is pushed to the limit, Linux's OOM mechanism may kill the service process due to excessive RAM usage; this can be avoided by setting oom_score_adj to -1000 to prevent the system from mistakenly killing it.
Long-context reasoning accumulates large attention matrices in GPU VRAM, and when the context window approaches the model's maximum length limit, VRAM consumption rises sharply, leading to out-of-memory (OOM) errors.
All Facts (9)
A severely overspecified model parameter count will cause an OOM (out-of-memory) error and prevent startup, while a slightly overspecified count will borrow regular memory and cause a significant drop in speed
50%UnverifiedOOM, or Out of Memory, refers to a situation where the memory requested by a program exceeds the available system limit, causing the operating system to forcibly terminate the process.
50%UnverifiedFor devices with tight memory, blindly using a larger quantization model may result in frequent OOM errors.
50%UnverifiedOOM (out-of-memory / GPU memory overflow) is a stability threat unique to large model inference services, because GPU VRAM is a physically isolated fixed resource that cannot be temporarily expanded using virtual memory or swap the way CPU memory can.
50%UnverifiedAfter loading a 70B-parameter model onto an 80GB A100, the space left for the KV cache is extremely limited, and when concurrency increases or context length grows, the KV cache can rapidly exhaust VRAM, triggering OOM and crashing the inference process.
50%UnverifiedVRAM fragmentation can cause large tensor allocation to fail with an OOM error even when the total remaining VRAM is sufficient, because no contiguous block of adequate size can be found.
50%UnverifiedOne of the core bottlenecks for running large language models locally is memory capacity, as model parameters need to be loaded into RAM or VRAM.
50%UnverifiedWhen memory is pushed to the limit, Linux's OOM mechanism may kill the service process due to excessive RAM usage; this can be avoided by setting oom_score_adj to -1000 to prevent the system from mistakenly killing it.
50%UnverifiedLong-context reasoning accumulates large attention matrices in GPU VRAM, and when the context window approaches the model's maximum length limit, VRAM consumption rises sharply, leading to out-of-memory (OOM) errors.
50%